SteNicholas opened a new issue, #1425:
URL: https://github.com/apache/paimon-rust/issues/1425

   ### Search before asking
   
   - [x] I searched existing issues and pull requests. No open issue or active 
CTAS PR was found on November 17, 2026. Related closed PR #1021 is linked below.
   
   ### Motivation
   
   Persistent `CREATE TABLE ... AS SELECT ...` is rejected by 
[SQLContext::handle_create_table](https://github.com/apache/paimon-rust/blob/1920eab1c60d2b5680fb767dbd60ae0cf668db78/crates/integrations/datafusion/src/sql_context.rs#L1136-L1140).
 Users must create a destination schema manually and then issue a separate 
INSERT to materialize a query.
   
   For example, with an existing source table:
   
   ```sql
   CREATE TABLE paimon.test_db.query_result
   WITH ('bucket' = '-1')
   AS SELECT id, name FROM paimon.test_db.source WHERE id > 10;
   ```
   
   The destination should be a persistent Paimon table whose schema is inferred 
from the SELECT and whose rows are the query result.
   
   PR #1021 attempted this feature but was closed without merging. Its 
discussion identified source-namespace, identifier quoting, IF NOT EXISTS and 
cleanup-ownership problems. This request tracks a new implementation with those 
contracts made explicit rather than assuming that an unguarded CREATE plus 
INSERT is sufficient.
   
   ### Solution
   
   Support persistent CTAS for native Paimon tables through SQLContext.
   
   - Infer the destination schema from the analyzed query output, preserving 
supported types and nullability. Apply supported table options and partitioning 
through existing schema validation. An empty result should still create a table 
with the inferred schema.
   - Reuse the same analyzed source plan for schema inference and population. 
Resolve unqualified source tables and catalog SQL functions in the current 
session namespace, independently of the destination namespace. Reusing a plan 
does not imply adding a transaction across unrelated source tables.
   - Preserve quoted/case-sensitive source and destination identifiers. Avoid 
rebuilding an INSERT string that can normalize the destination differently or 
resolve the source again.
   - Treat `IF NOT EXISTS` on an existing target as a no-op: do not append rows 
or execute the source data query. Distinguish missing tables from 
authorization, transport and other catalog errors.
   - Handle concurrent creation atomically. After an optional existence check, 
create strictly and handle a raced AlreadyExists before population. A 
successful lenient no-op create is not proof that this statement owns the 
target.
   - Define failure cleanup using target ownership and publication state. Never 
append to or drop a target created by a competing client. Do not 
unconditionally drop a possibly published table after a lost commit response, 
or delete unrelated concurrent data.
   - Reject unsupported CREATE modifiers and explicit typed column declarations 
in the first version rather than silently ignoring them. CREATE OR REPLACE and 
CTAS into external-engine tables are outside the initial scope.
   
   Suggested validation:
   
   - Inferred schemas, nullability, filters/expressions, partitioning/options, 
empty results and invalid output schemas.
   - Quoted identifiers and cross-catalog/database destinations where 
unqualified source names still resolve in the session namespace.
   - Registered scalar UDFs and REST catalog SQL functions, with inference and 
execution using the same resolved inputs.
   - Existing-target IF NOT EXISTS without population, including an existing 
externally served target.
   - A deterministic create race where another client creates and populates the 
target between the check and create.
   - Failures during source execution, writing and publication, proving that 
cleanup affects only owned, safely uncommitted resources.
   
   ### Anything else?
   
   References:
   
   - [Closed, unmerged CTAS PR 
#1021](https://github.com/apache/paimon-rust/pull/1021), especially its 
source-resolution and create-after-check ownership discussion.
   - Java 
[SparkCatalog.createTable](https://github.com/apache/paimon/blob/00b4d05a6cc118091a38eeca23549519cfb775f4/paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/SparkCatalog.java#L427-L449)
 uses strict catalog creation.
   - Java 
[stageCreateDirectly](https://github.com/apache/paimon/blob/00b4d05a6cc118091a38eeca23549519cfb775f4/paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/SparkCatalog.java#L666-L682)
 provides a staged-table/rollback reference; the Rust error and publication 
contract should be explicit.
   
   Source baseline: paimon-rust `1920eab1c60d2b5680fb767dbd60ae0cf668db78`, 
compared with Java Paimon `00b4d05a6cc118091a38eeca23549519cfb775f4`. This is a 
source-based feature request, not a claim that new failure probes were executed.
   
   ### Willingness to contribute
   
   - [ ] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to