SteNicholas opened a new issue, #1425: URL: https://github.com/apache/paimon-rust/issues/1425
### Search before asking - [x] I searched existing issues and pull requests. No open issue or active CTAS PR was found on November 17, 2026. Related closed PR #1021 is linked below. ### Motivation Persistent `CREATE TABLE ... AS SELECT ...` is rejected by [SQLContext::handle_create_table](https://github.com/apache/paimon-rust/blob/1920eab1c60d2b5680fb767dbd60ae0cf668db78/crates/integrations/datafusion/src/sql_context.rs#L1136-L1140). Users must create a destination schema manually and then issue a separate INSERT to materialize a query. For example, with an existing source table: ```sql CREATE TABLE paimon.test_db.query_result WITH ('bucket' = '-1') AS SELECT id, name FROM paimon.test_db.source WHERE id > 10; ``` The destination should be a persistent Paimon table whose schema is inferred from the SELECT and whose rows are the query result. PR #1021 attempted this feature but was closed without merging. Its discussion identified source-namespace, identifier quoting, IF NOT EXISTS and cleanup-ownership problems. This request tracks a new implementation with those contracts made explicit rather than assuming that an unguarded CREATE plus INSERT is sufficient. ### Solution Support persistent CTAS for native Paimon tables through SQLContext. - Infer the destination schema from the analyzed query output, preserving supported types and nullability. Apply supported table options and partitioning through existing schema validation. An empty result should still create a table with the inferred schema. - Reuse the same analyzed source plan for schema inference and population. Resolve unqualified source tables and catalog SQL functions in the current session namespace, independently of the destination namespace. Reusing a plan does not imply adding a transaction across unrelated source tables. - Preserve quoted/case-sensitive source and destination identifiers. Avoid rebuilding an INSERT string that can normalize the destination differently or resolve the source again. - Treat `IF NOT EXISTS` on an existing target as a no-op: do not append rows or execute the source data query. Distinguish missing tables from authorization, transport and other catalog errors. - Handle concurrent creation atomically. After an optional existence check, create strictly and handle a raced AlreadyExists before population. A successful lenient no-op create is not proof that this statement owns the target. - Define failure cleanup using target ownership and publication state. Never append to or drop a target created by a competing client. Do not unconditionally drop a possibly published table after a lost commit response, or delete unrelated concurrent data. - Reject unsupported CREATE modifiers and explicit typed column declarations in the first version rather than silently ignoring them. CREATE OR REPLACE and CTAS into external-engine tables are outside the initial scope. Suggested validation: - Inferred schemas, nullability, filters/expressions, partitioning/options, empty results and invalid output schemas. - Quoted identifiers and cross-catalog/database destinations where unqualified source names still resolve in the session namespace. - Registered scalar UDFs and REST catalog SQL functions, with inference and execution using the same resolved inputs. - Existing-target IF NOT EXISTS without population, including an existing externally served target. - A deterministic create race where another client creates and populates the target between the check and create. - Failures during source execution, writing and publication, proving that cleanup affects only owned, safely uncommitted resources. ### Anything else? References: - [Closed, unmerged CTAS PR #1021](https://github.com/apache/paimon-rust/pull/1021), especially its source-resolution and create-after-check ownership discussion. - Java [SparkCatalog.createTable](https://github.com/apache/paimon/blob/00b4d05a6cc118091a38eeca23549519cfb775f4/paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/SparkCatalog.java#L427-L449) uses strict catalog creation. - Java [stageCreateDirectly](https://github.com/apache/paimon/blob/00b4d05a6cc118091a38eeca23549519cfb775f4/paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/SparkCatalog.java#L666-L682) provides a staged-table/rollback reference; the Rust error and publication contract should be explicit. Source baseline: paimon-rust `1920eab1c60d2b5680fb767dbd60ae0cf668db78`, compared with Java Paimon `00b4d05a6cc118091a38eeca23549519cfb775f4`. This is a source-based feature request, not a claim that new failure probes were executed. ### Willingness to contribute - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
