SteNicholas opened a new pull request, #1448: URL: https://github.com/apache/paimon-rust/pull/1448
### Purpose Linked issue: #1425 Materializing a query in a persistent Paimon table currently requires a schema declaration and a separate INSERT. This adds native `CREATE TABLE AS SELECT` to the DataFusion SQL context, inferring and populating the table from the same resolved query while preserving table identity across concurrent DDL and retaining files when publication is uncertain. ### Brief change log - Infer column names, supported types and nullability from the analyzed query, including empty results. Reuse CREATE TABLE validation for partition keys, options and comments. Reject explicit column declarations, unsupported modifiers and time outputs that would lose precision. - Resolve source tables and SQL functions in the session namespace independently of the destination. Preserve quoted identifiers and qualified time-travel references, execute the same resolved plan used for inference, and normalize scalar and nested Arrow output layouts for writing. - Skip existing native and external targets with IF NOT EXISTS before source planning or execution. Publish initial schemas atomically, bind writers to a unique table creation identity, and coordinate filesystem catalog DDL and snapshot publication with database lifecycle locks. Enforce the same identity check in the filesystem-backed REST server. - Retain the created table after population errors and preserve concurrent writers' rows. Clean up owned output after pre-commit failures and cancellation, finish commits already started, and retain prepared files after commit errors because publication may have succeeded. ### Tests - Integration tests in `crates/integrations/datafusion/tests/create_table_as.rs`: inferred schemas, empty results, nullability, partitioning, options, nested Arrow layouts, time precision, quoted names, cross-catalog destinations, time travel and REST SQL functions. Verify existing-target no-ops and rejected statements preserve catalog and source data. - Execution tests in `crates/integrations/datafusion/tests/create_table_as/execution.rs`: reuse of one source plan, creation races, lookup errors, concurrent writes, dropped or replaced targets, source and write failures, failed or uncertain publication, and cancellation before and during commit. Check table retention, committed rows and cleanup of owned files. - Reuse `read_files_behavior` coverage for temporary and persistent CTAS. Extend filesystem catalog, FileIO and REST server tests for atomic creation, lifecycle locking and stale-writer rejection, and run existing SQLContext and SQL integration tests against the shared CREATE TABLE validation. - Validation: 374 related tests passed after rebasing onto upstream/main: 205 SQLContext unit tests, 107 DataFusion integration tests across `create_table_as`, `sql_context_tests` and `read_files_behavior`, 48 filesystem catalog/FileIO tests, and 14 REST server tests. `cargo +1.98.0 fmt --all -- --check` and `cargo +1.98.0 clippy --locked --all-targets --workspace --features vortex -- -D warnings` passed. ### API and Format Add `Catalog::create_table_for_write`, `TableSchema::creation_id`, `FileSystemCatalog::commit_snapshot` and `Error::TableReplaced`. Persist an internal `internal.table-creation-id` schema option for owned writes; data-file and snapshot encodings are unchanged. Filesystem table creation requires atomic initial-schema publication, and concurrent DDL clients must share the database locking protocol. REST catalogs must preserve creation identity and validate it during snapshot publication. CTAS supports native Paimon tables; OR REPLACE and external-engine CTAS remain unsupported. ### Documentation Update `docs/src/sql.md` and `crates/integrations/datafusion/README.md` with CTAS syntax, schema inference, supported time precision, namespace resolution and IF NOT EXISTS behavior. Document table retention, cancellation and uncertain commits, database lifecycle locking, and recovery of abandoned object-storage locks. 🤖 Assisted with Codex. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
