SteNicholas opened a new pull request, #1448:
URL: https://github.com/apache/paimon-rust/pull/1448

   ### Purpose
   
   Linked issue: #1425
   
   Materializing a query in a persistent Paimon table currently requires a 
schema declaration and a separate INSERT. This adds native `CREATE TABLE AS 
SELECT` to the DataFusion SQL context, inferring and populating the table from 
the same resolved query while preserving table identity across concurrent DDL 
and retaining files when publication is uncertain.
   
   ### Brief change log
   
   - Infer column names, supported types and nullability from the analyzed 
query, including empty results. Reuse CREATE TABLE validation for partition 
keys, options and comments. Reject explicit column declarations, unsupported 
modifiers and time outputs that would lose precision.
   - Resolve source tables and SQL functions in the session namespace 
independently of the destination. Preserve quoted identifiers and qualified 
time-travel references, execute the same resolved plan used for inference, and 
normalize scalar and nested Arrow output layouts for writing.
   - Skip existing native and external targets with IF NOT EXISTS before source 
planning or execution. Publish initial schemas atomically, bind writers to a 
unique table creation identity, and coordinate filesystem catalog DDL and 
snapshot publication with database lifecycle locks. Enforce the same identity 
check in the filesystem-backed REST server.
   - Retain the created table after population errors and preserve concurrent 
writers' rows. Clean up owned output after pre-commit failures and 
cancellation, finish commits already started, and retain prepared files after 
commit errors because publication may have succeeded.
   
   ### Tests
   
   - Integration tests in 
`crates/integrations/datafusion/tests/create_table_as.rs`: inferred schemas, 
empty results, nullability, partitioning, options, nested Arrow layouts, time 
precision, quoted names, cross-catalog destinations, time travel and REST SQL 
functions. Verify existing-target no-ops and rejected statements preserve 
catalog and source data.
   - Execution tests in 
`crates/integrations/datafusion/tests/create_table_as/execution.rs`: reuse of 
one source plan, creation races, lookup errors, concurrent writes, dropped or 
replaced targets, source and write failures, failed or uncertain publication, 
and cancellation before and during commit. Check table retention, committed 
rows and cleanup of owned files.
   - Reuse `read_files_behavior` coverage for temporary and persistent CTAS. 
Extend filesystem catalog, FileIO and REST server tests for atomic creation, 
lifecycle locking and stale-writer rejection, and run existing SQLContext and 
SQL integration tests against the shared CREATE TABLE validation.
   
   - Validation: 374 related tests passed after rebasing onto upstream/main: 
205 SQLContext unit tests, 107 DataFusion integration tests across 
`create_table_as`, `sql_context_tests` and `read_files_behavior`, 48 filesystem 
catalog/FileIO tests, and 14 REST server tests. `cargo +1.98.0 fmt --all -- 
--check` and `cargo +1.98.0 clippy --locked --all-targets --workspace 
--features vortex -- -D warnings` passed.
   
   ### API and Format
   
   Add `Catalog::create_table_for_write`, `TableSchema::creation_id`, 
`FileSystemCatalog::commit_snapshot` and `Error::TableReplaced`. Persist an 
internal `internal.table-creation-id` schema option for owned writes; data-file 
and snapshot encodings are unchanged. Filesystem table creation requires atomic 
initial-schema publication, and concurrent DDL clients must share the database 
locking protocol. REST catalogs must preserve creation identity and validate it 
during snapshot publication. CTAS supports native Paimon tables; OR REPLACE and 
external-engine CTAS remain unsupported.
   
   ### Documentation
   
   Update `docs/src/sql.md` and `crates/integrations/datafusion/README.md` with 
CTAS syntax, schema inference, supported time precision, namespace resolution 
and IF NOT EXISTS behavior. Document table retention, cancellation and 
uncertain commits, database lifecycle locking, and recovery of abandoned 
object-storage locks.
   
   🤖 Assisted with Codex.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to