JingsongLi opened a new pull request, #1315: URL: https://github.com/apache/paimon-rust/pull/1315
### Purpose Use DataFusion 55's native MERGE planner and Paimon's table provider for execution. Planning and EXPLAIN must not mutate tables; executing the DML plan applies the merge and returns its affected-row count. ### Brief change log - Implement TableProvider::merge_into with typed DataFusion expressions and a DataSinkExec, replacing the custom MERGE AST-to-SQL execution path. - Preserve data-evolution partial-column updates, copy-on-write rewrites, atomic publication, and five-attempt conflict retries. Capture the source once and rebuild the target scan and writers on retry, including structured row-ID existence conflicts. - Preserve positional inserts excluding the virtual _ROW_ID and source-scoped unqualified NOT MATCHED expressions. Honor SQLOptions and retain Paimon's existing table/action restrictions. - Keep target projection pushdown for partial updates, avoid collisions with internal tracking-column names, and reject duplicate target actions before publication. - Fix the two timestamp procedure tests: $snapshots.commit_time is a local TIMESTAMP, while procedure arguments are epoch milliseconds. Use deterministic snapshot metadata times and assert exact/in-between timestamp boundaries and the selected snapshots. ### Tests - RUST_MIN_STACK=16777216 cargo test --locked -p paimon-datafusion --all-targets (no skip filters). - Timestamp procedure tests with TZ=UTC and TZ=Asia/Shanghai. - Regression coverage for side-effect-free planning/EXPLAIN, direct native-session execution, conflict retries with one source capture, atomic failures, identifier/column mappings, BLOB projection, and internal tracking-column collisions. - cargo fmt --all -- --check - cargo clippy --locked -p paimon-datafusion --all-targets --no-deps -- -D warnings - git diff --check ### API and Format MERGE through SQLContext now returns a native lazy DML DataFrame; collect/execute the plan to perform the write. Storage formats and public Rust signatures are unchanged. The executor materializes the captured source and CoW target batches; large-input memory/performance behavior has not been benchmarked. Alias resolution follows the native planner; existing data-evolution action limitations remain. ### Documentation Update SQL support, MERGE execution and syntax limits, SQLOptions behavior, and the epoch-millisecond contract for timestamp procedures. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
