JingsongLi opened a new pull request, #1068: URL: https://github.com/apache/paimon-rust/pull/1068
### Purpose Linked issue: closes #1067 Provide prepared MERGE operations for PyPaimon's existing single-process batch and stream update API. SQLContext's SQL MERGE publishes a snapshot directly; TableUpdate needs to return commit messages for the caller's matching committer. ### Brief change log - Add core `TableUpdate::merge_into` with materialized Arrow input or self-merge on `_ROW_ID`, column references/literals, ordered matched UPDATE/DELETE and unmatched INSERT clauses. - Pin one target snapshot and reuse row-id update, deletion-vector and append writers. Core owns matching, cardinality checks, clause selection and staging. - Follow Paimon Spark's source-match cardinality placement and sole unconditional DELETE exemption. NULL keys do not match; NULL action conditions fall through to later clauses. - Add a Rust DataFusion condition adapter: compile ordinary expressions once, execute complete plans for scalar/IN/EXISTS subqueries, preserve input row order, and collect correlated dependencies for self-merge projection. - Add thin batch/stream Python wrappers and API stubs. Reuse shared update-assignment conversion, including deferred scalar errors and complete chunked output. - Retain earlier prepared files if a later action fails, including failures that cannot return partial messages. Publication remains an explicit caller operation. ### Tests - `cargo test --locked -p paimon --lib --features fulltext,vortex`: 3,891 passed, 6 ignored. - Core MERGE: 6 focused tests covering row IDs, NULL keys, ambiguous source matches before conditions, unconditional DELETE, self-merge, chunked assignments and inconsistent source schemas. - DataFusion adapter: 3 focused tests covering projected/reordered schemas, SQL subqueries, correlated references and current-thread Tokio evaluation. - Python binding suite: 333 passed, using the local native build and precomputed binding warehouse. - PyPaimon native update/MERGE regression suite: 994 passed, 2 skipped, 42 subtests passed; planning/read/write/update/commit gates enabled, including 28 native MERGE operations. - Clippy 1.98.0: all targets/workspace with `fulltext,vortex` and `-D warnings`; rustfmt and changed binding test lint passed. ### API and Format Additive core `MergeSource`, `MergeCondition`, `MergeAssignment`, `WhenMatched` and `WhenNotMatched` types and batch/stream Python `merge_into` methods. Writes use existing Parquet delta files, deletion vectors and append commit messages. ### Documentation Document prepared MERGE, Spark cardinality semantics and prepared-file retention in the Python binding README and API stubs. The paired PyPaimon PR enables native dispatch for its existing MERGE API. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
