SteNicholas opened a new issue, #1443: URL: https://github.com/apache/paimon-rust/issues/1443
### Search before asking - [x] I searched existing issues and open pull requests. No issue or active PR covering this specific scope was found on November 18, 2026. ### Motivation DataFusion's Paimon integration supports SQL UPDATE and DELETE on deduplicate primary-key tables, but [plan_merge](https://github.com/apache/paimon-rust/blob/650d26277b5e5bc656d9667c535d42e00aa4b9aa/crates/integrations/datafusion/src/merge_into.rs#L107-L130) still rejects every primary-key target. Users should be able to apply updates, deletes and inserts from a source in one MERGE operation, reusing the existing primary-key writer and commit pipeline. Java/Spark already has [performMergeForPkTable](https://github.com/apache/paimon/blob/2bf10c194b9213e406b8cf0732b9e947f835cebd/paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/commands/MergeIntoPaimonTable.scala#L87-L108). Example of the requested behavior, with a Paimon catalog configured: ```sql CREATE TABLE test_db.target ( id INT NOT NULL, v INT, PRIMARY KEY (id) ) WITH ('bucket' = '1', 'merge-engine' = 'deduplicate'); CREATE TABLE test_db.source (id INT, v INT); INSERT INTO test_db.target VALUES (1, 10), (2, 20); INSERT INTO test_db.source VALUES (1, 100), (2, NULL), (3, 300); MERGE INTO test_db.target AS t USING test_db.source AS s ON t.id = s.id WHEN MATCHED AND s.v IS NULL THEN DELETE WHEN MATCHED THEN UPDATE SET v = s.v WHEN NOT MATCHED THEN INSERT (id, v) VALUES (s.id, s.v); ``` Expected target rows are `(1, 100)` and `(3, 300)`. ### Solution Start with ordinary primary-key tables using `merge-engine=deduplicate`. - Support conditional WHEN MATCHED UPDATE/DELETE and WHEN NOT MATCHED INSERT, preserving clause order and SQL NULL behavior. Validate target configuration and unsupported syntax before staging output. - Resolve target and source once and define the captured read view, including self-merge. Define duplicate source-to-target match handling consistently with Java instead of silently choosing an arbitrary update. - Evaluate assignments against the original matched row, preserve unassigned fields, and reject primary-key or partition-key changes in the first version. - Produce complete changed rows with the appropriate row kinds and use the existing write/commit path. Publish all actions from the statement together; do not commit UPDATE, DELETE and INSERT branches separately. - Preserve configured sequence semantics. An UPDATE/MERGE must not be assumed to override an existing or concurrent version with a higher-priority sequence. - Reuse applicable protections from primary-key UPDATE/DELETE for query authorization, rowkind.field, temporary-object shadowing, branch/time-travel targets and target replacement. - Retry publication with prepared messages rather than rerunning the source query. Clean up safely unpublished output on execution failure, and preserve files when publication outcome is uncertain. A statement producing no changes should not publish a snapshot. #### Acceptance criteria - [ ] Mixed update/delete/insert, no matches, empty source, multiple ordered conditions, NULL predicates, and unchanged columns. - [ ] Composite keys, partitioned tables, multiple stored versions, quoted identifiers and source aliases. - [ ] Duplicate source matches and self-merge have explicit, tested semantics. - [ ] Key-changing assignments and unsupported table configurations/clauses fail before writes. - [ ] Ascending/descending sequence behavior and concurrent upserts remain consistent with the writer contract. - [ ] One publication covers all action kinds; retries do not rescan, and partial failures do not leave incorrectly published actions. ### Anything else? Related: #1379 and #1424 added primary-key DELETE and UPDATE. #1424 explicitly excludes MERGE INTO. Existing append-table and Data Evolution MERGE behavior should remain covered by regression tests. Additional merge engines, key-changing MERGE, WHEN NOT MATCHED BY SOURCE, branch-write expansion and a new cross-table transaction model are outside the initial scope. Rust reuse points: [primary_key_dml.rs](https://github.com/apache/paimon-rust/blob/650d26277b5e5bc656d9667c535d42e00aa4b9aa/crates/integrations/datafusion/src/primary_key_dml.rs#L42-L115) and [MERGE planning/execution](https://github.com/apache/paimon-rust/blob/650d26277b5e5bc656d9667c535d42e00aa4b9aa/crates/integrations/datafusion/src/merge_into.rs). Source baseline: paimon-rust `650d26277b5e5bc656d9667c535d42e00aa4b9aa`, compared with Java Paimon `2bf10c194b9213e406b8cf0732b9e947f835cebd`. These findings come from source inspection; no new runtime reproduction or performance measurement is claimed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
