SteNicholas opened a new issue, #1443:
URL: https://github.com/apache/paimon-rust/issues/1443

   ### Search before asking
   
   - [x] I searched existing issues and open pull requests. No issue or active 
PR covering this specific scope was found on November 18, 2026.
   
   ### Motivation
   
   DataFusion's Paimon integration supports SQL UPDATE and DELETE on 
deduplicate primary-key tables, but 
[plan_merge](https://github.com/apache/paimon-rust/blob/650d26277b5e5bc656d9667c535d42e00aa4b9aa/crates/integrations/datafusion/src/merge_into.rs#L107-L130)
 still rejects every primary-key target.
   
   Users should be able to apply updates, deletes and inserts from a source in 
one MERGE operation, reusing the existing primary-key writer and commit 
pipeline. Java/Spark already has 
[performMergeForPkTable](https://github.com/apache/paimon/blob/2bf10c194b9213e406b8cf0732b9e947f835cebd/paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/commands/MergeIntoPaimonTable.scala#L87-L108).
   
   Example of the requested behavior, with a Paimon catalog configured:
   
   ```sql
   CREATE TABLE test_db.target (
       id INT NOT NULL,
       v INT,
       PRIMARY KEY (id)
   ) WITH ('bucket' = '1', 'merge-engine' = 'deduplicate');
   
   CREATE TABLE test_db.source (id INT, v INT);
   INSERT INTO test_db.target VALUES (1, 10), (2, 20);
   INSERT INTO test_db.source VALUES (1, 100), (2, NULL), (3, 300);
   
   MERGE INTO test_db.target AS t
   USING test_db.source AS s
   ON t.id = s.id
   WHEN MATCHED AND s.v IS NULL THEN DELETE
   WHEN MATCHED THEN UPDATE SET v = s.v
   WHEN NOT MATCHED THEN INSERT (id, v) VALUES (s.id, s.v);
   ```
   
   Expected target rows are `(1, 100)` and `(3, 300)`.
   
   ### Solution
   
   Start with ordinary primary-key tables using `merge-engine=deduplicate`.
   
   - Support conditional WHEN MATCHED UPDATE/DELETE and WHEN NOT MATCHED 
INSERT, preserving clause order and SQL NULL behavior. Validate target 
configuration and unsupported syntax before staging output.
   - Resolve target and source once and define the captured read view, 
including self-merge. Define duplicate source-to-target match handling 
consistently with Java instead of silently choosing an arbitrary update.
   - Evaluate assignments against the original matched row, preserve unassigned 
fields, and reject primary-key or partition-key changes in the first version.
   - Produce complete changed rows with the appropriate row kinds and use the 
existing write/commit path. Publish all actions from the statement together; do 
not commit UPDATE, DELETE and INSERT branches separately.
   - Preserve configured sequence semantics. An UPDATE/MERGE must not be 
assumed to override an existing or concurrent version with a higher-priority 
sequence.
   - Reuse applicable protections from primary-key UPDATE/DELETE for query 
authorization, rowkind.field, temporary-object shadowing, branch/time-travel 
targets and target replacement.
   - Retry publication with prepared messages rather than rerunning the source 
query. Clean up safely unpublished output on execution failure, and preserve 
files when publication outcome is uncertain. A statement producing no changes 
should not publish a snapshot.
   
   #### Acceptance criteria
   
   - [ ] Mixed update/delete/insert, no matches, empty source, multiple ordered 
conditions, NULL predicates, and unchanged columns.
   - [ ] Composite keys, partitioned tables, multiple stored versions, quoted 
identifiers and source aliases.
   - [ ] Duplicate source matches and self-merge have explicit, tested 
semantics.
   - [ ] Key-changing assignments and unsupported table configurations/clauses 
fail before writes.
   - [ ] Ascending/descending sequence behavior and concurrent upserts remain 
consistent with the writer contract.
   - [ ] One publication covers all action kinds; retries do not rescan, and 
partial failures do not leave incorrectly published actions.
   
   ### Anything else?
   
   Related: #1379 and #1424 added primary-key DELETE and UPDATE. #1424 
explicitly excludes MERGE INTO. Existing append-table and Data Evolution MERGE 
behavior should remain covered by regression tests.
   
   Additional merge engines, key-changing MERGE, WHEN NOT MATCHED BY SOURCE, 
branch-write expansion and a new cross-table transaction model are outside the 
initial scope.
   
   Rust reuse points: 
[primary_key_dml.rs](https://github.com/apache/paimon-rust/blob/650d26277b5e5bc656d9667c535d42e00aa4b9aa/crates/integrations/datafusion/src/primary_key_dml.rs#L42-L115)
 and [MERGE 
planning/execution](https://github.com/apache/paimon-rust/blob/650d26277b5e5bc656d9667c535d42e00aa4b9aa/crates/integrations/datafusion/src/merge_into.rs).
   
   Source baseline: paimon-rust `650d26277b5e5bc656d9667c535d42e00aa4b9aa`, 
compared with Java Paimon `2bf10c194b9213e406b8cf0732b9e947f835cebd`. These 
findings come from source inspection; no new runtime reproduction or 
performance measurement is claimed.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to