dragonknightk opened a new pull request, #1434:
URL: https://github.com/apache/paimon-rust/pull/1434

   ### Purpose
   
   Linked issue: close #1430
   
   With `partial-update.remove-record-on-delete`, a key whose last message is a 
lone UPDATE_BEFORE (-U) now reads no row, as in Java with [apache/paimon PR 
11178](https://github.com/apache/paimon/pull/11178). Before this change the 
Rust merge ignored -U under this option.
   
   The rules follow the Java `PartialUpdateMergeFunction` of that PR:
   
   - A -U marks the row removed and does not re-initialize it. It also writes 
its own non-null fields, as an INSERT does, so a later +I or +U brings the row 
back with the fields of other writers.
   - A merge result removed only by -U is UPDATE_BEFORE, including a key with 
only -U records. If a DELETE re-initialized the row in the same merge, the 
result is DELETE as before.
   - `partial-update.remove-record-on-sequence-group` and sequence groups keep 
their current behavior.
   
   The gap has been reproduced against [apache/paimon PR 
11178](https://github.com/apache/paimon/pull/11178): on a fixed-bucket Parquet 
table with this option, a committed `+I(1, 10, 20)` followed by a lone `-U(1, 
10, NULL)` reads no row in Java and `(1, 10, 20)` through the PyPaimon Native 
reader on pypaimon-rust 0.5.0.
   
   ### Brief change log
   
   - `PartialUpdateMergeFunction` applies the -U rules above and reports 
UPDATE_BEFORE or DELETE through `MergeResult::MaterializedRetractRow`, which 
now carries the row kind.
   - The write pre-merge in `kv_file_writer` stores that row kind instead of 
always DELETE, so merging a pre-merged -U again does not re-initialize the row 
and drop older columns. Reads still drop both kinds.
   - Primary key compaction rejects only compound DELETE states with field 
aggregators, so UPDATE_BEFORE states are compacted instead of failing.
   
   ### Tests
   
   - For every split of the events into commits, each flushed through the write 
pre-merge, another writer's column is kept across a -U and +U pair.
   - A key that ends with a lone -U reads no row until another writer restores 
it with the kept fields. This fails on the old code.
   - A key with only -U records keeps older columns. This fails on the old code 
and when a -U-only pre-merge writes DELETE.
   - With `last_non_null_value`, compaction of +I and -U in one or two commits 
succeeds under the `none` and `lookup` producers, the row stays removed, and a 
later +U restores the other writer's column. It fails without the compaction 
fix and when the merge result is DELETE again. Without a field aggregator the 
same cases pass, and a DELETE result instead drops the other writer's column 
when +I and -U are pre-merged in one commit.
   - Existing expectations that relied on -U being ignored now follow the Java 
rules: a lone -U removes the row on read, and a -U after a DELETE applies its 
fields while the result stays DELETE.
   
   ### API and Format
   
   No public API or file format change.
   
   ### Documentation
   
   `docs/src/compaction.md` documents this rule.
   
   ## AI Disclosure
   
   Human-led contribution. Grok Bot, working for dragonknightk, assisted with 
this change, including the code, tests, and description. All submitted content 
was reviewed by a human before submission.
   
   (๑•̀ㅂ•́)و✧
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to