dragonknightk opened a new pull request, #1434: URL: https://github.com/apache/paimon-rust/pull/1434
### Purpose Linked issue: close #1430 With `partial-update.remove-record-on-delete`, a key whose last message is a lone UPDATE_BEFORE (-U) now reads no row, as in Java with [apache/paimon PR 11178](https://github.com/apache/paimon/pull/11178). Before this change the Rust merge ignored -U under this option. The rules follow the Java `PartialUpdateMergeFunction` of that PR: - A -U marks the row removed and does not re-initialize it. It also writes its own non-null fields, as an INSERT does, so a later +I or +U brings the row back with the fields of other writers. - A merge result removed only by -U is UPDATE_BEFORE, including a key with only -U records. If a DELETE re-initialized the row in the same merge, the result is DELETE as before. - `partial-update.remove-record-on-sequence-group` and sequence groups keep their current behavior. The gap has been reproduced against [apache/paimon PR 11178](https://github.com/apache/paimon/pull/11178): on a fixed-bucket Parquet table with this option, a committed `+I(1, 10, 20)` followed by a lone `-U(1, 10, NULL)` reads no row in Java and `(1, 10, 20)` through the PyPaimon Native reader on pypaimon-rust 0.5.0. ### Brief change log - `PartialUpdateMergeFunction` applies the -U rules above and reports UPDATE_BEFORE or DELETE through `MergeResult::MaterializedRetractRow`, which now carries the row kind. - The write pre-merge in `kv_file_writer` stores that row kind instead of always DELETE, so merging a pre-merged -U again does not re-initialize the row and drop older columns. Reads still drop both kinds. - Primary key compaction rejects only compound DELETE states with field aggregators, so UPDATE_BEFORE states are compacted instead of failing. ### Tests - For every split of the events into commits, each flushed through the write pre-merge, another writer's column is kept across a -U and +U pair. - A key that ends with a lone -U reads no row until another writer restores it with the kept fields. This fails on the old code. - A key with only -U records keeps older columns. This fails on the old code and when a -U-only pre-merge writes DELETE. - With `last_non_null_value`, compaction of +I and -U in one or two commits succeeds under the `none` and `lookup` producers, the row stays removed, and a later +U restores the other writer's column. It fails without the compaction fix and when the merge result is DELETE again. Without a field aggregator the same cases pass, and a DELETE result instead drops the other writer's column when +I and -U are pre-merged in one commit. - Existing expectations that relied on -U being ignored now follow the Java rules: a lone -U removes the row on read, and a -U after a DELETE applies its fields while the result stays DELETE. ### API and Format No public API or file format change. ### Documentation `docs/src/compaction.md` documents this rule. ## AI Disclosure Human-led contribution. Grok Bot, working for dragonknightk, assisted with this change, including the code, tests, and description. All submitted content was reviewed by a human before submission. (๑•̀ㅂ•́)و✧ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
