dragonknightk opened a new pull request, #11178: URL: https://github.com/apache/paimon/pull/11178
### Purpose Fixes #6862 With `partial-update.remove-record-on-delete`, the merge result now becomes DELETE when the last message of a key is a lone UPDATE_BEFORE (-U). This is a behavior contract change. On -U the merge function keeps the merged row, while DELETE re-initializes the row from its own value. A -U only carries one writer's view, so re-initializing from it would let a following +U drop columns written by other writers. The trigger is a common Flink pattern: `INSERT INTO t SELECT * FROM src WHERE delete = 0`. When a row flips to `delete = 1`, the filter passes the -U and drops the +U. Before this change, the merge function only reacted to DELETE, so the -U was ignored and the stale row survived. Old and new results within one merge for `+I(1, 10, null)` and `+I(1, null, 20)` from two writers, then `-U(1, 10, null)`: ``` old: -U ignored -> +I(1, 10, 20) stale row new: -U -> -D(1, 10, 20) merged row kept in the DELETE new: then +U(1, 11, null) -> +I(1, 11, 20) f2 from the other writer kept ``` - With `partial-update.remove-record-on-sequence-group`, the merge result also becomes DELETE after an accepted -U of a configured group. The group fields are retracted as before instead of re-initialized from the -U. - A later accepted +I or +U restores the row, under the same rules as after a DELETE. - When -U and +U carry different partition values, they reach different keys, so the merge result in the old partition becomes DELETE. - `aggregation.remove-record-on-delete` is unchanged. Its docs define -U as a per-field retraction, so changing it is outside issue 6862 and needs its own discussion. The root cause was confirmed in the review of #6844, which was closed without merging after a rebase request. ### Tests - Without sequence groups, the merge result becomes DELETE after a lone -U and carries the merged row. A following +U in the same merge restores the row on top of the merged row. - With a configured sequence group, the merge result becomes DELETE after an accepted lone -U. The row kind stays INSERT after an older -U of that group or a -U of another group. An accepted +U restores the row. - At table level, a read after a committed lone -U returns no row for both options. - In a chain table, the merge result becomes DELETE in the partition of a -U whose +U lands in another partition. - With `remove-record-on-delete`, a key that receives only one -U and no prior INSERT merges to DELETE. This already holds on the code before this change, and a test locks it in. Every other new test and each test with a changed expectation fails on the code before this change. ## AI Disclosure Human-led contribution. Grok Bot, working for dragonknightk, assisted with this change, including the code, tests, and description. All submitted content was reviewed by a human before submission. (๑•̀ㅂ•́)و✧ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
