dragonknightk commented on PR #11178: URL: https://github.com/apache/paimon/pull/11178#issuecomment-6912961063
@Akash3121, you were right about the pre-merge boundary. While reworking it, I found that keeping the merged row on a -U cannot be made consistent, so I changed the approach. With `partial-update.remove-record-on-delete` or `partial-update.remove-record-on-sequence-group`, a -U is now handled exactly like a -D. It re-initializes the row from its own values and removes the row. A lone -U means the row left the upstream result set, for example when a Flink filter drops the +U. So a row-level delete matches what these options are for. Why not keep the merged row: the write buffer and minor compactions write the merged result as one record, and a later merge reads it as a raw -U or -D. With a summed field in another sequence group, `b = 5` committed, then `+I(b = 11)` and a -U of the other group, then a +U of that group: - Keeping the merged row and writing it as -D gave b = 11 in one write buffer and b = 16 in separate commits. - Keeping the merged row and writing it as -U gave b = -6 in one write buffer and b = 16 in separate commits. - Treating -U as -D gives b = NULL in both cases. Columns that other writers wrote before the -U may not be kept, the same as with -D; it depends on whether the -U and the later write are merged before being written. For example, after `+I(1, 10, null)` and `+I(1, null, 20)`, a -U and +U in one write buffer give `(1, 11, 20)`, and in separate commits give `(1, 11, null)`. A -D followed by a +I already behaves this way, so I left that limit out of this PR. The docs now describe this limit. On your other two points: - You suggested checking that `b = 11, seq1 = 2` is kept across the seq2 -U and +U. `testPartialUpdateRemoveRecordOnSequenceGroup` now passes null for `b` and `seq1` in the seq2 steps and asserts the opposite on purpose: after the seq2 -U, the +U row has `seq1 = NULL`, because the -U removed the row like a -D. A separate step checks that a -U of `seq1` only clears `b` and keeps the row. - I removed `testRemoveRecordOnDeleteWithOnlyUpdateBefore`. With -U now handled like -D, that case goes through the same path as the other -U tests, and `!meetInsert` already gave DELETE there. The table-level tests cover a lone -U on read instead. The new regression tests write the same +I and -U for one key in a single write buffer and for another key in separate commits, and assert that the following +U gives the same row for both keys, under both options. Thanks for the detailed review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
