dragonknightk commented on PR #11178:
URL: https://github.com/apache/paimon/pull/11178#issuecomment-6912961063

   @Akash3121, you were right about the pre-merge boundary. While reworking it, 
I found that keeping the merged row on a -U cannot be made consistent, so I 
changed the approach.
   
   With `partial-update.remove-record-on-delete` or 
`partial-update.remove-record-on-sequence-group`, a -U is now handled exactly 
like a -D. It re-initializes the row from its own values and removes the row. A 
lone -U means the row left the upstream result set, for example when a Flink 
filter drops the +U. So a row-level delete matches what these options are for.
   
   Why not keep the merged row: the write buffer and minor compactions write 
the merged result as one record, and a later merge reads it as a raw -U or -D. 
With a summed field in another sequence group, `b = 5` committed, then `+I(b = 
11)` and a -U of the other group, then a +U of that group:
   
   - Keeping the merged row and writing it as -D gave b = 11 in one write 
buffer and b = 16 in separate commits.
   - Keeping the merged row and writing it as -U gave b = -6 in one write 
buffer and b = 16 in separate commits.
   - Treating -U as -D gives b = NULL in both cases.
   
   Columns that other writers wrote before the -U may not be kept, the same as 
with -D; it depends on whether the -U and the later write are merged before 
being written. For example, after `+I(1, 10, null)` and `+I(1, null, 20)`, a -U 
and +U in one write buffer give `(1, 11, 20)`, and in separate commits give 
`(1, 11, null)`. A -D followed by a +I already behaves this way, so I left that 
limit out of this PR. The docs now describe this limit.
   
   On your other two points:
   
   - You suggested checking that `b = 11, seq1 = 2` is kept across the seq2 -U 
and +U. `testPartialUpdateRemoveRecordOnSequenceGroup` now passes null for `b` 
and `seq1` in the seq2 steps and asserts the opposite on purpose: after the 
seq2 -U, the +U row has `seq1 = NULL`, because the -U removed the row like a 
-D. A separate step checks that a -U of `seq1` only clears `b` and keeps the 
row.
   - I removed `testRemoveRecordOnDeleteWithOnlyUpdateBefore`. With -U now 
handled like -D, that case goes through the same path as the other -U tests, 
and `!meetInsert` already gave DELETE there. The table-level tests cover a lone 
-U on read instead.
   
   The new regression tests write the same +I and -U for one key in a single 
write buffer and for another key in separate commits, and assert that the 
following +U gives the same row for both keys, under both options. Thanks for 
the detailed review.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to