dragonknightk commented on PR #11178:
URL: https://github.com/apache/paimon/pull/11178#issuecomment-6921725112

   @JingsongLi @Akash3121 Thanks to both of you for the careful reviews. You 
found cases I had not thought through, especially how a pre-merged -U behaves 
once it is flushed to a file.
   
   **The goal.** In Flink a -U means "this row leaves the result". When a 
filter drops the matching +U, as with `WHERE delete = 0`, the -U is effectively 
a delete. `partial-update.remove-record-on-delete` is the option that makes 
deletes take effect, and a -D already removes the row under it, but a lone -U 
is ignored, so the stale row stays. This PR makes a lone -U take effect under 
this option.
   
   **How it works, from the basics.**
   
   - A key's records are merged in segments: the write buffer pre-merges before 
a flush, and minor compactions merge again. Each segment's result is written as 
one record and merged again later, so where the write buffer is flushed should 
not change the row.
   - Under partial-update, a row is assembled from the columns of several 
writers, so a -U must remove the row from reads without wiping the columns that 
others wrote.
   - A -D result cannot carry both facts, because a later merge re-initializes 
the row when it sees a -D. That is the buffer-boundary data loss you both 
reproduced, and it is why the previous revision was wrong.
   - So a -U is now handled as a retraction. It marks the row removed, keeps 
the merged fields, and stays UPDATE_BEFORE in the merge result, so a read drops 
it like a DELETE and a later merge still sees a retraction. A -U also applies 
its non-null fields, so the rebuilt row is the same however the records are 
split.
   
   **What this gives.**
   
   - A key whose last message is a lone -U reads no row, which fixes issue 6862 
for this option. When the -U and +U carry different partition values, the row 
is dropped from the old partition.
   - A later +I or +U restores the row together with the columns of other 
writers.
   - The result no longer depends on write-buffer boundaries. Your probe is now 
a table-level test with `write-only`: another writer commits `b = 20`, then 
`+I(a = 10)`, `-U(a = 10)` and `+U(a = 11)` go into one commit. In one buffer, 
flushed after the -U, or flushed after each record, all three read `a = 11, b = 
20`.
   - The known costs are the same as for -D on master and are written in the 
docs: a retracted value comes back when another writer restores the row, and a 
compaction that drops the removed row also drops the columns of other writers. 
This affects -D on master in the same way, and this PR leaves it unchanged.
   - `partial-update.remove-record-on-sequence-group` is back to the behavior 
on master. With an aggregated field, neither result kind keeps the sum right 
after a flush (expected 16, UPDATE_BEFORE gives -6, DELETE gives NULL), so I 
would like to discuss it separately.
   
   **Native readers.** Since this changes how a lone -U in committed files is 
read, Java and paimon-rust have to agree, or the same table reads differently. 
My suggestion is to align paimon-rust with the same rule, so that a lone -U 
removes the row under this option and both sides read the same result. 
@JingsongLi, does that direction work for you?
   
   Thanks again ʕ•ᴥ•ʔ
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to