dragonknightk commented on PR #11178: URL: https://github.com/apache/paimon/pull/11178#issuecomment-6921725112
@JingsongLi @Akash3121 Thanks to both of you for the careful reviews. You found cases I had not thought through, especially how a pre-merged -U behaves once it is flushed to a file. **The goal.** In Flink a -U means "this row leaves the result". When a filter drops the matching +U, as with `WHERE delete = 0`, the -U is effectively a delete. `partial-update.remove-record-on-delete` is the option that makes deletes take effect, and a -D already removes the row under it, but a lone -U is ignored, so the stale row stays. This PR makes a lone -U take effect under this option. **How it works, from the basics.** - A key's records are merged in segments: the write buffer pre-merges before a flush, and minor compactions merge again. Each segment's result is written as one record and merged again later, so where the write buffer is flushed should not change the row. - Under partial-update, a row is assembled from the columns of several writers, so a -U must remove the row from reads without wiping the columns that others wrote. - A -D result cannot carry both facts, because a later merge re-initializes the row when it sees a -D. That is the buffer-boundary data loss you both reproduced, and it is why the previous revision was wrong. - So a -U is now handled as a retraction. It marks the row removed, keeps the merged fields, and stays UPDATE_BEFORE in the merge result, so a read drops it like a DELETE and a later merge still sees a retraction. A -U also applies its non-null fields, so the rebuilt row is the same however the records are split. **What this gives.** - A key whose last message is a lone -U reads no row, which fixes issue 6862 for this option. When the -U and +U carry different partition values, the row is dropped from the old partition. - A later +I or +U restores the row together with the columns of other writers. - The result no longer depends on write-buffer boundaries. Your probe is now a table-level test with `write-only`: another writer commits `b = 20`, then `+I(a = 10)`, `-U(a = 10)` and `+U(a = 11)` go into one commit. In one buffer, flushed after the -U, or flushed after each record, all three read `a = 11, b = 20`. - The known costs are the same as for -D on master and are written in the docs: a retracted value comes back when another writer restores the row, and a compaction that drops the removed row also drops the columns of other writers. This affects -D on master in the same way, and this PR leaves it unchanged. - `partial-update.remove-record-on-sequence-group` is back to the behavior on master. With an aggregated field, neither result kind keeps the sum right after a flush (expected 16, UPDATE_BEFORE gives -6, DELETE gives NULL), so I would like to discuss it separately. **Native readers.** Since this changes how a lone -U in committed files is read, Java and paimon-rust have to agree, or the same table reads differently. My suggestion is to align paimon-rust with the same rule, so that a lone -U removes the row under this option and both sides read the same result. @JingsongLi, does that direction work for you? Thanks again ʕ•ᴥ•ʔ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
