houlinchn opened a new issue, #11273: URL: https://github.com/apache/paimon/issues/11273
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version master (`df565c64f7`). First observed in production on 1.4.1. The behavior was introduced by #7645; release-1.4, release-2.0 and release-2.1 have the same code. ### Compute Engine Flink (also reproducible with the Java API; the bug is in paimon-core). ### Minimal reproduce step Primary-key table with `merge-engine=partial-update`, a sequence group and a `sum` aggregate: ``` 'bucket' = '1', 'merge-engine' = 'partial-update', 'fields.seq.sequence-group' = 'amount', 'fields.amount.aggregate-function' = 'sum' ``` 1. Commit `+I(id=1, amount=90, seq=1)`. Reading the table returns `amount = 90`. 2. Commit `-U(id=1, amount=90, seq=2)` and `+U(id=1, amount=90, seq=3)` in the same commit, so both records are merged by the same write-buffer flush. 3. Read the table. This is the normal pattern for a Flink job that consumes a changelog (for example a `lookup` changelog of an upstream partial-update table) and sinks into a `sum` column: every update arrives as an adjacent `-U/+U` pair. ### What doesn't meet your expectations? Expected `amount = 90` (the pair contributes `-90 + 90 = 0`). Actual `amount = 180`. The retraction is cancelled whenever it is the first record of a key in a merge: 1. `PartialUpdateMergeFunction.add` calls `initRow(row, kv.value())` for the first retract record. 2. Since #7645, `initRow` copies every field, including nullable aggregate fields, so the accumulator starts at `90`. 3. `retractWithSequenceGroup` then computes `sum.retract(90, 90) = 0`, and the `+U` adds `90`. 4. The flushed L0 delta is `90` instead of `0`. Compaction with the existing `90` produces `180`. Before #7645, `initRow` only filled NOT NULL fields, so the accumulator started at `null` and `sum.retract(null, 90) = -90`, which is correct. With `ignore-delete=true` the same seeding happens: the ignored retract fills the row, and the next positive record aggregates on top of it (`agg(90, 90) = 180`). Writes filter retractions in this mode, but historical delete records in data files still reach the merge function, as the comment in `add` notes. ### Anything else? Related: - #7645 changed `initRow` to copy nullable fields. - #10963 reports the same seeding for `AggregateMergeFunction` with `aggregation.remove-record-on-delete` (DELETE, then INSERT). That fix does not cover `PartialUpdateMergeFunction`. - #6862 / #11178 handle a lone `UPDATE_BEFORE` with `partial-update.remove-record-on-delete`. This issue is about a leading retraction followed by records of the same key, with sequence groups. Proposed fix: when the first record of a merge is a retraction and sequence groups are enabled, do not copy the retraction's values into retractable aggregate fields, so the aggregate starts from `null`. Keep the full-row `initRow` for the whole-row delete paths (`remove-record-on-delete`, `remove-record-on-sequence-group`) that #7645 fixed. With `ignore-delete`, keep the retract payload for a DELETE-only result, but reset the row before the first positive record. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
