houlinchn opened a new issue, #11273:
URL: https://github.com/apache/paimon/issues/11273

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/paimon/issues) 
and found nothing similar.
   
   ### Paimon version
   
   master (`df565c64f7`). First observed in production on 1.4.1. The behavior 
was introduced by #7645; release-1.4, release-2.0 and release-2.1 have the same 
code.
   
   ### Compute Engine
   
   Flink (also reproducible with the Java API; the bug is in paimon-core).
   
   ### Minimal reproduce step
   
   Primary-key table with `merge-engine=partial-update`, a sequence group and a 
`sum` aggregate:
   
   ```
   'bucket' = '1',
   'merge-engine' = 'partial-update',
   'fields.seq.sequence-group' = 'amount',
   'fields.amount.aggregate-function' = 'sum'
   ```
   
   1. Commit `+I(id=1, amount=90, seq=1)`. Reading the table returns `amount = 
90`.
   2. Commit `-U(id=1, amount=90, seq=2)` and `+U(id=1, amount=90, seq=3)` in 
the same commit, so both records are merged by the same write-buffer flush.
   3. Read the table.
   
   This is the normal pattern for a Flink job that consumes a changelog (for 
example a `lookup` changelog of an upstream partial-update table) and sinks 
into a `sum` column: every update arrives as an adjacent `-U/+U` pair.
   
   ### What doesn't meet your expectations?
   
   Expected `amount = 90` (the pair contributes `-90 + 90 = 0`). Actual `amount 
= 180`.
   
   The retraction is cancelled whenever it is the first record of a key in a 
merge:
   
   1. `PartialUpdateMergeFunction.add` calls `initRow(row, kv.value())` for the 
first retract record.
   2. Since #7645, `initRow` copies every field, including nullable aggregate 
fields, so the accumulator starts at `90`.
   3. `retractWithSequenceGroup` then computes `sum.retract(90, 90) = 0`, and 
the `+U` adds `90`.
   4. The flushed L0 delta is `90` instead of `0`. Compaction with the existing 
`90` produces `180`.
   
   Before #7645, `initRow` only filled NOT NULL fields, so the accumulator 
started at `null` and `sum.retract(null, 90) = -90`, which is correct.
   
   With `ignore-delete=true` the same seeding happens: the ignored retract 
fills the row, and the next positive record aggregates on top of it (`agg(90, 
90) = 180`). Writes filter retractions in this mode, but historical delete 
records in data files still reach the merge function, as the comment in `add` 
notes.
   
   ### Anything else?
   
   Related:
   - #7645 changed `initRow` to copy nullable fields.
   - #10963 reports the same seeding for `AggregateMergeFunction` with 
`aggregation.remove-record-on-delete` (DELETE, then INSERT). That fix does not 
cover `PartialUpdateMergeFunction`.
   - #6862 / #11178 handle a lone `UPDATE_BEFORE` with 
`partial-update.remove-record-on-delete`. This issue is about a leading 
retraction followed by records of the same key, with sequence groups.
   
   Proposed fix: when the first record of a merge is a retraction and sequence 
groups are enabled, do not copy the retraction's values into retractable 
aggregate fields, so the aggregate starts from `null`. Keep the full-row 
`initRow` for the whole-row delete paths (`remove-record-on-delete`, 
`remove-record-on-sequence-group`) that #7645 fixed. With `ignore-delete`, keep 
the retract payload for a DELETE-only result, but reset the row before the 
first positive record.
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to