HaHaJeff opened a new issue, #524:
URL: https://github.com/apache/paimon-cpp/issues/524

   ## Summary
   
   Partial-update merging can produce incorrect results when intermediate 
merged records are merged again: a delete boundary can be lost, or a field 
value can lose its association with its originating sequence number.
   
   Both cases below were reproduced with the ordinary Writer on baseline 
`becc48a51c8a740bf36f201df5fb23fbb2d2b0f1`, without production-code changes. 
Only diagnostic tests were added. These existing defects are tracked 
independently from the realtime partial-update feature.
   
   ## Reproduction setup
   
   The examples below are **pseudocode**, not runnable Paimon APIs. Each 
`write_batch` represents one input batch; record order, batch boundaries, and 
commit order are intentional. `commit(writer, n)` represents ordinary Write → 
PrepareCommit(true, n) → Commit. `read_all()` performs a fresh scan/plan/read 
of committed data; `reopen_and_read_all()` destroys and recreates the 
writer/commit context before reading again. No realtime context, transport 
offsets, or progress APIs are involved.
   
   Common options for both cases:
   
   ```text
   file.format=parquet
   file-system=local
   bucket=1
   bucket-key=id
   target-file-size=1048576
   merge-engine=partial-update
   read.batch-size=1
   num-sorted-run.compaction-trigger=100
   num-sorted-run.stop-trigger=200
   ```
   
   The ordinary Writer retained its normal MergeTreeCompactManager; compaction 
was not triggered. This was not a Noop-compaction control, and `write-only` was 
not explicitly set.
   
   ## Case 1: delete followed by reinsertion in the same batch retains an old 
field
   
   Schema: `id BIGINT, a BIGINT, b BIGINT`, primary key `id`.
   
   Additional option:
   
   ```text
   partial-update.remove-record-on-delete=true
   ```
   
   ```text
   writer.write_batch([INSERT(id=1, a=10, b=20)])
   commit(writer, 0)
   
   writer.write_batch([
       DELETE(id=1, a=null, b=null),
       UPDATE_AFTER(id=1, a=30, b=null)
   ])
   commit(writer, 1)
   
   read_all()
   # expected: [(id=1, a=30, b=null)]
   # actual:   [(id=1, a=30, b=20)]
   
   reopen_and_read_all()
   # actual:   [(id=1, a=30, b=20)]
   ```
   
   The delete should reset the record. The subsequent partial update must not 
resurrect `b=20` from before the delete.
   
   **Passing control:** with the same schema, options, seed, and operation 
order, committing the DELETE separately as commit 1 and then the reinsertion 
separately as commit 2 passes. Commit 1 and its reopened read are empty; commit 
2 and its reopened read return `[(id=1, a=30, b=null)]`.
   
   Source analysis suggests that an intermediate summary loses the preceding 
delete boundary, allowing a later null field to retain an older value. This is 
a hypothesis from source analysis, not an instrumented observation of 
intermediate records.
   
   ## Case 2: sum is correct but last_non_null_value comes from the wrong 
sequence
   
   Schema: `id BIGINT, seq BIGINT, total BIGINT, last STRING`, primary key `id`.
   
   Additional options:
   
   ```text
   fields.seq.sequence-group=total,last
   fields.total.aggregate-function=sum
   fields.last.aggregate-function=last_non_null_value
   ```
   
   ```text
   writer.write_batch([
       INSERT(id=1, seq=9, total=6, last="nine"),
       INSERT(id=1, seq=10, total=4, last="ten")
   ])
   commit(writer, 0)
   # committed seed: [(id=1, seq=10, total=10, last="ten")]
   
   writer.write_batch([
       INSERT(id=1, seq=30, total=30, last="thirty"),
       INSERT(id=1, seq=20, total=20, last="twenty")
   ])
   writer.write_batch([
       UPDATE_AFTER(id=1, seq=25, total=5, last="middle"),
       UPDATE_BEFORE(id=1, seq=20, total=20, last=null),
       UPDATE_AFTER(id=1, seq=30, total=2, last=null)
   ])
   commit(writer, 1)
   
   read_all()
   # expected: [(id=1, seq=30, total=47, last="thirty")]
   # actual:   [(id=1, seq=30, total=47, last="middle")]
   
   reopen_and_read_all()
   # actual:   [(id=1, seq=30, total=47, last="middle")]
   ```
   
   There is no commit or explicit ordinary Seal/spill between the two batches 
preceding commit 1; the ordinary reproduction uses one flush per commit. The 
sum remains correct at 47, but `last` should retain `"thirty"` from sequence 30 
rather than `"middle"` from sequence 25.
   
   Source analysis suggests that staged merging loses the association between a 
field value and its originating sequence: a summary may associate `"middle"` 
with sequence 30 despite its origin at sequence 25. The original run did not 
instrument intermediate stages, so the precise mechanism still needs 
confirmation.
   
   ## Evidence and scope
   
   The original test patches, exact inputs, logs, JUnit results, and passing 
separate-commit control are preserved locally. They have not been uploaded as 
public attachments; no external evidence links are claimed here. Expected 
results were preserved without weakening assertions or replacing them with the 
observed incorrect output.
   
   The realtime feature reuses the same merge implementation and may inherit 
these limitations. These examples were not independently reproduced on the 
current realtime feature commit `2e2f194ccdab76831b2988a9b749f5d8c299d8f7`; 
this issue does not claim that realtime introduced these defects or that the 
feature fixes them. Separate hidden sequence-group projection/deletion 
dependency problems are outside the two reproductions documented here.
   
   For a fixed operation order and configuration, expected logical results must 
remain invariant under changes to batch or intermediate-merge boundaries. An 
independent fix should preserve the expectations above, retain the passing 
separate-commit control, and verify ordinary committed and reopened reads. 
Realtime coverage should be established separately against an explicitly 
identified source revision and its observable stages.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to