HaHaJeff opened a new issue, #524:
URL: https://github.com/apache/paimon-cpp/issues/524
## Summary
Partial-update merging can produce incorrect results when intermediate
merged records are merged again: a delete boundary can be lost, or a field
value can lose its association with its originating sequence number.
Both cases below were reproduced with the ordinary Writer on baseline
`becc48a51c8a740bf36f201df5fb23fbb2d2b0f1`, without production-code changes.
Only diagnostic tests were added. These existing defects are tracked
independently from the realtime partial-update feature.
## Reproduction setup
The examples below are **pseudocode**, not runnable Paimon APIs. Each
`write_batch` represents one input batch; record order, batch boundaries, and
commit order are intentional. `commit(writer, n)` represents ordinary Write →
PrepareCommit(true, n) → Commit. `read_all()` performs a fresh scan/plan/read
of committed data; `reopen_and_read_all()` destroys and recreates the
writer/commit context before reading again. No realtime context, transport
offsets, or progress APIs are involved.
Common options for both cases:
```text
file.format=parquet
file-system=local
bucket=1
bucket-key=id
target-file-size=1048576
merge-engine=partial-update
read.batch-size=1
num-sorted-run.compaction-trigger=100
num-sorted-run.stop-trigger=200
```
The ordinary Writer retained its normal MergeTreeCompactManager; compaction
was not triggered. This was not a Noop-compaction control, and `write-only` was
not explicitly set.
## Case 1: delete followed by reinsertion in the same batch retains an old
field
Schema: `id BIGINT, a BIGINT, b BIGINT`, primary key `id`.
Additional option:
```text
partial-update.remove-record-on-delete=true
```
```text
writer.write_batch([INSERT(id=1, a=10, b=20)])
commit(writer, 0)
writer.write_batch([
DELETE(id=1, a=null, b=null),
UPDATE_AFTER(id=1, a=30, b=null)
])
commit(writer, 1)
read_all()
# expected: [(id=1, a=30, b=null)]
# actual: [(id=1, a=30, b=20)]
reopen_and_read_all()
# actual: [(id=1, a=30, b=20)]
```
The delete should reset the record. The subsequent partial update must not
resurrect `b=20` from before the delete.
**Passing control:** with the same schema, options, seed, and operation
order, committing the DELETE separately as commit 1 and then the reinsertion
separately as commit 2 passes. Commit 1 and its reopened read are empty; commit
2 and its reopened read return `[(id=1, a=30, b=null)]`.
Source analysis suggests that an intermediate summary loses the preceding
delete boundary, allowing a later null field to retain an older value. This is
a hypothesis from source analysis, not an instrumented observation of
intermediate records.
## Case 2: sum is correct but last_non_null_value comes from the wrong
sequence
Schema: `id BIGINT, seq BIGINT, total BIGINT, last STRING`, primary key `id`.
Additional options:
```text
fields.seq.sequence-group=total,last
fields.total.aggregate-function=sum
fields.last.aggregate-function=last_non_null_value
```
```text
writer.write_batch([
INSERT(id=1, seq=9, total=6, last="nine"),
INSERT(id=1, seq=10, total=4, last="ten")
])
commit(writer, 0)
# committed seed: [(id=1, seq=10, total=10, last="ten")]
writer.write_batch([
INSERT(id=1, seq=30, total=30, last="thirty"),
INSERT(id=1, seq=20, total=20, last="twenty")
])
writer.write_batch([
UPDATE_AFTER(id=1, seq=25, total=5, last="middle"),
UPDATE_BEFORE(id=1, seq=20, total=20, last=null),
UPDATE_AFTER(id=1, seq=30, total=2, last=null)
])
commit(writer, 1)
read_all()
# expected: [(id=1, seq=30, total=47, last="thirty")]
# actual: [(id=1, seq=30, total=47, last="middle")]
reopen_and_read_all()
# actual: [(id=1, seq=30, total=47, last="middle")]
```
There is no commit or explicit ordinary Seal/spill between the two batches
preceding commit 1; the ordinary reproduction uses one flush per commit. The
sum remains correct at 47, but `last` should retain `"thirty"` from sequence 30
rather than `"middle"` from sequence 25.
Source analysis suggests that staged merging loses the association between a
field value and its originating sequence: a summary may associate `"middle"`
with sequence 30 despite its origin at sequence 25. The original run did not
instrument intermediate stages, so the precise mechanism still needs
confirmation.
## Evidence and scope
The original test patches, exact inputs, logs, JUnit results, and passing
separate-commit control are preserved locally. They have not been uploaded as
public attachments; no external evidence links are claimed here. Expected
results were preserved without weakening assertions or replacing them with the
observed incorrect output.
The realtime feature reuses the same merge implementation and may inherit
these limitations. These examples were not independently reproduced on the
current realtime feature commit `2e2f194ccdab76831b2988a9b749f5d8c299d8f7`;
this issue does not claim that realtime introduced these defects or that the
feature fixes them. Separate hidden sequence-group projection/deletion
dependency problems are outside the two reproductions documented here.
For a fixed operation order and configuration, expected logical results must
remain invariant under changes to batch or intermediate-merge boundaries. An
independent fix should preserve the expectations above, retain the passing
separate-commit control, and verify ordinary committed and reopened reads.
Realtime coverage should be established separately against an explicitly
identified source revision and its observable stages.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]