TheR1sing3un opened a new issue, #373:
URL: https://github.com/apache/paimon-rust/issues/373
### Bug description
`TableScan::plan_snapshot` bin-packs the files of a (partition, bucket)
purely
by size (`split_for_batch`), and the PK read path builds one sort-merge
reader
per `DataSplit`. When files holding versions of the same primary key land in
different splits, each split emits its own version of the key — SELECT
returns
multiple rows where it should return one merged row.
### Reproduce
```sql
CREATE TABLE t (id INT NOT NULL, value INT, PRIMARY KEY (id)) WITH (
'bucket' = '1',
'source.split.target-size' = '1b',
'source.split.open-file-cost' = '1b'
);
INSERT INTO t VALUES (1, 10);
INSERT INTO t VALUES (1, 20);
INSERT INTO t VALUES (1, 30);
SELECT * FROM t; -- returns 3 rows; expected 1 row (1, 30)
```
### Affected
All PK merge engines on the non-data-evolution path (`deduplicate`,
`partial-update`; `aggregation` once #340 lands). A second instance of the
same root cause: `read_pk` raw-reads splits that contain no level-0 files,
but compacted files on different levels can still overlap on key range
(e.g. written by Java/Spark compaction) and need merging.
### Fix direction
Port Java's `MergeTreeSplitGenerator` / `IntervalPartition`: group a bucket's
files into key-range-overlap sections, bin-pack whole sections into splits,
and make the read-side kv/raw dispatch overlap-aware. Deletion-vector and
first-row tables keep the raw fast path (stale rows are masked by DVs /
level-0 is skipped), mirroring Java's fast path.
First reported by @JingsongLi in
https://github.com/apache/paimon-rust/pull/340#discussion_r3384978245
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
