wangyong9999 opened a new issue, #298: URL: https://github.com/apache/paimon-cpp/issues/298
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon-cpp/issues) and found nothing similar. ### Paimon-cpp version main (ded7cfb8) ### Minimal reproduce step 1. Create a partitioned primary key table with two buckets and write one partition, for example with Flink SQL: ```sql CREATE TABLE t (dt STRING, k STRING, v STRING, PRIMARY KEY (dt, k) NOT ENFORCED) PARTITIONED BY (dt) WITH ('bucket' = '2'); INSERT INTO t VALUES ('p1', 'k01', 'a'), ('p1', 'k02', 'b'), ...; -- 20 rows ``` 2. Raise the bucket number and write a second partition. Paimon applies the new number to partitions written afterwards only, so `p1` keeps two buckets (its manifest entries carry `total_buckets = 2`) while `p2` gets four: ```sql ALTER TABLE t SET ('bucket' = '4'); INSERT INTO t VALUES ('p2', 'k01', 'a'), ...; -- 20 rows ``` 3. Read single rows of `p1` through paimon-cpp with an equality predicate on every primary key (`dt = 'p1' AND k = 'k01'`), i.e. `ScanContextBuilder::SetPredicate` followed by `TableScan::CreatePlan` and `TableRead`. ### What doesn't meet your expectations? 12 of the 20 `p1` keys come back empty: the plan has no split for them, so the read returns no row and no error. The same statements in Flink return all 20 rows. The keys that still work are the ones whose `hash % 4` happens to equal `hash % 2`. The scan turns the key predicate into exactly one bucket computed with the bucket number of the table schema: `KeyValueFileStoreScan::SplitAndSetKeyValueFilter` calls `BucketSelectConverter::Convert(..., core_options_.GetBucket(), ...)` and stores the result with `SetBucketFilterIfAbsent`. From then on `FilterManifestEntry` keeps entries with `entry.Bucket() == bucket_filter_`, `FilterManifestFileMeta` prunes manifests by that bucket, and the snapshot live manifest entry cache and `ManifestFile::ReadBucketEntries` read that bucket only. `ManifestEntry::TotalBuckets()` is never consulted. Java evaluates the same predicate per entry: `KeyValueFileStoreScan#withTotalAwareBucketFilter` feeds `BucketSelector.test(bucket, numBucket)` with each entry's own `totalBuckets`, so a table whose partitions were written with different bucket numbers still reads correctly. The unpartitioned variant shows up as well: a long-lived reader that captured the table schema before `ALTER TABLE ... SET ('bucket' = '4')` plus `INSERT OVERWRITE` keeps selecting buckets with the old number and misses about half of the keys after the rewrite. ### Anything else? The bucket number a partition was written with is always the `bucket` option of one of the table's schema versions (a writer with a different number cannot commit into an existing partition). Computing one candidate bucket per distinct bucket number found in the schema history, and checking `entry.Bucket() == bucket(key, entry.TotalBuckets())` per entry, keeps tables with a single bucket number on the current single-bucket path while making the others read like Java. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
