fastio opened a new issue, #493:
URL: https://github.com/apache/paimon-cpp/issues/493
### Search before asking
- [x] I searched in the issues and found nothing similar.
### Motivation
Primary-key tables created by Paimon Java with `bucket = -1`
(`HASH_DYNAMIC`, or `KEY_DYNAMIC`
when the primary key does not contain all partition keys) are common in
Flink and Spark
pipelines. Paimon C++ currently rejects every scan of such a table:
```
do not support pk table bucket=-1 in scan process
```
Batch planning does not need the dynamic bucket assigner. It only uses the
buckets recorded in
the manifest entries, so a batch read can be supported without implementing
dynamic bucket
writes.
One prerequisite is the deletion vector index. `IndexManifestFile` and
`IndexManifestFileHandler`
take the configured `bucket` option (`int32_t`) and treat `-1` as
`BUCKET_UNAWARE`, which then
combines deletion vectors by data file name. A dynamic bucket primary-key
table also has
`bucket = -1`, but keeps one deletion vector index per (partition, bucket),
like Java's
`BucketMode.HASH_DYNAMIC`. The existing comment in
`GetIndexManifestFileCombine` notes this is
only safe while dynamic bucket primary-key tables cannot be written, and
that lifting the
restriction requires passing the resolved `BucketMode`.
### Solution
1. Pass the resolved `BucketMode` (`ResolveBucketMode(options.GetBucket(),
table_schema)`) to
`IndexManifestFile` / `IndexManifestFileHandler` instead of the raw
`bucket` option, and only
use the global deletion vector combiner for `BUCKET_UNAWARE`. No behavior
change for tables
that can be written today.
2. In `NewDataTableScan`, keep rejecting dynamic bucket primary-key tables
only in streaming
mode, and allow batch scans.
3. Add an integration test reading a Java-written dynamic bucket primary-key
table with
deletion vectors spread over several buckets (insert, upsert, delete
snapshots).
Writing dynamic bucket primary-key tables and streaming reads stay out of
scope.
### Anything else?
I will split this into two PRs: the `BucketMode` change first, then the
batch read with the
test fixture.
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]