fastio opened a new issue, #493:
URL: https://github.com/apache/paimon-cpp/issues/493

   ### Search before asking
   
   - [x] I searched in the issues and found nothing similar.
   
   ### Motivation
   
   Primary-key tables created by Paimon Java with `bucket = -1` 
(`HASH_DYNAMIC`, or `KEY_DYNAMIC`
   when the primary key does not contain all partition keys) are common in 
Flink and Spark
   pipelines. Paimon C++ currently rejects every scan of such a table:
   
   ```
   do not support pk table bucket=-1 in scan process
   ```
   
   Batch planning does not need the dynamic bucket assigner. It only uses the 
buckets recorded in
   the manifest entries, so a batch read can be supported without implementing 
dynamic bucket
   writes.
   
   One prerequisite is the deletion vector index. `IndexManifestFile` and 
`IndexManifestFileHandler`
   take the configured `bucket` option (`int32_t`) and treat `-1` as 
`BUCKET_UNAWARE`, which then
   combines deletion vectors by data file name. A dynamic bucket primary-key 
table also has
   `bucket = -1`, but keeps one deletion vector index per (partition, bucket), 
like Java's
   `BucketMode.HASH_DYNAMIC`. The existing comment in 
`GetIndexManifestFileCombine` notes this is
   only safe while dynamic bucket primary-key tables cannot be written, and 
that lifting the
   restriction requires passing the resolved `BucketMode`.
   
   ### Solution
   
   1. Pass the resolved `BucketMode` (`ResolveBucketMode(options.GetBucket(), 
table_schema)`) to
      `IndexManifestFile` / `IndexManifestFileHandler` instead of the raw 
`bucket` option, and only
      use the global deletion vector combiner for `BUCKET_UNAWARE`. No behavior 
change for tables
      that can be written today.
   2. In `NewDataTableScan`, keep rejecting dynamic bucket primary-key tables 
only in streaming
      mode, and allow batch scans.
   3. Add an integration test reading a Java-written dynamic bucket primary-key 
table with
      deletion vectors spread over several buckets (insert, upsert, delete 
snapshots).
   
   Writing dynamic bucket primary-key tables and streaming reads stay out of 
scope.
   
   ### Anything else?
   
   I will split this into two PRs: the `BucketMode` change first, then the 
batch read with the
   test fixture.
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to