fastio opened a new pull request, #499:
URL: https://github.com/apache/paimon-cpp/pull/499
### Purpose
Linked issue: close #493
`NewDataTableScan` rejected every scan of a primary-key table with `bucket =
-1`
(`do not support pk table bucket=-1 in scan process`). Such tables (dynamic
bucket, written by
Paimon Java from Flink or Spark) can be read in batch mode: batch planning
only uses the buckets
recorded in the manifest entries and does not need the dynamic bucket
assigner. Since #494,
deletion vectors of these tables are combined per (partition, bucket), as
Java does for
`BucketMode.HASH_DYNAMIC`.
This change keeps rejecting dynamic bucket primary-key tables only in
streaming mode, and allows
batch scans. The streaming error message now says `streaming scan process`.
Writing dynamic bucket primary-key tables and streaming reads stay out of
scope.
### Tests
-
`ScanAndReadDynamicBucketInteTest.TestBatchReadPkDynamicBucketWithDeletionVectors`:
reads a
table written by Spark 3.5.3 with Paimon 1.3.2
(`test/test_data/parquet/pk_dynamic_bucket_dv.db`,
`bucket = -1`, `deletion-vectors.enabled = true`,
`dynamic-bucket.target-row-num = 2`, see its
README). Keys are spread over several buckets, each with its own deletion
vector index. The test
reads three snapshots (insert, upsert, delete), checks that the plan has
more than one split,
and compares the rows with what Spark returns.
- `ScanInteTest.TestScanPKWithInvalidOptions`: streaming scans of dynamic
bucket primary-key
tables are still rejected, with the updated message.
Without the change in `table.cpp`, the new test fails with
`do not support pk table bucket=-1 in scan process`.
Verified on Ubuntu 24.04 (aarch64), GCC 13, Debug: `paimon-scan-inte-test`
(87 tests),
`paimon-scan-and-read-inte-test` (512 tests) and `paimon-core-test` (2536
tests) pass.
### API and Format
No public API change. No storage format or protocol change.
### Documentation
No.
### Generative AI tooling
Generated-by: Claude Code (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]