fastio opened a new pull request, #499:
URL: https://github.com/apache/paimon-cpp/pull/499

   ### Purpose
   
   Linked issue: close #493
   
   `NewDataTableScan` rejected every scan of a primary-key table with `bucket = 
-1`
   (`do not support pk table bucket=-1 in scan process`). Such tables (dynamic 
bucket, written by
   Paimon Java from Flink or Spark) can be read in batch mode: batch planning 
only uses the buckets
   recorded in the manifest entries and does not need the dynamic bucket 
assigner. Since #494,
   deletion vectors of these tables are combined per (partition, bucket), as 
Java does for
   `BucketMode.HASH_DYNAMIC`.
   
   This change keeps rejecting dynamic bucket primary-key tables only in 
streaming mode, and allows
   batch scans. The streaming error message now says `streaming scan process`.
   
   Writing dynamic bucket primary-key tables and streaming reads stay out of 
scope.
   
   ### Tests
   
   - 
`ScanAndReadDynamicBucketInteTest.TestBatchReadPkDynamicBucketWithDeletionVectors`:
 reads a
     table written by Spark 3.5.3 with Paimon 1.3.2 
(`test/test_data/parquet/pk_dynamic_bucket_dv.db`,
     `bucket = -1`, `deletion-vectors.enabled = true`, 
`dynamic-bucket.target-row-num = 2`, see its
     README). Keys are spread over several buckets, each with its own deletion 
vector index. The test
     reads three snapshots (insert, upsert, delete), checks that the plan has 
more than one split,
     and compares the rows with what Spark returns.
   - `ScanInteTest.TestScanPKWithInvalidOptions`: streaming scans of dynamic 
bucket primary-key
     tables are still rejected, with the updated message.
   
   Without the change in `table.cpp`, the new test fails with
   `do not support pk table bucket=-1 in scan process`.
   
   Verified on Ubuntu 24.04 (aarch64), GCC 13, Debug: `paimon-scan-inte-test` 
(87 tests),
   `paimon-scan-and-read-inte-test` (512 tests) and `paimon-core-test` (2536 
tests) pass.
   
   ### API and Format
   
   No public API change. No storage format or protocol change.
   
   ### Documentation
   
   No.
   
   ### Generative AI tooling
   
   Generated-by: Claude Code (Claude Opus 5.5)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to