lxy-9602 commented on code in PR #499:
URL: https://github.com/apache/paimon-cpp/pull/499#discussion_r4462575195
##########
test/test_data/parquet/pk_dynamic_bucket_dv.db/pk_dynamic_bucket_dv/README:
##########
@@ -0,0 +1,21 @@
+Written by Spark 3.5.3 with Paimon 1.3.2.
+
+id:int not null v:string
+primary key: id
+bucket: -1 (dynamic bucket)
+dynamic-bucket.target-row-num = 2
+deletion-vectors.enabled = true
+file.format = parquet
+
+CREATE TABLE dv (id INT, v STRING)
+TBLPROPERTIES ('primary-key' = 'id', 'bucket' = '-1',
'deletion-vectors.enabled' = 'true',
+ 'file.format' = 'parquet', 'dynamic-bucket.target-row-num' =
'2');
+
+INSERT INTO dv VALUES (1, 'a'), (2, 'b'), (3, 'x'), (4, 'y'), (5, 'z');
+-- snapshot 2: (1, a) (2, b) (3, x) (4, y) (5, z)
+INSERT INTO dv VALUES (1, 'c'), (4, 'w');
+-- snapshot 4: (1, c) (2, b) (3, x) (4, w) (5, z)
+DELETE FROM dv WHERE id IN (2, 5);
+-- snapshot 6: (1, c) (3, x) (4, w)
Review Comment:
Thanks for adding the test data and integration tests! However, even though
the test data construction removes `id IN (2, 5)`, the actual dataset doesn’t
produce any DV. This might be because the dataset is too small, and during
compaction the files get merged directly without generating DV. Please adjust
the test-data generation to ensure DV is covered.
Tip: You can try making the `v` field extremely large or increasing the
number of rows to satisfy the DV conditions.
##########
src/paimon/core/table/table.cpp:
##########
@@ -871,9 +871,12 @@ Result<std::unique_ptr<TableScan>> NewDataTableScan(
ValidateRealtimeScan(*table_schema, core_options, *context,
read_optimized));
// validate options
if (core_options.GetBucket() == -1) {
- if (!table_schema->PrimaryKeys().empty()) {
- return Status::NotImplemented(fmt::format(
- "do not support pk table bucket={} in scan process",
core_options.GetBucket()));
+ // Batch planning of dynamic bucket primary key tables only depends on
the buckets
+ // recorded in the manifest, and deletion vectors are looked up per
(partition, bucket).
+ if (!table_schema->PrimaryKeys().empty() &&
context->IsStreamingMode()) {
+ return Status::NotImplemented(
+ fmt::format("do not support pk table bucket={} in streaming
scan process",
+ core_options.GetBucket()));
Review Comment:
Loosening this condition seems to also enable `KEY_DYNAMIC` in the
cross-partition mode. Please confirm whether this scenario should be supported.
If yes, please add the corresponding integration test.
##########
src/paimon/core/table/table.cpp:
##########
@@ -871,9 +871,12 @@ Result<std::unique_ptr<TableScan>> NewDataTableScan(
ValidateRealtimeScan(*table_schema, core_options, *context,
read_optimized));
// validate options
if (core_options.GetBucket() == -1) {
- if (!table_schema->PrimaryKeys().empty()) {
- return Status::NotImplemented(fmt::format(
- "do not support pk table bucket={} in scan process",
core_options.GetBucket()));
+ // Batch planning of dynamic bucket primary key tables only depends on
the buckets
+ // recorded in the manifest, and deletion vectors are looked up per
(partition, bucket).
+ if (!table_schema->PrimaryKeys().empty() &&
context->IsStreamingMode()) {
+ return Status::NotImplemented(
+ fmt::format("do not support pk table bucket={} in streaming
scan process",
+ core_options.GetBucket()));
Review Comment:
Also, it seems `ResolveBucketMode` should distinguish between
`BucketMode.KEY_DYNAMIC` and `BucketMode.HASH_DYNAMIC` based on
`crossPartitionUpdate`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]