wangyong9999 commented on code in PR #525:
URL: https://github.com/apache/paimon-cpp/pull/525#discussion_r4508977537
##########
src/paimon/core/utils/primary_key_table_utils.cpp:
##########
@@ -103,13 +103,20 @@ Status
PrimaryKeyTableUtils::ValidateRealtimeOptions(const CoreOptions& options,
if (options.GetBucket() <= 0) {
return Status::NotImplemented("PK realtime requires fixed buckets");
}
- if (options.GetMergeEngine() != MergeEngine::DEDUPLICATE) {
- return Status::NotImplemented("PK realtime supports only the
DEDUPLICATE merge engine");
+ if (options.GetMergeEngine() != MergeEngine::DEDUPLICATE &&
+ options.GetMergeEngine() != MergeEngine::PARTIAL_UPDATE) {
Review Comment:
With partial-update and DVs disabled, the realtime disk scan can still drop
the base row by value stats.
`DataTableBatchScan` sets no level filter in this case, so
`WholeBucketFilterEnabled()` is true for any non-PK predicate. When a bucket
has one file, or all its files are on one non-zero level,
`FilterWholeBucketByStats` takes the per-file branch and drops files whose
value stats don't match. That is fine for disk-only reads, but here memory rows
still need those files to complete the merge. `FilterWholeBucketAllFiles`
already skips partial-update; the per-file branch doesn't.
```text
Disk (one file): (id=1, a=10, b=20)
Memory: (id=1, a=30, b=null)
Query: WHERE a = 30
Expected: (id=1, a=30, b=20)
Actual: (id=1, a=30, b=null)
```
Deduplicate is not affected, since the memory row replaces the whole row.
Suggest planning disk files with key predicates only for realtime
partial-update (e.g. skip value-stats pruning in `DataTableBatchScan` when
`realtime_pk_scan` is set and the engine is partial-update), and adding a test
with a value predicate like the one above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]