zhuqi-lucas commented on code in PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#discussion_r3421005710
##########
datafusion/datasource-parquet/src/opener/mod.rs:
##########
@@ -1419,16 +1412,51 @@ impl RowGroupsPrunedParquetOpen {
let files_ranges_pruned_statistics =
prepared.file_metrics.files_ranges_pruned_statistics.clone();
+
+ // Build a dynamic row-group pruner only when all three conditions
hold:
+ // 1) the scan has a predicate (so there is something to evaluate),
+ // 2) the predicate has at least one not-yet-complete dynamic filter
+ // (`DynamicFilterTracking::Watching`) — static or
already-complete
+ // predicates were fully consumed by `prune_by_statistics` at file
+ // open, so re-evaluating them per RG boundary would be wasted
work,
+ // 3) there is at least one pending RG that could be skipped.
+ // The pruner subscribes once to every still-incomplete dynamic filter
+ // via the `DynamicFilterTracker` watch channel (#22460), so detecting
+ // a threshold change is a single atomic load — not a tree walk per
+ // RG check.
+ let row_group_pruner = match (&prepared.predicate, rg_plan.len() > 1) {
+ (Some(predicate), true)
+ if DynamicFilterTracking::classify(predicate)
+ .contains_dynamic_filter() =>
Review Comment:
Good suggestion, addressed in latest PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
