kosiew commented on code in PR #25811:
URL: https://github.com/apache/datafusion/pull/25811#discussion_r4583037017


##########
datafusion/datasource-parquet/src/opener/mod.rs:
##########
@@ -1595,53 +1551,28 @@ impl RowGroupsPrunedParquetOpen {
                             continue;
                         }
                     };
-                    row_group_filters.insert(
-                        column_name,
-                        bf,
-                        *physical_type,
-                        *type_length,
-                    );
+                    any_filter_loaded = true;
+
+                    if self.row_groups.prune_row_group_with_bloom_filter(

Review Comment:
   @haohuijin, could you assess whether interleaving Bloom filter reads and 
checks poses a concern for your production workloads, particularly with many 
predicate columns and little early rejection? Reads were already sequential, 
but checks now delay subsequent requests. Your assessment would help resolve 
the remaining scheduling concern; the current evidence does not demonstrate a 
regression.



##########
datafusion/datasource-parquet/src/opener/mod.rs:
##########
@@ -5229,6 +5160,402 @@ mod test {
         assert_eq!(matched, 0);
     }
 
+    #[tokio::test]
+    async fn 
incremental_bloom_filter_pruning_short_circuits_remaining_columns() {

Review Comment:
   Non-blocking: the column-local checks eliminate repeated full-predicate 
evaluation, but a deterministic surviving-workload probe-count regression would 
strengthen coverage. Consider testing N surviving single-literal columns and 
asserting exactly N membership checks, independently of timings and I/O counts.



##########
datafusion/datasource-parquet/src/row_group_filter.rs:
##########
@@ -493,12 +496,78 @@ impl RowGroupAccessPlanFilter {
 
             if prune_group {
                 metrics.row_groups_pruned_bloom_filter.add_pruned(1);
-                self.access_plan.skip(idx)
+                self.access_plan.skip(idx);
             } else {
                 metrics.row_groups_pruned_bloom_filter.add_matched(1);
             }
         }
     }
+
+    /// Extracts column guarantees from a [`PruningPredicate`] that can be 
evaluated against
+    /// Parquet Bloom filters.
+    pub(crate) fn prepare_bloom_column_guarantees(
+        predicate: &PruningPredicate,
+        parquet_schema: &SchemaDescriptor,
+        file_schema: &Schema,
+    ) -> Vec<BloomColumnGuarantee> {
+        predicate
+            .literal_columns()
+            .into_iter()
+            .filter_map(|column_name| {
+                let (column_idx, _) = parquet::arrow::parquet_column(
+                    parquet_schema,
+                    file_schema,
+                    &column_name,
+                )?;
+                let in_guarantees: Vec<HashSet<ScalarValue>> = predicate
+                    .literal_guarantees()
+                    .iter()
+                    .filter(|lg| {
+                        lg.column.name == column_name && lg.guarantee == 
Guarantee::In
+                    })
+                    .map(|lg| lg.literals.clone())
+                    .collect();
+                Some(BloomColumnGuarantee {
+                    column_idx,
+                    physical_type: 
parquet_schema.column(column_idx).physical_type(),
+                    type_length: 
parquet_schema.column(column_idx).type_length(),
+                    in_guarantees,
+                })
+            })
+            .collect()
+    }
+
+    /// Evaluates whether a loaded column Bloom filter prunes the specified 
row group.
+    ///
+    /// If all values for any IN guarantee on this column are confirmed absent 
from the Bloom
+    /// filter, the row group is marked as skipped in the access plan, the
+    /// `row_groups_pruned_bloom_filter` metric is incremented, and `true` is 
returned to
+    /// indicate early rejection (allowing callers to skip loading remaining 
column filters).
+    pub(crate) fn prune_row_group_with_bloom_filter(

Review Comment:
   Non-blocking: the custom Criterion runner has been removed, but the 
requested SQL benchmark harness replacement is not included. It would be useful 
to retain early-rejection, late-rejection, and surviving-workload cases under 
cheap and high-latency reads in a follow-up benchmark.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to