jayzhan211 commented on code in PR #25811:
URL: https://github.com/apache/datafusion/pull/25811#discussion_r4585819320
##########
datafusion/datasource-parquet/src/opener/mod.rs:
##########
@@ -1553,38 +1522,25 @@ impl RowGroupsPrunedParquetOpen {
mem::replace(&mut prepared.async_file_reader,
replacement_reader),
reader_metadata,
);
- let parquet_columns: Vec<(String, usize, Type, i32)> = predicate
- .literal_columns()
- .into_iter()
- .filter_map(|column_name| {
- let parquet_schema = builder.parquet_schema();
- let (column_idx, _) = parquet_column(
- parquet_schema,
- &prepared.physical_file_schema,
- &column_name,
- )?;
- Some((
- column_name,
- column_idx,
- parquet_schema.column(column_idx).physical_type(),
- parquet_schema.column(column_idx).type_length(),
- ))
- })
- .collect();
- for idx in self.row_groups.row_group_indexes() {
- // Statistics have already proved that every row in this group
- // matches the predicate, so its Bloom filters cannot prune it.
+ let parquet_columns =
+ RowGroupAccessPlanFilter::prepare_bloom_column_guarantees(
+ predicate,
+ builder.parquet_schema(),
+ &prepared.physical_file_schema,
+ );
+
+ let row_group_indexes: Vec<usize> =
+ self.row_groups.row_group_indexes().collect();
+ for idx in row_group_indexes {
if self.row_groups.access_plan().is_fully_matched(idx) {
Review Comment:
Optional cleanup: this PR drops the comment that #25854 put on this check.
It's the only place that says why a fully matched row group skips its Bloom
reads.
```suggestion
// Statistics have already proved that every row in this
group
// matches the predicate, so its Bloom filters cannot prune
it.
if self.row_groups.access_plan().is_fully_matched(idx) {
```
##########
datafusion/core/tests/parquet/io/bloom_filter_io.rs:
##########
@@ -80,8 +78,40 @@ async fn test_parquet_bloom_filter_io_pruned() {
Read OffsetIndex for columns 'a', 'b' (rg 0) (offsets 192..213)
Request 4
Read Bloom filter for column 'a' (rg 0) (offsets 98..145)
+ ");
+}
+
+/// Multi-row group test demonstrating interleaved I/O and compute:
+/// Row group 0 is pruned after reading Bloom filter for column `a`, so column
+/// `b`'s Bloom filter is only loaded for row group 1.
+#[tokio::test]
+async fn test_parquet_bloom_filter_io_interleaved_multi_row_group() {
+ let test = bloom_filter_multi_row_group_test().await;
+ // Row group 0: a in [10, 20], b in [100, 200]
+ // Row group 1: a in [30, 40], b in [300, 400]
+ // For query WHERE a = 30 AND b = 300:
+ // - Bloom filter on `a` is read for row groups 0 and 1
+ // - Evaluated against `a = 30`: row group 0 is pruned immediately
+ // - Bloom filter on `b` is subsequently read ONLY for row group 1 (rg 0
is skipped!)
Review Comment:
Optional cleanup: these lines read as if `a`'s filter is fetched for both
row groups before anything is evaluated. The loop actually goes one row group
at a time, and that order is the I/O pattern this test exists to show.
```suggestion
// - Row group 0: `a`'s Bloom filter rules out 30, so the row group is
// pruned and `b`'s Bloom filter is never read
// - Row group 1: `a`'s filter may contain 30, so `b`'s filter is read
next
```
##########
datafusion/datasource-parquet/src/opener/mod.rs:
##########
@@ -1595,53 +1551,28 @@ impl RowGroupsPrunedParquetOpen {
continue;
}
};
- row_group_filters.insert(
- column_name,
- bf,
- *physical_type,
- *type_length,
- );
+ any_filter_loaded = true;
+
+ if self.row_groups.prune_row_group_with_bloom_filter(
Review Comment:
@kosiew the bot run on this PR already covers the wide surviving case:
`slow/survive_all/columns/32` (32 predicate columns, none rejects, 64 row
groups, 200 µs added to every request) takes 2.7 s on both main and the PR.
That run was on 2af7118ef6, whose load loop matches the current head apart from
moving code into helpers. Main already awaits each filter read in turn, so the
only new gap between reads is the check itself. The whole in-memory
`survive_all/columns/32` query, all 2,048 checks included, takes 14.7 ms on the
PR (27.6 ms on main), and that gap does not grow with store latency.
@haohuaijin (the ping above went to `haohuijin`), does that match what you
saw in your PoC, or is there a workload this bench misses?
##########
datafusion/datasource-parquet/src/row_group_filter.rs:
##########
@@ -493,12 +496,83 @@ impl RowGroupAccessPlanFilter {
if prune_group {
metrics.row_groups_pruned_bloom_filter.add_pruned(1);
- self.access_plan.skip(idx)
+ self.access_plan.skip(idx);
} else {
metrics.row_groups_pruned_bloom_filter.add_matched(1);
}
}
}
+
+ /// Extracts column guarantees from a [`PruningPredicate`] that can be
evaluated against
+ /// Parquet Bloom filters.
+ pub fn prepare_bloom_column_guarantees(
+ predicate: &PruningPredicate,
+ parquet_schema: &SchemaDescriptor,
+ file_schema: &Schema,
+ ) -> Vec<BloomColumnGuarantee> {
+ predicate
+ .literal_columns()
+ .into_iter()
+ .filter_map(|column_name| {
+ let (column_idx, _) = parquet::arrow::parquet_column(
+ parquet_schema,
+ file_schema,
+ &column_name,
+ )?;
+ let in_guarantees: Vec<HashSet<ScalarValue>> = predicate
+ .literal_guarantees()
+ .iter()
+ .filter(|lg| {
+ lg.column.name == column_name && lg.guarantee ==
Guarantee::In
+ })
+ .map(|lg| lg.literals.clone())
+ .collect();
+ Some(BloomColumnGuarantee {
Review Comment:
This still applies at the current head: `WHERE a <> 999 AND b = 999` on the
`bloom_filter_io` fixture still reads `a`'s Bloom filter (request 4) before
`b`'s. Fine to leave for a follow-up PR; could you link an issue for it here so
it isn't lost?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]