alamb commented on code in PR #12344:
URL: https://github.com/apache/arrow-rs/pull/12344#discussion_r4652382866


##########
parquet/src/arrow/arrow_reader/mod.rs:
##########
@@ -1346,6 +1346,107 @@ impl<T: ChunkReader + 'static> 
ParquetRecordBatchReaderBuilder<T> {
         Sbbf::read_from_column_chunk(column_metadata, &self.input.0)
     }
 
+    /// Reads Bloom filters for the physical column chunks selected by `mask`.
+    ///
+    /// The mask is independent of this builder's projection and row-group 
selection.
+    /// To avoid fetching filters for row groups already pruned, include only 
the

Review Comment:
   while this is good performance advice ("don't fetch useless bloom filters") 
I wonder if this is the best place for it -- it may be hard to discover 🤔 
   
   In other words, perhasp we can move this suggestion to a general purpose 
"performance tips" section on the ParquetMetaDataReader docs or something



##########
parquet/src/arrow/arrow_reader/mod.rs:
##########
@@ -1346,6 +1346,107 @@ impl<T: ChunkReader + 'static> 
ParquetRecordBatchReaderBuilder<T> {
         Sbbf::read_from_column_chunk(column_metadata, &self.input.0)
     }
 
+    /// Reads Bloom filters for the physical column chunks selected by `mask`.
+    ///
+    /// The mask is independent of this builder's projection and row-group 
selection.
+    /// To avoid fetching filters for row groups already pruned, include only 
the
+    /// remaining file row-group indices in `mask`. Select the physical leaf 
columns
+    /// used by predicates, which may differ from the output projection. 
Fetching
+    /// filters does not change either builder setting.
+    ///
+    /// See [`ParquetMetaDataReader::read_bloom_filters`] for details.
+    ///
+    /// # Example
+    ///
+    /// For a file with two flat `Int32` columns, `id` and `output`, load 
filters

Review Comment:
   while I do love a good example, this seems like a pretty complicated example 
to me as it is describing some detailed query engine reader state. I think it 
would be ok to simple say "the engine has determined it only needs to read 
filters for row groups 0 and 2" or something like that 



##########
parquet/src/arrow/async_reader/mod.rs:
##########
@@ -580,6 +582,28 @@ impl<T: AsyncFileReader + Send + 'static> 
ParquetRecordBatchStreamBuilder<T> {
         Sbbf::read_from_column_chunk_async(column_metadata, &mut 
self.input.0).await
     }
 
+    /// Reads Bloom filters for the physical column chunks selected by `mask`.
+    ///
+    /// The mask is independent of this builder's projection and row-group 
selection.

Review Comment:
   same comments as above on performance notes. Even better that putting it 
somewhere else would avoid the duplication



##########
parquet/src/file/serialized_reader.rs:
##########
@@ -313,13 +313,16 @@ impl<R: 'static + ChunkReader> FileReader for 
SerializedFileReader<R> {
     }
 }
 
+// Standalone row groups store filters at local row group coordinate 0.
+type RowGroupBloomFilters = BloomFilters;

Review Comment:
   What is the purpose of this alias? Why not just use the struct directly?



##########
parquet/src/file/properties.rs:
##########
@@ -2158,14 +2157,28 @@ impl ReaderPropertiesBuilder {
         self
     }
 
-    /// Enable/disable reading bloom filter
+    /// Enables or disables reading Bloom filters for all columns in each row 
group.
     ///
-    /// If reading bloom filter is enabled, bloom filter will be read from the 
file.
-    /// If reading bloom filter is disabled, bloom filter will not be read 
from the file.
-    ///
-    /// By default bloom filter is set to be read.
+    /// Defaults to disabled. Replaces any previous Bloom filter selection.
     pub fn set_read_bloom_filter(mut self, value: bool) -> Self {
-        self.read_bloom_filter = Some(value);
+        self.bloom_filter_mask = Some(if value {

Review Comment:
   👍 



##########
parquet/src/file/metadata/bloom_filter.rs:
##########
@@ -0,0 +1,258 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: See the NOTICE file distributed with this work for 
additional information regarding copyright ownership
+// SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) 
under one or more contributor license agreements
+
+//! Sparse decoded Bloom filters keyed by row-group and column coordinates.
+
+use crate::DecodeResult;
+use crate::bloom_filter::{Sbbf, SbbfPushDecoder};
+use crate::errors::{ParquetError, Result};
+use crate::file::metadata::column_chunk::{ColumnChunkEntries, ColumnChunkKey};
+use crate::file::metadata::{
+    ColumnChunkMask, ColumnChunkMetaData, ParquetMetaData, RowGroupMetaData,
+};
+use crate::file::reader::ChunkReader;
+use bytes::Bytes;
+use std::ops::Range;
+
+/// Sparse decoded Bloom filters keyed by row group and physical leaf column.
+#[derive(Debug, Clone)]
+pub struct BloomFilters {
+    entries: ColumnChunkEntries<Sbbf>,

Review Comment:
   I wonder if we should name this more consistently -- like the bloom filters 
themselves are called `Sbbf` but this is called `BloomFilters` -- how about 
`Sbbfs` (sounds like BFFs 😆 ) or `SbbfSet` 🤔 



##########
parquet/tests/arrow_reader/bloom_filter/batch.rs:
##########
@@ -0,0 +1,601 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: See the NOTICE file distributed with this work for 
additional information regarding copyright ownership
+// SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) 
under one or more contributor license agreements
+
+use super::{RecordingReader, encode};
+use arrow_array::{ArrayRef, Int32Array, RecordBatch};
+use bytes::Bytes;
+use parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder;
+use parquet::arrow::{ArrowWriter, ProjectionMask};
+use parquet::bloom_filter::Sbbf;
+use parquet::file::metadata::page_index::RowGroupPageIndex;
+use parquet::file::metadata::{
+    ColumnChunkMask, ColumnChunkMetaData, FileMetaData, ParquetMetaData, 
ParquetMetaDataReader,
+    RowGroupMetaData,
+};
+use parquet::file::properties::{ReaderProperties, WriterProperties};
+use parquet::file::reader::{FileReader, RowGroupReader};
+use parquet::file::serialized_reader::{
+    ReadOptionsBuilder, SerializedFileReader, SerializedRowGroupReader,
+};
+use parquet::schema::{parser::parse_message_type, types::SchemaDescriptor};
+use std::ops::Range;
+use std::sync::Arc;
+
+// Three distinct filters, widely separated, in an asymmetric physical-leaf 
grid.
+fn fixture(unknown_length: bool) -> (ParquetMetaData, Bytes, Vec<Range<u64>>) {
+    let schema = Arc::new(SchemaDescriptor::new(Arc::new(
+        parse_message_type(
+            "message schema { REQUIRED GROUP nested { REQUIRED INT32 a; 
REQUIRED INT32 b; } REQUIRED INT32 value; }",
+        )
+        .unwrap(),
+    )));
+    let mut bytes = vec![0; 13];
+    let mut ranges = Vec::new();
+    let mut rows = Vec::new();
+    for row in 0..2 {
+        let mut columns = Vec::new();
+        for col in 0..3 {
+            let mut builder = ColumnChunkMetaData::builder(schema.column(col));
+            if [(0, 0), (0, 2), (1, 1)].contains(&(row, col)) {
+                let mut filter = Sbbf::new_with_num_of_bytes(32);
+                filter.insert(&((row * 3 + col) as i32));
+                let start = bytes.len();
+                filter.write(&mut bytes).unwrap();
+                ranges.push(start as u64..bytes.len() as u64);
+                builder = builder.set_bloom_filter_offset(Some(start as i64));
+                if !unknown_length || (row, col) != (0, 2) {
+                    builder = 
builder.set_bloom_filter_length(Some((bytes.len() - start) as i32));
+                }
+                bytes.resize(bytes.len() + 257, 0);
+            }
+            columns.push(builder.build().unwrap());
+        }
+        rows.push(
+            RowGroupMetaData::builder(schema.clone())
+                .set_column_metadata(columns)
+                .build()
+                .unwrap(),
+        );
+    }
+    (
+        ParquetMetaData::new(FileMetaData::new(1, 0, None, None, schema, 
None), rows),
+        Bytes::from(bytes),
+        ranges,
+    )
+}
+
+// Two row groups with a Bloom filter on the predicate column, but not the 
output column.
+fn written_file() -> Bytes {
+    let batches: Vec<_> = [10, 30]
+        .into_iter()
+        .map(|start| {
+            RecordBatch::try_from_iter([
+                (
+                    "value",
+                    Arc::new(Int32Array::from(vec![start, start + 1])) as 
ArrayRef,
+                ),
+                (
+                    "output",
+                    Arc::new(Int32Array::from(vec![start + 100, start + 101])) 
as ArrayRef,
+                ),
+            ])
+            .unwrap()
+        })
+        .collect();
+    let properties = WriterProperties::builder()
+        .set_column_bloom_filter_enabled("value".into(), true)
+        .build();
+    let mut writer =
+        ArrowWriter::try_new(Vec::new(), batches[0].schema(), 
Some(properties)).unwrap();
+    for batch in batches {
+        writer.write(&batch).unwrap();
+        writer.flush().unwrap();
+    }
+    Bytes::from(writer.into_inner().unwrap())
+}
+
+#[test]
+fn serialized_row_group_bloom_loading_is_sparse_and_selective() {
+    for unknown in [false, true] {
+        let (metadata, bytes, _) = fixture(unknown);
+        let row_group = metadata.row_group(0);
+        // Column 0 has an invalid filter location but must not be examined 
when unselected.
+        let mut columns = row_group.columns().to_vec();
+        columns[0] = 
ColumnChunkMetaData::builder(row_group.schema_descr().column(0))
+            .set_bloom_filter_offset(Some(-1))
+            .build()
+            .unwrap();
+        let row_group = RowGroupMetaData::builder(row_group.schema_descr_ptr())
+            .set_column_metadata(columns)
+            .set_ordinal(7)
+            .build()
+            .unwrap();
+        let props = ReaderProperties::builder()
+            .set_read_bloom_filter(true)
+            .set_read_bloom_filter_columns([2, 1, 2, 99, usize::MAX])
+            .build();
+        let reader = SerializedRowGroupReader::new(
+            Arc::new(bytes.clone()),
+            &row_group,
+            RowGroupPageIndex::new(7, None),
+            Arc::new(props),
+        )
+        .unwrap();
+        assert!(reader.get_column_bloom_filter(0).is_none());
+        assert!(reader.get_column_bloom_filter(1).is_none());
+        assert!(reader.get_column_bloom_filter(2).unwrap().check(&2_i32));
+
+        // Setter equivalence is covered by the properties tests; test the 
default no-I/O path here.
+        let disabled = SerializedRowGroupReader::new(
+            Arc::new(RecordingReader::default()),
+            &row_group,
+            RowGroupPageIndex::new(7, None),
+            Arc::new(ReaderProperties::builder().build()),
+        )
+        .unwrap();
+        for column in 0..3 {
+            assert!(disabled.get_column_bloom_filter(column).is_none());
+        }
+        assert!(
+            SerializedRowGroupReader::new(
+                Arc::new(RecordingReader::default()),
+                &row_group,
+                RowGroupPageIndex::new(7, None),
+                Arc::new(
+                    ReaderProperties::builder()
+                        .set_read_bloom_filter(true)
+                        .build()
+                ),
+            )
+            .is_err()
+        );
+    }
+}
+
+#[test]
+fn serialized_file_bloom_selection_survives_row_group_filtering() {
+    let options = ReadOptionsBuilder::new()
+        .with_predicate(Box::new(|_, index| index == 1))
+        .with_reader_properties(
+            ReaderProperties::builder()
+                .set_read_bloom_filter_columns([0])
+                .build(),
+        )
+        .build();
+    let reader = SerializedFileReader::new_with_options(written_file(), 
options).unwrap();
+    assert_eq!(reader.num_row_groups(), 1);
+    let row_group = reader.get_row_group(0).unwrap();
+    assert!(row_group.get_column_bloom_filter(0).unwrap().check(&30_i32));
+    assert!(row_group.get_column_bloom_filter(1).is_none());
+}
+
+#[test]
+fn sync_builder_bloom_mask_is_explicit_and_builder_remains_usable() {
+    let builder = 
ParquetRecordBatchReaderBuilder::try_new(written_file()).unwrap();
+    let metadata = builder.metadata().clone();
+    let projection = ProjectionMask::leaves(builder.parquet_schema(), [1]);
+    let builder = builder.with_projection(projection).with_row_groups(vec![0]);
+    let mask = ColumnChunkMask::row_groups_and_columns([1], [0]);
+    let filters = builder.get_bloom_filters(&mask).unwrap();
+    assert!(filters.get(1, 0).unwrap().unwrap().check(&30_i32));
+    assert!(filters.get(0, 0).unwrap().is_none());
+    assert!(filters.get(1, 1).unwrap().is_none());
+    assert!(Arc::ptr_eq(builder.metadata(), &metadata));
+    // Fetching a different row group's filter must not change the data 
selection.
+    let batches = builder
+        .with_batch_size(1)
+        .build()
+        .unwrap()
+        .collect::<Result<Vec<_>, _>>()
+        .unwrap();
+    assert_projected_output(&batches);
+    assert!(filters.get(1, 0).unwrap().unwrap().check(&31_i32));
+}
+
+fn assert_projected_output(batches: &[RecordBatch]) {
+    let values: Vec<_> = batches
+        .iter()
+        .flat_map(|batch| {
+            assert_eq!(batch.num_columns(), 1);
+            batch
+                .column(0)
+                .as_any()
+                .downcast_ref::<Int32Array>()
+                .unwrap()
+                .values()
+                .iter()
+                .copied()
+        })
+        .collect();
+    assert_eq!(values, vec![110, 111]);
+}
+
+fn responses(bytes: &Bytes, ranges: &[Range<u64>]) -> Vec<Bytes> {
+    ranges
+        .iter()
+        .map(|range| bytes.slice(range.start as usize..range.end as usize))
+        .collect()
+}
+
+#[test]
+fn batch_masks_preserve_physical_coordinates_and_optional_lookup() {
+    let (metadata, bytes, _) = fixture(false);
+    for mask in [
+        ColumnChunkMask::all(),
+        ColumnChunkMask::none(),
+        ColumnChunkMask::columns([2, 0, 2, 99, usize::MAX]),
+        ColumnChunkMask::row_groups([1, 1, 99, usize::MAX]),
+        ColumnChunkMask::row_groups_and_columns([1, 0, 1, 99], [2, 2, 
usize::MAX]),
+        ColumnChunkMask::row_groups_and_columns([1], [0, 2]),
+    ] {
+        let result = ParquetMetaDataReader::read_bloom_filters(&bytes, 
&metadata, &mask).unwrap();
+        for row in 0..2 {
+            for col in 0..3 {
+                let filter = result.get(row, col).unwrap();
+                if !mask.includes_row_group(row, 2) || 
!mask.includes_column(col, 3) {
+                    assert!(filter.is_none());
+                    continue;
+                }
+                let single =
+                    ParquetMetaDataReader::read_column_bloom_filter(&bytes, 
&metadata, row, col)
+                        .unwrap();
+                match (filter, single) {
+                    (Some(batch), Some(single)) => {
+                        let mut batch_bytes = Vec::new();
+                        let mut single_bytes = Vec::new();
+                        batch.write(&mut batch_bytes).unwrap();
+                        single.write(&mut single_bytes).unwrap();
+                        assert_eq!(batch_bytes, single_bytes);
+                        assert!(batch.check(&((row * 3 + col) as i32)));
+                    }
+                    (None, None) => {}
+                    _ => panic!("wrong coverage at ({row}, {col})"),
+                }
+            }
+        }
+        for (row, col) in [(2, 0), (0, 3), (usize::MAX, usize::MAX)] {
+            assert!(result.get(row, col).is_err());
+        }
+    }
+}
+
+#[test]
+fn batch_sync_stages_requests_and_skips_absences() {
+    let (metadata, bytes, ranges) = fixture(true);
+    let first = vec![
+        ranges[0].clone(),
+        ranges[1].start..ranges[1].start + 20,
+        ranges[2].clone(),
+    ];
+    let bitset = ranges[1].end - 32..ranges[1].end;
+    let mut expected = first.clone();
+    expected.push(bitset);
+    let reader = RecordingReader::new(responses(&bytes, &expected));
+    let result =
+        ParquetMetaDataReader::read_bloom_filters(&reader, &metadata, 
&ColumnChunkMask::all())
+            .unwrap();
+    assert_eq!(reader.requests(), expected);
+    assert!(result.get(0, 2).unwrap().unwrap().check(&2_i32));
+    let no_reads = RecordingReader::default();
+    for mask in [
+        ColumnChunkMask::none(),
+        ColumnChunkMask::row_groups_and_columns([1], [0, 2]),
+    ] {
+        let result =
+            ParquetMetaDataReader::read_bloom_filters(&no_reads, &metadata, 
&mask).unwrap();
+        for row in 0..2 {
+            for col in 0..3 {
+                assert!(result.get(row, col).unwrap().is_none());
+            }
+        }
+        assert!(result.get(2, 0).is_err());
+        assert!(result.get(0, 3).is_err());
+    }
+    let absent = super::metadata(None, Some(-1));
+    let result =
+        ParquetMetaDataReader::read_bloom_filters(&no_reads, &absent, 
&ColumnChunkMask::all())
+            .unwrap();
+    assert!(result.get(0, 0).unwrap().is_none());
+    assert!(result.get(1, 0).is_err());
+    assert!(result.get(0, 1).is_err());
+    assert!(no_reads.requests().is_empty());
+}
+
+#[test]
+fn batch_sync_errors_fail_the_whole_request() {
+    let (metadata, bytes, ranges) = fixture(false);
+    let valid = responses(&bytes, &ranges);
+    // A bad second filter must fail a multi-filter read, not yield a partial 
result.
+    for response in [valid[1].slice(..valid[1].len() - 1), encode(33, 32)] {
+        let mut buffers = valid.clone();
+        buffers[1] = response;
+        let reader = RecordingReader::new(buffers);
+        assert!(
+            ParquetMetaDataReader::read_bloom_filters(&reader, &metadata, 
&ColumnChunkMask::all())
+                .is_err()
+        );
+    }
+    let reader = RecordingReader::new(valid);
+    reader.fail_at(1);
+    assert!(
+        ParquetMetaDataReader::read_bloom_filters(&reader, &metadata, 
&ColumnChunkMask::all())
+            .is_err()
+    );
+    let reader = RecordingReader::default();
+    assert!(
+        ParquetMetaDataReader::read_bloom_filters(
+            &reader,
+            &super::metadata(Some(-1), None),
+            &ColumnChunkMask::all(),
+        )
+        .is_err()
+    );
+    assert!(reader.requests().is_empty());
+}
+
+#[cfg(feature = "async")]
+mod r#async {
+    use super::super::metadata as bloom_metadata;
+    use super::*;
+    use crate::io::TestReader;
+    use futures::TryStreamExt;
+    use futures::future::{BoxFuture, ready};
+    use parquet::arrow::ParquetRecordBatchStreamBuilder;
+    use parquet::arrow::{
+        arrow_reader::{ArrowReaderMetadata, ArrowReaderOptions},
+        async_reader::{AsyncFileReader, MetadataFetch},
+    };
+    use parquet::errors::{ParquetError, Result};
+    use std::collections::VecDeque;
+    use std::rc::Rc;
+    use std::sync::Mutex;
+
+    struct BatchReader {
+        rounds: Arc<Mutex<Vec<Vec<Range<u64>>>>>,
+
+        responses: VecDeque<Result<Vec<Bytes>>>,
+    }
+
+    impl AsyncFileReader for BatchReader {
+        fn get_bytes(&mut self, _: Range<u64>) -> BoxFuture<'_, Result<Bytes>> 
{
+            panic!("batch adapter must use get_byte_ranges")
+        }
+        fn get_byte_ranges(
+            &mut self,
+            ranges: Vec<Range<u64>>,
+        ) -> BoxFuture<'_, Result<Vec<Bytes>>> {
+            self.rounds.lock().unwrap().push(ranges);
+            Box::pin(ready(self.responses.pop_front().expect("unexpected 
batch")))
+        }
+        fn get_metadata<'a>(
+            &'a mut self,
+            _: Option<&'a ArrowReaderOptions>,
+        ) -> BoxFuture<'a, Result<Arc<ParquetMetaData>>> {
+            panic!("must not reload footer")
+        }
+    }
+
+    #[tokio::test]
+    async fn batch_async_gathers_each_round_through_metadata_and_builder() {

Review Comment:
   I find these tests really hard to understand what behavior they are testing 
as the setup and assertions are in terms of low level coding primitives (file 
ranges, etc)
   
   May I (selfishly) suggest something like 
https://andrew.nerdnetworks.org/blog/2026/11/13/transcript-testing/



##########
parquet/tests/arrow_reader/bloom_filter/batch.rs:
##########
@@ -0,0 +1,601 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: See the NOTICE file distributed with this work for 
additional information regarding copyright ownership
+// SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) 
under one or more contributor license agreements
+
+use super::{RecordingReader, encode};
+use arrow_array::{ArrayRef, Int32Array, RecordBatch};
+use bytes::Bytes;
+use parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder;
+use parquet::arrow::{ArrowWriter, ProjectionMask};
+use parquet::bloom_filter::Sbbf;
+use parquet::file::metadata::page_index::RowGroupPageIndex;
+use parquet::file::metadata::{
+    ColumnChunkMask, ColumnChunkMetaData, FileMetaData, ParquetMetaData, 
ParquetMetaDataReader,
+    RowGroupMetaData,
+};
+use parquet::file::properties::{ReaderProperties, WriterProperties};
+use parquet::file::reader::{FileReader, RowGroupReader};
+use parquet::file::serialized_reader::{
+    ReadOptionsBuilder, SerializedFileReader, SerializedRowGroupReader,
+};
+use parquet::schema::{parser::parse_message_type, types::SchemaDescriptor};
+use std::ops::Range;
+use std::sync::Arc;
+
+// Three distinct filters, widely separated, in an asymmetric physical-leaf 
grid.
+fn fixture(unknown_length: bool) -> (ParquetMetaData, Bytes, Vec<Range<u64>>) {
+    let schema = Arc::new(SchemaDescriptor::new(Arc::new(
+        parse_message_type(
+            "message schema { REQUIRED GROUP nested { REQUIRED INT32 a; 
REQUIRED INT32 b; } REQUIRED INT32 value; }",
+        )
+        .unwrap(),
+    )));
+    let mut bytes = vec![0; 13];
+    let mut ranges = Vec::new();
+    let mut rows = Vec::new();
+    for row in 0..2 {
+        let mut columns = Vec::new();
+        for col in 0..3 {
+            let mut builder = ColumnChunkMetaData::builder(schema.column(col));
+            if [(0, 0), (0, 2), (1, 1)].contains(&(row, col)) {
+                let mut filter = Sbbf::new_with_num_of_bytes(32);
+                filter.insert(&((row * 3 + col) as i32));
+                let start = bytes.len();
+                filter.write(&mut bytes).unwrap();
+                ranges.push(start as u64..bytes.len() as u64);
+                builder = builder.set_bloom_filter_offset(Some(start as i64));
+                if !unknown_length || (row, col) != (0, 2) {
+                    builder = 
builder.set_bloom_filter_length(Some((bytes.len() - start) as i32));
+                }
+                bytes.resize(bytes.len() + 257, 0);
+            }
+            columns.push(builder.build().unwrap());
+        }
+        rows.push(
+            RowGroupMetaData::builder(schema.clone())
+                .set_column_metadata(columns)
+                .build()
+                .unwrap(),
+        );
+    }
+    (
+        ParquetMetaData::new(FileMetaData::new(1, 0, None, None, schema, 
None), rows),
+        Bytes::from(bytes),
+        ranges,
+    )
+}
+
+// Two row groups with a Bloom filter on the predicate column, but not the 
output column.
+fn written_file() -> Bytes {
+    let batches: Vec<_> = [10, 30]
+        .into_iter()
+        .map(|start| {
+            RecordBatch::try_from_iter([
+                (
+                    "value",
+                    Arc::new(Int32Array::from(vec![start, start + 1])) as 
ArrayRef,
+                ),
+                (
+                    "output",
+                    Arc::new(Int32Array::from(vec![start + 100, start + 101])) 
as ArrayRef,
+                ),
+            ])
+            .unwrap()
+        })
+        .collect();
+    let properties = WriterProperties::builder()
+        .set_column_bloom_filter_enabled("value".into(), true)
+        .build();
+    let mut writer =
+        ArrowWriter::try_new(Vec::new(), batches[0].schema(), 
Some(properties)).unwrap();
+    for batch in batches {
+        writer.write(&batch).unwrap();
+        writer.flush().unwrap();
+    }
+    Bytes::from(writer.into_inner().unwrap())
+}
+
+#[test]
+fn serialized_row_group_bloom_loading_is_sparse_and_selective() {
+    for unknown in [false, true] {
+        let (metadata, bytes, _) = fixture(unknown);
+        let row_group = metadata.row_group(0);
+        // Column 0 has an invalid filter location but must not be examined 
when unselected.
+        let mut columns = row_group.columns().to_vec();
+        columns[0] = 
ColumnChunkMetaData::builder(row_group.schema_descr().column(0))
+            .set_bloom_filter_offset(Some(-1))
+            .build()
+            .unwrap();
+        let row_group = RowGroupMetaData::builder(row_group.schema_descr_ptr())
+            .set_column_metadata(columns)
+            .set_ordinal(7)
+            .build()
+            .unwrap();
+        let props = ReaderProperties::builder()
+            .set_read_bloom_filter(true)
+            .set_read_bloom_filter_columns([2, 1, 2, 99, usize::MAX])
+            .build();
+        let reader = SerializedRowGroupReader::new(
+            Arc::new(bytes.clone()),
+            &row_group,
+            RowGroupPageIndex::new(7, None),
+            Arc::new(props),
+        )
+        .unwrap();
+        assert!(reader.get_column_bloom_filter(0).is_none());
+        assert!(reader.get_column_bloom_filter(1).is_none());
+        assert!(reader.get_column_bloom_filter(2).unwrap().check(&2_i32));
+
+        // Setter equivalence is covered by the properties tests; test the 
default no-I/O path here.
+        let disabled = SerializedRowGroupReader::new(
+            Arc::new(RecordingReader::default()),
+            &row_group,
+            RowGroupPageIndex::new(7, None),
+            Arc::new(ReaderProperties::builder().build()),
+        )
+        .unwrap();
+        for column in 0..3 {
+            assert!(disabled.get_column_bloom_filter(column).is_none());
+        }
+        assert!(
+            SerializedRowGroupReader::new(
+                Arc::new(RecordingReader::default()),
+                &row_group,
+                RowGroupPageIndex::new(7, None),
+                Arc::new(
+                    ReaderProperties::builder()
+                        .set_read_bloom_filter(true)
+                        .build()
+                ),
+            )
+            .is_err()
+        );
+    }
+}
+
+#[test]
+fn serialized_file_bloom_selection_survives_row_group_filtering() {
+    let options = ReadOptionsBuilder::new()
+        .with_predicate(Box::new(|_, index| index == 1))
+        .with_reader_properties(
+            ReaderProperties::builder()
+                .set_read_bloom_filter_columns([0])
+                .build(),
+        )
+        .build();
+    let reader = SerializedFileReader::new_with_options(written_file(), 
options).unwrap();
+    assert_eq!(reader.num_row_groups(), 1);
+    let row_group = reader.get_row_group(0).unwrap();
+    assert!(row_group.get_column_bloom_filter(0).unwrap().check(&30_i32));
+    assert!(row_group.get_column_bloom_filter(1).is_none());
+}
+
+#[test]
+fn sync_builder_bloom_mask_is_explicit_and_builder_remains_usable() {
+    let builder = 
ParquetRecordBatchReaderBuilder::try_new(written_file()).unwrap();
+    let metadata = builder.metadata().clone();
+    let projection = ProjectionMask::leaves(builder.parquet_schema(), [1]);
+    let builder = builder.with_projection(projection).with_row_groups(vec![0]);
+    let mask = ColumnChunkMask::row_groups_and_columns([1], [0]);
+    let filters = builder.get_bloom_filters(&mask).unwrap();
+    assert!(filters.get(1, 0).unwrap().unwrap().check(&30_i32));
+    assert!(filters.get(0, 0).unwrap().is_none());
+    assert!(filters.get(1, 1).unwrap().is_none());
+    assert!(Arc::ptr_eq(builder.metadata(), &metadata));
+    // Fetching a different row group's filter must not change the data 
selection.
+    let batches = builder
+        .with_batch_size(1)
+        .build()
+        .unwrap()
+        .collect::<Result<Vec<_>, _>>()
+        .unwrap();
+    assert_projected_output(&batches);
+    assert!(filters.get(1, 0).unwrap().unwrap().check(&31_i32));
+}
+
+fn assert_projected_output(batches: &[RecordBatch]) {
+    let values: Vec<_> = batches
+        .iter()
+        .flat_map(|batch| {
+            assert_eq!(batch.num_columns(), 1);
+            batch
+                .column(0)
+                .as_any()
+                .downcast_ref::<Int32Array>()
+                .unwrap()
+                .values()
+                .iter()
+                .copied()
+        })
+        .collect();
+    assert_eq!(values, vec![110, 111]);
+}
+
+fn responses(bytes: &Bytes, ranges: &[Range<u64>]) -> Vec<Bytes> {
+    ranges
+        .iter()
+        .map(|range| bytes.slice(range.start as usize..range.end as usize))
+        .collect()
+}
+
+#[test]
+fn batch_masks_preserve_physical_coordinates_and_optional_lookup() {

Review Comment:
   I don't understand the rationale for this test. What property it is testing? 
It seems like some sort of deep test of the internals



##########
parquet/tests/arrow_reader/bloom_filter/batch.rs:
##########
@@ -0,0 +1,601 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: See the NOTICE file distributed with this work for 
additional information regarding copyright ownership
+// SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) 
under one or more contributor license agreements
+
+use super::{RecordingReader, encode};
+use arrow_array::{ArrayRef, Int32Array, RecordBatch};
+use bytes::Bytes;
+use parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder;
+use parquet::arrow::{ArrowWriter, ProjectionMask};
+use parquet::bloom_filter::Sbbf;
+use parquet::file::metadata::page_index::RowGroupPageIndex;
+use parquet::file::metadata::{
+    ColumnChunkMask, ColumnChunkMetaData, FileMetaData, ParquetMetaData, 
ParquetMetaDataReader,
+    RowGroupMetaData,
+};
+use parquet::file::properties::{ReaderProperties, WriterProperties};
+use parquet::file::reader::{FileReader, RowGroupReader};
+use parquet::file::serialized_reader::{
+    ReadOptionsBuilder, SerializedFileReader, SerializedRowGroupReader,
+};
+use parquet::schema::{parser::parse_message_type, types::SchemaDescriptor};
+use std::ops::Range;
+use std::sync::Arc;
+
+// Three distinct filters, widely separated, in an asymmetric physical-leaf 
grid.

Review Comment:
   I think a comment / diagram (ascii art perhaps?) would help me understand 
this as I am not sure what a assymetric physical-leaf grid means



##########
parquet/src/file/metadata/reader.rs:
##########
@@ -888,6 +890,43 @@ impl ParquetMetaDataReader {
         Sbbf::read_from_column_chunk_with_fetch(column, fetch).await
     }
 
+    /// Reads Bloom filters for the physical column chunks selected by `mask`.
+    ///
+    /// The metadata and reader must refer to the same file. Missing Bloom 
filter

Review Comment:
   What happens when there are missing offsets? I seems like that means the 
reader would error (as there are no bloom filters to read?)



##########
parquet/src/file/metadata/bloom_filter.rs:
##########
@@ -0,0 +1,258 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: See the NOTICE file distributed with this work for 
additional information regarding copyright ownership
+// SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) 
under one or more contributor license agreements
+
+//! Sparse decoded Bloom filters keyed by row-group and column coordinates.
+
+use crate::DecodeResult;
+use crate::bloom_filter::{Sbbf, SbbfPushDecoder};
+use crate::errors::{ParquetError, Result};
+use crate::file::metadata::column_chunk::{ColumnChunkEntries, ColumnChunkKey};
+use crate::file::metadata::{
+    ColumnChunkMask, ColumnChunkMetaData, ParquetMetaData, RowGroupMetaData,
+};
+use crate::file::reader::ChunkReader;
+use bytes::Bytes;
+use std::ops::Range;
+
+/// Sparse decoded Bloom filters keyed by row group and physical leaf column.
+#[derive(Debug, Clone)]
+pub struct BloomFilters {
+    entries: ColumnChunkEntries<Sbbf>,
+}
+
+impl BloomFilters {
+    /// Looks up a decoded filter by row group and physical leaf column.
+    ///
+    /// Returns `Ok(None)` for unselected chunks or chunks without a Bloom 
filter
+    /// offset, and an error for out-of-bounds coordinates.
+    pub fn get(&self, row_group_idx: usize, column_idx: usize) -> 
Result<Option<&Sbbf>> {
+        self.entries.get(row_group_idx, column_idx)
+    }
+
+    // A standalone row group uses local row coordinate 0, regardless of its 
file ordinal.
+    pub(crate) fn read_row_group<R: ChunkReader>(

Review Comment:
   since this isn't pub anyway, why not just call 
BloomFilterBatch::try_new_row_group when needed and avoid this wrapper?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to