etseidl opened a new issue, #12339:
URL: https://github.com/apache/arrow-rs/issues/12339

   ### Is your feature request related to a problem or challenge?
   
   A caller may already have a file's `ParquetMetaData` and want to inspect 
Bloom filters for several columns across the row groups still under 
consideration. For example, a query engine can first use statistics to narrow 
the candidate row groups, then load the Bloom filters for the physical leaf 
columns referenced by its predicates.
   
   The single-filter `ParquetMetaDataReader::read_column_bloom_filter` and 
async counterpart already support this workflow, but the caller must loop over 
the selected column chunks and maintain its own collection keyed by row group 
and column. The Arrow reader builders also expose individual filter reads. The 
serialized reader's existing Bloom-loading option is all-or-none for columns 
within each row group.
   
   This request is for a convenience API that packages selection, reading and 
lookup into one operation.
   
   ### Describe the solution you'd like
   
   Add synchronous and asynchronous methods to `ParquetMetaDataReader` that 
accept existing metadata and a `ColumnChunkMask`, returning a sparse 
`BloomFilters` collection. Reuse the existing row-group/physical-leaf selection 
model rather than introduce a separate Bloom-specific mask. The source and 
metadata must refer to the same file.
   
   For example, with an existing reader and metadata:
   
   ```rust
   use parquet::file::metadata::{ColumnChunkMask, ParquetMetaDataReader};
   
   let mask = ColumnChunkMask::row_groups_and_columns([1, 3], [0, 2]);
   let filters = ParquetMetaDataReader::read_bloom_filters(&reader, &metadata, 
&mask)?;
   
   if let Some(filter) = filters.get(1, 2)? {
       // Use the filter when evaluating the caller's pruning predicate.
   }
   ```
   The column indices are physical Parquet leaf indices, not top-level Arrow 
field indices. The desired behavior is:
   
   - Read only selected filters, using the supplied metadata without reloading 
the
     footer or modifying `ParquetMetaData`.
   - Return `Some(&Sbbf)` for a loaded filter and `None` for an unselected 
chunk or
     a chunk without a Bloom filter offset; reject out-of-bounds lookup 
coordinates.
   - Require no filter I/O for chunks without Bloom filter offsets.
   - Return an error for read/decode failures rather than a partial collection.
   - Support files both with and without advertised Bloom filter lengths through
     the existing decoding machinery.
   
   Expose the same convenience through `get_bloom_filters(mask)` on the 
synchronous and asynchronous Arrow reader builders, and allow the serialized 
reader to select physical leaf columns with `set_read_bloom_filter_columns`.
   
   For async sources, an optional multi-range fetch method can pass each round 
of filter requests to the backend together, with a sequential default. Backends 
may batch or coalesce requests, but this API does not promise fewer network 
requests, lower latency, or a particular amount of wire-level I/O.
   
   
   ### Describe alternatives you've considered
   
   Continue calling the existing single-filter APIs and collecting their 
results in a caller-owned map. This works and remains appropriate for 
individual filters; the proposed API removes repeated selection and 
result-bookkeeping code when loading a set of filters.
   
   ### Additional context
   
   This builds on [#12090](https://github.com/apache/arrow-rs/pull/12090), 
which added single-filter access to `ParquetMetaDataReader`, and the existing 
`ColumnChunkMask` and sparse column-chunk infrastructure used for page indexes.
   
   
   ### Willingness to contribute
   
   I plan to contribute to this feature independently


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to