etseidl opened a new issue, #12339:
URL: https://github.com/apache/arrow-rs/issues/12339
### Is your feature request related to a problem or challenge?
A caller may already have a file's `ParquetMetaData` and want to inspect
Bloom filters for several columns across the row groups still under
consideration. For example, a query engine can first use statistics to narrow
the candidate row groups, then load the Bloom filters for the physical leaf
columns referenced by its predicates.
The single-filter `ParquetMetaDataReader::read_column_bloom_filter` and
async counterpart already support this workflow, but the caller must loop over
the selected column chunks and maintain its own collection keyed by row group
and column. The Arrow reader builders also expose individual filter reads. The
serialized reader's existing Bloom-loading option is all-or-none for columns
within each row group.
This request is for a convenience API that packages selection, reading and
lookup into one operation.
### Describe the solution you'd like
Add synchronous and asynchronous methods to `ParquetMetaDataReader` that
accept existing metadata and a `ColumnChunkMask`, returning a sparse
`BloomFilters` collection. Reuse the existing row-group/physical-leaf selection
model rather than introduce a separate Bloom-specific mask. The source and
metadata must refer to the same file.
For example, with an existing reader and metadata:
```rust
use parquet::file::metadata::{ColumnChunkMask, ParquetMetaDataReader};
let mask = ColumnChunkMask::row_groups_and_columns([1, 3], [0, 2]);
let filters = ParquetMetaDataReader::read_bloom_filters(&reader, &metadata,
&mask)?;
if let Some(filter) = filters.get(1, 2)? {
// Use the filter when evaluating the caller's pruning predicate.
}
```
The column indices are physical Parquet leaf indices, not top-level Arrow
field indices. The desired behavior is:
- Read only selected filters, using the supplied metadata without reloading
the
footer or modifying `ParquetMetaData`.
- Return `Some(&Sbbf)` for a loaded filter and `None` for an unselected
chunk or
a chunk without a Bloom filter offset; reject out-of-bounds lookup
coordinates.
- Require no filter I/O for chunks without Bloom filter offsets.
- Return an error for read/decode failures rather than a partial collection.
- Support files both with and without advertised Bloom filter lengths through
the existing decoding machinery.
Expose the same convenience through `get_bloom_filters(mask)` on the
synchronous and asynchronous Arrow reader builders, and allow the serialized
reader to select physical leaf columns with `set_read_bloom_filter_columns`.
For async sources, an optional multi-range fetch method can pass each round
of filter requests to the backend together, with a sequential default. Backends
may batch or coalesce requests, but this API does not promise fewer network
requests, lower latency, or a particular amount of wire-level I/O.
### Describe alternatives you've considered
Continue calling the existing single-filter APIs and collecting their
results in a caller-owned map. This works and remains appropriate for
individual filters; the proposed API removes repeated selection and
result-bookkeeping code when loading a set of filters.
### Additional context
This builds on [#12090](https://github.com/apache/arrow-rs/pull/12090),
which added single-filter access to `ParquetMetaDataReader`, and the existing
`ColumnChunkMask` and sparse column-chunk infrastructure used for page indexes.
### Willingness to contribute
I plan to contribute to this feature independently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]