ranflarion opened a new issue, #10962:
URL: https://github.com/apache/arrow-rs/issues/10962
### Is your feature request related to a problem or challenge?
When bloom filters are enabled for a column, the parquet writer emits one
for every column chunk, including chunks whose data pages are all dictionary
encoded. For such a chunk the bloom filter carries no information the file does
not already have: the dictionary page is the exact set of distinct values in
the chunk, so a reader that wants to know whether a value can be present can
read the (small) dictionary page and get an exact answer instead of a
probabilistic one.
The cost is paid on the write side. Every value is hashed into the filter as
it is encoded, and the filter is serialized after the chunk (32 bytes minimum
per chunk after the folding added in #9628, up to the ndv/fpp-derived size for
high-cardinality columns that stayed dictionary encoded). For low-cardinality
columns, which are the ones most likely to stay dictionary encoded, that is
pure overhead.
parquet-java made this decision in the writer in PARQUET-2251
(apache/parquet-java#1033, released in 1.13.0):
`ParquetFileWriter.writeColumnChunk` only keeps the bloom filter if at least
one data page of the chunk uses an encoding other than `PLAIN_DICTIONARY` /
`RLE_DICTIONARY`. Files written by Spark, Hive, Iceberg and other parquet-java
based writers therefore never have a bloom filter on a dictionary-only chunk.
There is currently no way to get the same output from the `parquet` crate,
which matters for anyone producing files that need to be byte-for-byte
comparable with parquet-java output (we hit this comparing native Spark writes
against Spark's own).
### Describe the solution you'd like
A `WriterProperties` option that makes the column writer drop the bloom
filter of a chunk whose data pages are all dictionary encoded. The writer
already tracks per-page encodings in `encoding_stats` for the chunk metadata,
so the decision is a scan of that list at `GenericColumnWriter::close`, right
where `flush_bloom_filter()` is called today.
Roughly:
```rust
let props = WriterProperties::builder()
.set_bloom_filter_enabled(true)
.set_bloom_filter_for_dictionary_encoded_chunks(false)
.build();
```
Naming is open. It could also live on the per-column bloom filter
configuration discussed in #9667, but the rule does not seem column specific in
practice.
I would suggest keeping the current behaviour (always write) as the default,
since dropping the filter changes what downstream readers see and some of them
prune only on bloom filters, not on dictionary pages. Happy to make it the
default instead if maintainers prefer matching parquet-java out of the box.
### Describe alternatives you've considered
- Doing it outside the writer. `ArrowColumnChunk::close_mut` (#9773) allows
clearing `bloom_filter` before appending a chunk, but only on the low-level
per-column API; `ArrowWriter` users have no equivalent hook, and
re-implementing the rule requires inspecting encoding stats that the writer
already has.
- Reading side: readers could prune dictionary-only chunks by scanning the
dictionary page when no bloom filter is present. That is complementary and does
not remove the write-side cost or the output mismatch with parquet-java.
- Disabling dictionary encoding for bloom-filtered columns. That trades away
the encoding to get consistent bloom filter presence, which is the wrong
direction.
### Additional context
- parquet-java rule:
https://github.com/apache/parquet-java/blob/apache-parquet-1.13.1/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileWriter.java#L883-L896
- Current unconditional flush in the crate:
`parquet/src/column/writer/mod.rs`, `bloom_filter:
self.encoder.flush_bloom_filter()` in `close()`.
- I have a working implementation (new property, `encoding_stats` check in
`close`, unit test covering the dictionary on/off × option on/off matrix) and
can open a PR once the shape here is agreed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
