ranflarion opened a new pull request, #10963: URL: https://github.com/apache/arrow-rs/pull/10963
# Which issue does this PR close? - Closes #10962. # Rationale for this change A column chunk whose data pages are all dictionary encoded carries its exact set of distinct values in the dictionary page, so a bloom filter for it adds nothing a reader cannot already get exactly, while every value is still hashed into the filter during the write and the filter is serialized after the chunk. parquet-java stopped writing these in PARQUET-2251 (apache/parquet-java#1033, 1.13.0), so files from Spark, Hive and Iceberg never have a bloom filter on a dictionary-only chunk, and there was no way to get the same output from this crate. Details in #10962. # What changes are included in this PR? - `WriterProperties::bloom_filter_for_dictionary_encoded_chunks` with `WriterPropertiesBuilder::set_bloom_filter_for_dictionary_encoded_chunks` and `DEFAULT_BLOOM_FILTER_FOR_DICTIONARY_ENCODED_CHUNKS = true`, so the default output is unchanged. - In `GenericColumnWriter::close`, when the option is `false`, the bloom filter is dropped unless `encoding_stats` records at least one `DATA_PAGE`/`DATA_PAGE_V2` whose encoding is not `PLAIN_DICTIONARY` or `RLE_DICTIONARY`, the same test `ParquetFileWriter.writeColumnChunk` applies in parquet-java. `flush_bloom_filter` is still called so the encoder state is reset as before. # Are these changes tested? Yes, `test_bloom_filter_for_dictionary_encoded_chunks` writes a small dictionary-friendly Int32 column across the dictionary on/off × option on/off matrix and asserts a filter is present in every case except dictionary on with the option off. # Are there any user-facing changes? One new writer property, opt-in, documented on the setter. No breaking changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
