andygrove opened a new issue, #11384: URL: https://github.com/apache/arrow-rs/issues/11384
### Is your feature request related to a problem or challenge? parquet-rs writes a dictionary page for high-cardinality columns where parquet-java writes none. That matters to anyone replacing a parquet-java writer with parquet-rs. DataFusion Comet's native Iceberg writer ran into it (apache/datafusion-comet#6114), and #9699 came from the same need. parquet-java decides per column chunk, on the chunk's first data page (`FallbackValuesWriter`): unless the dictionary-encoded page plus the dictionary comes out smaller than the plain page, the whole chunk is written plain and has no dictionary page. parquet-rs never makes that comparison. It keeps a column dictionary-encoded until the dictionary reaches `dictionary_page_size_limit`, then falls back, and still writes the dictionary page because the earlier pages reference it. So columns such as ids, UUIDs and timestamps each carry a dictionary page of up to the limit, which every selective read of the chunk has to fetch. Comet measured 1M rows of a sorted unique `BIGINT` and a unique 16-character string, with Iceberg's 2 MiB dictionary limit. parquet-rs wrote a 2.1 MB dictionary page for each column and parquet-java wrote none, and `WHERE id BETWEEN 1000 AND 1100` read 4.7 MB of the parquet-rs file against 0.6 MB of the parquet-java one. Comet now makes parquet-java's decision itself (apache/datafusion-comet#6241), and @adriangb [asked there](https://github.com/apache/datafusion-comet/pull/6241#issuecomment-5879512836) whether the work could be upstreamed. I'd like to see that happen, but I'm not sure when I'll have time for it, so I'm filing this in case someone wants to pick it up. ### Describe the solution you'd like A function in the parquet crate that takes the Arrow schema, the `WriterProperties` and the first rows to be written, replays parquet-java's first-page accounting for each leaf column, and returns the properties with `set_column_dictionary_enabled(path, false)` for every column parquet-java would write plain. The writer itself doesn't change, and nothing changes for callers who don't use it, so this is closer to the [sampling helper](https://github.com/apache/arrow-rs/pull/10775#issuecomment-5498015501) alamb suggested in #10775 than to a new fallback policy. The goal is to write what parquet-java writes, not to find the best encoding, so [etseidl's point](https://github.com/apache/arrow-rs/pull/9700#issuecomment-4240256987) that one page is a small sample still applies. It's for callers that need to match the writer they are replacing. Comet's version is `DictionaryChooser` in [`iceberg_dictionary.rs`](https://github.com/apache/datafusion-comet/blob/916103f21a2f959e9bb70a25896df6a46cbbd746/native/core/src/execution/operators/iceberg_dictionary.rs). It is about 570 lines plus tests and uses only arrow and parquet, apart from a `HashSet` and an error type from DataFusion. It ports the size accounting of `FallbackValuesWriter` and `DictionaryValuesWriter`, including the 4-byte length of binary values, the fallback when the dictionary outgrows its limit within the first page, and `RunLengthBitPackingHybridEncoder`'s run splitting for the ids. It handles nested leaves and pages that hold only nulls, and reads one column at a time. Its tests compare it with answers recorded from parquet-mr, which were the same in 1.13.1, 1.15.0, 1.16.0 and 1.17.1. Over about 3,000 column and setting pairs it disagreed with parquet-mr on five, all close to the cut-off with 32–64 KiB pages. Those come from where a page ends: it stops at the first row past parquet-mr's size threshold, while parquet-mr waits for its next periodic size check. At the default page settings there were no mismatches. Choosing over 20,000 rows of 75 columns took 27 ms. A general version would need a few things Comet does without. Comet skips INT96 leaves, which Iceberg doesn't have, and FIXED_LEN_BYTE_ARRAY leaves, which parquet-rs only dictionary-encodes under format v2. It hardcodes the row of parquet-mr's first page size check (100, the default). And it makes the choice once, before the file is opened, where parquet-java decides again for every column chunk. ### Describe alternatives you've considered The writer option #9699 asks for, which #9700 started as `DictionaryFallback::OnUnfavorableAfter`. Deciding inside the column writer would match parquet-java for every row group without callers holding rows back, but the writer would have to drop the dictionary and re-encode the first page, which runs into #9739. With #10917's per-row-group properties, a caller could also run the helper again for each row group. Otherwise each engine keeps its own copy, which is where Comet is today. ### Additional context To make the choice, Comet holds back the first page of rows for each partition it writes. In the worst case, where every partition has fewer rows than a page, every row is sampled and that adds about 40% to Parquet encoding. With bigger partitions the choice pays for itself, because high-cardinality columns are cheaper to write plain: on the same 20,000 rows, a zstd write took 48 ms with the chosen properties and 54 ms with dictionary encoding on for every column. _I drafted this issue with AI assistance (Claude Code), and the Comet code linked above was written with it too._ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
