Weijun-H opened a new pull request, #12291: URL: https://github.com/apache/arrow-rs/pull/12291
# Which issue does this PR close? Closes #11384. # Rationale for this change Callers can sample Arrow batches before opening a writer and disable dictionaries when the sampled first page would not benefit from them, without changing writer defaults. # What changes are included in this PR? Adds `choose_dictionary_encoding` and `DictionaryEncodingOptions` under `parquet::arrow::arrow_writer`. The helper returns adjusted `WriterProperties`, preserves other properties, and never enables disabled dictionaries. The first page-size check is configurable. For nonempty samples, Parquet v1 fixed-length byte-array dictionaries are disabled to match parquet-java, even though the Rust writer supports them. Adapted from [Comet's DictionaryChooser](https://github.com/apache/datafusion-comet/blob/916103f21a2f959e9bb70a25896df6a46cbbd746/native/core/src/execution/operators/iceberg_dictionary.rs). This approximates parquet-java's first-page decision: page boundaries and mixed-NaN behavior can differ. It targets the inferred Parquet schema, not a custom physical schema. Sampling and buffering remain caller-controlled; small samples can reject dictionaries useful over a full row group. # Are these changes tested? The 23 unit tests and public example cover parquet-java decisions recorded through Iceberg (ported from Comet's tests), physical conversions, column limits, first-check configuration, nested row boundaries and slices, encoded Arrow arrays, floating-point keys, property preservation, and dictionary-page metadata with complete file readback. # Are there any significant user-facing changes? New opt-in API only. Existing writer behavior is unchanged. # AI usage disclosure AI tools adapted the reference implementation and generated the helper, tests, and documentation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
