Rich-T-kid opened a new pull request, #26315: URL: https://github.com/apache/datafusion/pull/26315
## Which issue does this PR close? - Closes #24113. - Part of #24111. ## Rationale for this change `enable_rle_to_dictionary` promotes eligible Parquet string and binary columns whenever dictionary pages are present. High-cardinality columns can therefore be inferred as large dictionary arrays without a user-defined bound. ## What changes are included in this PR? Adds `datafusion.execution.parquet.rle_to_dictionary_max_distinct_values`, an optional absolute NDV cap for dictionary read inference. When set, columns are promoted only when row-group NDV statistics prove they are within the cap; columns with missing statistics or values above the cap remain plain. The option is also wired through multi-file schema inference, protobuf, and documentation. ## What is the testing strategy for this PR? - Extended `parquet_rle_to_dictionary.slt` for missing, below-cap, and above-cap NDV behavior. - Added a protobuf round-trip test for the new option. - Ran focused crate checks and the repository pre-push format, Clippy, and typo checks. ## Are there any user-facing changes? Yes. Users can set `datafusion.execution.parquet.rle_to_dictionary_max_distinct_values` to an absolute distinct-value limit. The default is `NULL`, which preserves the existing dictionary inference behavior. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
