Rich-T-kid opened a new pull request, #27341: URL: https://github.com/apache/datafusion/pull/27341
## Which issue does this PR close? - Related to #24822. ## Rationale for this change Parquet dictionaries are scoped to a row group, so the row-group row count is a safe upper bound for dictionary cardinality. This experiment uses that bound to avoid Int32 keys for small files. ## What changes are included in this PR? RLE dictionary inference now chooses Int8, Int16, or Int32 keys from the largest row group and keeps the existing cross-file widening behavior. ## What is the testing strategy for this PR? A three-run RLE-dictionary TPC-DS comparison regressed by 0.97% on summed query medians and 1.72% on geometric mean. This remains a draft experiment; hosted TPC-H, TPC-DS, and partitioned ClickBench results are pending. ## Are there any user-facing changes? With RLE-to-dictionary reads enabled, inferred dictionary key types can now be narrower than Int32. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
