Rich-T-kid opened a new pull request, #26789: URL: https://github.com/apache/datafusion/pull/26789
# 100% ai generated have not reviewed yet > [!IMPORTANT] > **WIP:** This is still a draft while the thresholds and broader ClickBench coverage are reviewed. Closes #24116. Closes #24117. ## Why Parquet dictionary reads can help low-cardinality group keys, but carrying a large dictionary through partial aggregation, hash repartitioning, and final aggregation is expensive. Absolute distinct count is not enough: the useful signal is how often each dictionary value is reused, how wide the values are, and whether the plan actually consumes the column as a key. ## What changed This is intentionally a one-file physical optimizer change in `datafusion/physical-optimizer/src/optimizer.rs`. The rule only considers direct `Dictionary<Int32, Utf8/Binary>` group keys. It keeps dictionaries when exact statistics show at least 64 rows per distinct value and values at least 16 bytes wide. When Parquet NDV is unavailable, it uses source encoded-byte density to keep compact dictionaries and decode dense, wide ones. Groups containing a tiny dimension dictionary stay entirely dictionary encoded, avoiding the mixed-key regression seen on TPC-DS Q72. Unresolved cases decode before single or partial aggregation, stay decoded through repartition and final aggregation, and restore the original dictionary output schema above the aggregate. Non-dictionary inputs are unchanged, and the rewrite is idempotent. ## Results Hosted benchmarks compare main `c18e5d974` with this PR at `523969b7c`, with dictionary reads enabled on both sides. | Suite | Faster | Slower | Result | |---|---:|---:|---| | [ClickBench partitioned](https://github.com/apache/datafusion/pull/26789#issuecomment-6449912064) | 12 | 0 | 72.51s -> 35.00s | | [TPC-H](https://github.com/apache/datafusion/pull/26789#issuecomment-6449689235) | 1 | 0 | Q10 1.23x faster | | [TPC-DS](https://github.com/apache/datafusion/pull/26789#issuecomment-6450299363) | 3 | 0 | Q4 1.27x, Q11 1.37x, Q74 1.17x faster | | [Wide schema](https://github.com/apache/datafusion/pull/26789#issuecomment-6450080428) | 0 | 0 | neutral | ## Test coverage - Cost boundaries for reuse, value width, source byte density, tiny dimension dictionaries, and missing statistics. - Single-stage and partial/final aggregate rewrites, including output-schema preservation. - Execution coverage for dictionary-to-value grouping and dictionary output restoration. - All 48 `datafusion-physical-optimizer` tests pass. - Full pre-push formatting, workspace Clippy, typos, release build, and diff checks pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
