Rich-T-kid opened a new pull request, #26789: URL: https://github.com/apache/datafusion/pull/26789
> [!IMPORTANT] > **WIP:** This is stacked on #26315. The fail-closed implementation is working, but the standard ClickBench dataset does not currently contain 5–10 safely eligible queries. Keep this as a draft until that benchmark target is met or the dataset/writer setup is adjusted. ## Which issue does this PR close? - Closes #24116. - Closes #24117. - Depends on #26315; its five commits are included in this branch and should be dropped after that PR merges. - Builds on @yinli-systems earlier group-by prototype: https://github.com/yinli-systems/datafusion/commit/2bdf91d9f ## Why An absolute NDV cap cannot distinguish 2,000 values over one million rows from 2,000 over 10,000 rows, cannot account for Utf8View/BinaryView inline values, and cannot tell whether a downstream operator can consume dictionary keys. This adds the missing per-row-group reuse/width evidence and plan-consumer check without inserting a runtime dictionary cast. This follows the design direction in https://github.com/apache/datafusion/issues/26696#issuecomment-6424987483. ## What changed - Add opt-in `datafusion.optimizer.enable_dictionary_optimization` with conservative defaults of 64 rows per dictionary entry and 16 average bytes per dictionary value. - Read bounded Parquet dictionary-page headers only when footer encoding stats prove every data page is dictionary encoded; derive worst-row-group reuse and width and fail closed on missing, malformed, overflowed, or fallback metadata. - Preserve native dictionaries selectively for direct equality/IN filter keys, grouped keys, and equi-join keys; restore the original SQL-visible schema above the consumer. - Carry the physical facts through listing metadata and caches, including cache entries populated before the option is enabled. - Add configuration and upgrade documentation plus Utf8View/BinaryView end-to-end coverage. ## Results The standard ClickBench dataset used here has 100 Parquet files, 99,997,497 rows, and about 14 GB of data. With the default 64-row/16-byte gates, it selects no query: wide/high-NDV keys have PLAIN fallback in at least one row group, while the fully dictionary-encoded `MobilePhoneModel` key fails the 16-byte width gate. The requested improvement in at least 5–10 standard queries is therefore **not met yet**, which is why this PR is WIP. As a control, I lowered only the average-width threshold to 1 byte to force the real Q10/Q11 BinaryView reader and aggregate path. These are release/LTO runs with 12 target partitions, one warmup pair, then 10 samples in alternating order; Q11 reverses the order. | Query | Plain median | Forced dictionary median | Difference | | --- | ---: | ---: | ---: | | Q10 | 47.5 ms | 52.0 ms | 9.5% slower | | Q11 | 55.0 ms | 58.0 ms | 5.5% slower | The control supports retaining the 16-byte fail-closed default. The next benchmark step is a ClickBench writer/dataset configuration that retains dictionary encoding for enough wide, reusable group/filter/join keys to evaluate the 5–10-query target. ## Test coverage - `cargo check -p datafusion-physical-optimizer -p datafusion-datasource-parquet -p datafusion` - Parquet dictionary-page parser and worst-row-group aggregation tests - Parquet selective dictionary pushdown tests - Physical optimizer filter/group/join/fail-closed tests - End-to-end Parquet test covering equality, IN, filtered group-by, semi-join, cache refresh, projection-only, Utf8View, and BinaryView - `cargo test -p datafusion-sqllogictest --test sqllogictests -- information_schema` (4/4) - `cargo clippy --all-targets --all-features -- -D warnings` - `cargo fmt --all -- --check`, `git diff --check`, generated config docs, and Markdown prettier - Clean merge-tree against current `upstream/main` ## Are there user-facing changes? Yes. This adds three opt-in optimizer settings. Dictionary optimization remains disabled by default. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
