Rich-T-kid opened a new pull request, #27479:
URL: https://github.com/apache/datafusion/pull/27479
## Which issue does this PR close?
- N/A. This performance opportunity was found while profiling ClickBench Q5.
## Rationale for this change
The inner aggregate for `COUNT(DISTINCT dictionary_column)` has no
accumulators,
but it currently interns one group key per input row. Repeated dictionary
keys
identify the same batch-local value, so this repeats group lookup work.
On ClickBench Q5 (`COUNT(DISTINCT SearchPhrase)`), a temporary trace recorded
12,312 eligible batches and reduced 99,997,497 input rows to 9,544,369 keys
before group interning, a 90.46% reduction.
A controlled local B-C-C-B comparison against `c177f5704`, with nine
iterations
per leg, measured:
| Leg | Median |
|---|---:|
| Base 1 | 611.97 ms |
| Candidate 1 | 557.78 ms |
| Candidate 2 | 551.47 ms |
| Base 2 | 558.45 ms |
The order-balanced median geomean improved by 5.13%. Excluding each leg's
first
iteration improved the median geomean by 5.62% and the mean by 6.01%.
Candidate
was faster in both pairings, and fair-pool peak was effectively unchanged
(-0.01%). A separate earlier clean run measured a 15.75% improvement.
## What changes are included in this PR?
For a zero-accumulator aggregate with exactly one dictionary group column,
this
reduces the input to one row per occupied dictionary key plus one null key.
Normal group interning still resolves duplicate logical values, null
dictionary
values, and null keys.
The existing path is retained unless dictionary metadata guarantees at least
a
25% reduction in rows.
## What is the testing strategy for this PR?
- Unit tests cover the production partial DISTINCT path, the 25% fallback
boundary, unsigned dictionary keys, duplicate logical values, null keys,
and
null dictionary values.
- `cargo xtask ci step test hash-collisions`
- Full local pre-push hook: formatting, workspace-wide Clippy with
integration
features, and typos.
- ClickBench Q5 correctness check comparing the dictionary expression with a
materialized plain-string expression.
- Hosted TPC-H, TPC-DS, and dictionary-enabled ClickBench runs will be linked
after completion.
## Are there any user-facing changes?
No API or behavior changes. Eligible DISTINCT aggregates do less group-key
work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]