Rich-T-kid opened a new pull request, #27262:
URL: https://github.com/apache/datafusion/pull/27262
## Which issue does this PR close?
- Related to #26812.
## Rationale for this change
Single dictionary group keys can avoid the general aggregate hash table
while batches share a values array. The current path still hashes/copies values
and can hit a sharp allocation boundary at high cardinality; sparse
dictionaries also pay for values never referenced by their keys.
## What changes are included in this PR?
```text
shared dictionary values
|
referenced keys only -> hash/equality once
|
key -> group cache -> group IDs
|
values Arc changes -> existing general column path
```
- Adds a specialized one-dictionary-key group store with direct key-to-group
lookup.
- Reserves the value table only when the first batch demonstrates a dense
dictionary, avoiding the cardinality cliff without over-allocating sparse
dictionaries.
- Hashes and compares only newly referenced dictionary values.
This is separate from #26826: that PR retains the general/multi-column
hash-cache optimization; this PR specializes the single-column
stable-dictionary case and falls back to the general path when values change.
## What is the testing strategy for this PR?
Unit tests cover shared and changed dictionaries, duplicate and unreferenced
values, nulls, partial emission, preserved selections, float normalization,
bounded reserve behavior, and forced hash collisions.
Clean isolated A/B benchmarks against main `b8534c7fd` (1M rows, median of 7
balanced-order runs; subsequent main commits do not touch this path):
| Case | Main | This PR | Change |
|---|---:|---:|---:|
| 524,288 NDV | 95.09 ms | 17.15 ms | -82.0% |
| 525,000 NDV | 44.83 ms | 18.40 ms | -59.0% |
| 10 referenced / 525K values | 6.77 ms | 1.83 ms | -73.0% |
The repository Criterion benchmark improved by 62.7%-70.7% at 100K-500K
dictionary values. Existing changed-dictionary/fallback cases were also
35.9%-45.1% faster at higher cardinalities; no benchmark case regressed.
Also passed `cargo fmt`, workspace clippy (all targets/features), the
extended workspace test suite, and the pre-push checks.
## Are there any user-facing changes?
No; this changes only the physical aggregation implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
