Rich-T-kid opened a new pull request, #27051:
URL: https://github.com/apache/datafusion/pull/27051
## Why
`DictionaryGroupValuesColumn` stores each group as an index into its
deduplicated values. Emitting a dictionary currently walks every group again to
convert those indices to dictionary keys. If the input contains nulls, it also
removes the null value with `take` and remaps the keys.
```text
Before
group slots: [0, 2, 1, 2] values: ["a", null, "b"]
| |
| convert + remap | take out null
+---------------+--------------+
|
validate every key
|
v
DictionaryArray
```
This made aggregate emission noticeably more expensive when Parquet RLE
values were preserved as dictionaries.
## What changed
The group mapping now uses the dictionary's native key type and keeps nulls
in a validity buffer. Nulls no longer occupy a slot in the values array, so
full emission can move the existing buffers directly into the result. Key
bounds are checked when groups are inserted, which makes a second validation
pass during emission redundant.
```text
After
keys: [0, 1, 0, 1] --------+
validity: [1, 1, 0, 1] --------+-- move --> DictionaryArray
values: ["a", "b"] ----------+
no conversion, remap, take, or validation pass
```
Partial emission still compacts the referenced values because the remaining
groups may use a sparse subset. The benchmark added here isolates
`GroupValues::emit` from interning.
## Results
The focused benchmark uses all-unique values and times only `EmitTo::All`:
| Groups | main | this PR | Change |
|---:|---:|---:|---:|
| 8,192 | 10.12 µs | 0.93 µs | -90.8% |
| 65,536 | 64.93 µs | 1.92 µs | -97.0% |
| 500,000 | 504.04 µs | 7.48 µs | -98.5% |
On TPC-DS SF1, I ran three order-balanced passes of all 99 queries with 16
partitions. Times below are medians from the `AggregateExec` metrics; "group
emission" subtracts accumulator state/evaluation time from `emitting_time`.
| RLE dictionary conversion | main group emission | this PR group emission |
|---|---:|---:|
| disabled | 3.44 ms | 3.23 ms |
| enabled | 113.95 ms | 5.81 ms |
With dictionary conversion enabled, group emission is 94.9% lower. The
remaining difference from the disabled/Utf8View path is 2.58 ms across the full
suite.
## Test coverage
- Full `datafusion-physical-plan` unit suite: 2,923 passed
- Multi-column group-value tests with forced hash collisions: 119 passed
- Dictionary tests cover null keys, null dictionary values, narrow-key
capacity, partial emission, selected values, and cache invalidation
- `cargo fmt --all -- --check`
- `cargo clippy -p datafusion-physical-plan --lib --benches -- -D warnings`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
