Rich-T-kid opened a new issue, #26812:
URL: https://github.com/apache/datafusion/issues/26812
Related to #26780.
`PartialAggregate` currently hashes dictionary values in two places:
```text
input DictionaryArray
|
+-- MultiGroupBy::intern -> create_hashes(values)
|
`-- DictionaryGroupValuesColumn -> hashes values for its lookup
```
Both paths use the aggregation hash seed. The dictionary keys may change
between batches while the values `Arc` remains the same, so the same values are
hashed again even though only the O(rows) key scatter is new work.
This is especially visible for grouped dictionary columns produced directly
by Parquet scans. Low cardinality keeps the waste small, while medium or high
cardinality makes it significant across many input batches.
The partial aggregate should compute dictionary value hashes once per
values-array identity and hash state, then reuse them for both group-ID
calculation and `DictionaryGroupValuesColumn` lookup. Composite group keys
still need their normal per-row hash combination.
Completion criteria:
- one values array is hashed once per compatible aggregation hash state
- both group-ID calculation and dictionary value lookup reuse those hashes
- the cache is bounded and included in aggregation memory accounting
- tests cover multiple batches sharing values, a changed values `Arc`, null
keys/values, and composite group keys
- instrumentation verifies values hashed rather than inferring reuse only
from elapsed time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
