Rich-T-kid opened a new issue, #26813:
URL: https://github.com/apache/datafusion/issues/26813
Related to #26780.
Partial hash aggregation can emit its materialized groups in multiple
slices. Slicing a dictionary rewrites the keys but retains the full values
array:
```text
partial aggregate dictionary
|
+-- slice 1 --+
+-- slice 2 --+--> FinalAggregate
+-- slice 3 --+ `-- hashes the same values for every slice
```
`DictionaryGroupValuesColumn` already reuses some lookup state while the
values `Arc` is unchanged, but final aggregation's outer group-ID hashing still
calls `create_hashes` for every incoming slice. The work therefore remains
O(slices x dictionary cardinality) before the rows are scattered through their
keys.
Final aggregation should retain dictionary value hashes across input batches
and recompute them only when the values-array identity or hash state changes.
The state should live with the final aggregation stream so independent
executions do not share mutable caches.
Completion criteria:
- slices sharing a values `Arc` hash the dictionary values once
- replacing the values `Arc` invalidates the cached hashes
- composite group keys, null keys/values, and multiple dictionary key types
remain correct
- cache memory is bounded and accounted
- a `PartialAggregate -> Repartition -> FinalAggregate` test records the
number of dictionary values hashed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
