Rich-T-kid opened a new issue, #26814:
URL: https://github.com/apache/datafusion/issues/26814
Related to #26780.
Hash `RepartitionExec` owns one `BatchPartitioner` per input partition, but
each call to `partition_iter` invokes `create_hashes` from scratch. For a
dictionary column, that hashes every dictionary value before scattering through
the row keys.
Filtering, taking, and slicing commonly produce batches with different keys
that retain the same dictionary values `Arc`:
```text
shared dictionary values
|
+-- batch 1 keys -> hash all values -> scatter rows
+-- batch 2 keys -> hash all values -> scatter rows
+-- batch 3 keys -> hash all values -> scatter rows
```
The O(rows) scatter is required; recomputing the O(dictionary cardinality)
value hashes is not. ClickBench Q33 showed the amplification clearly: each
producer partition emitted roughly 232 slices sharing about 1.9 million URL
values, and summed repartition time increased from 0.50s to 100.66s with
dictionary encoding.
`BatchPartitioner` should cache dictionary value hashes per partitioning
expression while the values `Arc` and repartition hash state are unchanged. A
one-entry cache per expression is sufficient: replace it when the pointer
changes, so memory cannot grow with the number of batches or dictionaries
observed.
Completion criteria:
- batches sharing dictionary values hash those values once per input
partition
- a new values `Arc` replaces the cached entry
- composite hash expressions, null keys/values, slices, and all dictionary
key types remain correct
- cache allocation is bounded and visible in memory accounting
- tests directly count dictionary values hashed
- benchmark ClickBench Q10/Q11 and the Q33-shaped grouped/repartitioned path
with dictionary reads enabled on both sides
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
