Rich-T-kid opened a new issue, #26815:
URL: https://github.com/apache/datafusion/issues/26815
Related to #26780.
Dictionary join keys can be hashed repeatedly as they pass through
partitioning and the join itself:
```text
dictionary input
|
+-- RepartitionExec routing hashes
|
`-- HashJoinExec build/probe hashes
```
Within `HashJoinExec`, consecutive batches also commonly share one
dictionary values `Arc` after scans, filters, or takes. Build and probe hashing
currently call `create_hashes` per batch, so each call hashes the complete
values array again before scattering through the keys.
Hash joins should cache dictionary value hashes independently for build and
probe streams, keyed by values-array identity and the exact hash state.
Repartition hashes cannot be reused unless their seed is compatible; different
hash states require separate entries. Equality checks must continue to resolve
collisions normally.
Joins between dictionaries with different mappings need explicit coverage:
equal logical values can have different keys and dictionary orderings, so
cached hashes must always be derived from values rather than raw keys.
Completion criteria:
- repeated build or probe batches sharing a values `Arc` hash the values
once per compatible hash state
- changed values arrays and different hash states do not reuse stale hashes
- tests cover different dictionary mappings, nulls, slices, multiple key
types, and forced hash collisions
- cache memory is bounded and accounted to the join
- partitioned and collect-left hash-join benchmarks cover low and medium
cardinalities
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
