Rich-T-kid opened a new pull request, #26825: URL: https://github.com/apache/datafusion/pull/26825
WIP: hosted benchmark results are pending. ## Why With Parquet RLE-to-dictionary enabled, consecutive batches can share the same dictionary values. `HashJoinExec` currently hashes those values again for every build and probe batch. This addresses the hash-join part of #26780 and closes #26815. It depends on the small hash helpers in #26817; #26816 shows the potential upside of reuse, but this PR intentionally uses a narrower design. ## What changed A small join-local cache keeps only the last dictionary values `Arc` and its hashes for each join key. Pointer equality identifies shared values; a different values array is hashed normally and replaces the entry. Build and probe state each own their cache, so the fixed hash state is implicit in the cache lifetime. Non-dictionary columns and collision checking are unchanged. ## Results Pending hosted TPC-H, TPC-DS, and partitioned ClickBench comparisons with RLE-to-dictionary enabled on both sides. The baseline will be the exact #26817 head (`4ede5833`), so the comparison isolates this PR. ## Test coverage - focused shared-values reuse test, including different key mappings and null keys - all 652 hash-join tests in normal and forced-collision modes - repository pre-push format, workspace Clippy, and typo checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
