Rich-T-kid opened a new pull request, #26816: URL: https://github.com/apache/datafusion/pull/26816
## Why Parquet dictionary batches commonly share one values array and differ only in their keys. `RepartitionExec` currently hashes every dictionary value for every batch, so a large dictionary split into many batches repeats the same cardinality-sized work. Closes #26814. Related to #26780. ## What changed - cache dictionary value hashes per input partition and repartition expression - reuse the hashes while the values `Arc` is unchanged, then rebuild them when it changes - keep only a weak reference to the values array and account the retained hash buffers in the memory pool - fall back to the existing uncached path if the memory pool cannot retain the cache ## Results Upstream TPCH, TPC-DS, and partitioned ClickBench results with RLE dictionary reads enabled will be linked here after the run completes. ## Test coverage The new tests cover shared values across batches, null keys and values, a changed values `Arc`, composite hashes, and end-to-end hash partition routing. The existing repartition tests and forced-hash-collision build also pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
