jayzhan211 commented on PR #26816: URL: https://github.com/apache/datafusion/pull/26816#issuecomment-6705841679
@Rich-T-kid since you asked about merging this one: #26703 has merged, and it affects this PR. It now gives each aggregate output slice with more values than rows only the values its rows reference ([ClickBench partitioned 78.8 s to 42.8 s](https://github.com/apache/datafusion/pull/26703#issuecomment-6703599262)). Those slices then have their own values `Arc`, so `CachingArrayHasher` would miss on them, and I expect most of this PR's ClickBench gain to overlap. I haven't measured the two together. The cache should still help where batches share a values `Arc` and reach the repartition directly, such as a scan hashed on a dictionary key. #26826 doesn't touch the repartition, so it only needs the helper. Could you rebase this on main and rerun `clickbench_partitioned`? That will show whether the repartition change still pays for itself, or whether the helper should move into #26826. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
