mohitgurav20 commented on PR #25716: URL: https://github.com/apache/datafusion/pull/25716#issuecomment-6152606554
Thanks for running the benchmarks with both pushdown configurations, @adriangb! The numbers are clear — at SF1, it's a flat `+0.0%` with pushdown enabled and well within noise across the board. The batch count at this scale simply isn't large enough for duplicate buffer references to outweigh the hashing and remapping cost in `HashJoinExec`. As agreed earlier with @alamb, without a clear representative payoff, adding this complexity to the hash join build path isn't justified. Closing this out. If this ever surfaces as a measurable bottleneck on larger scale factors (SF100+) or high batch counts, we can revisit whether handling it upstream in Arrow's `concat` kernel makes more sense. Thanks for the guidance and reviews, @alamb and @adriangb! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
