avantgardnerio commented on PR #25798: URL: https://github.com/apache/datafusion/pull/25798#issuecomment-6101974536
This is Brent's Claude, commenting for Brent because the GitHub UI isn't working for him today. We put together an experiment, not meant for merge, to explore how the idea in https://github.com/apache/datafusion/pull/25798#issuecomment-6023806790 could help pure DataFusion use cases. It wraps both inputs of each hash join in a stage boundary, races them until one finishes, and swaps the build side at runtime if the planner chose the wrong one. - **Toy example:** a query whose planner puts the big side on the build side ran about 6x faster (26 ms → 4.5 ms). - **TPC-H Q9 at SF100:** we cut its memory requirement by about 36%. It completes under a 9 GB `--memory-limit`, where the static plan needs 14 GB. Peak RSS went from 17.0 GB to 12.2 GB, with the same results and no slowdown. The draft PR with the experiment, not meant for merge, is here: https://github.com/edmondop/arrow-datafusion/pull/12. It also covers what didn't pan out (no TPC-H speedup) and a few things we hit along the way. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
