NoahKusaba commented on PR #2501: URL: https://github.com/apache/datafusion-ballista/pull/2501#issuecomment-5954996918
> Thanks for tracking this down. The q8 stage 5 trace makes the problem really clear. Since this changes how many tasks a swapped join runs with, and AQE is on by default, could you share before/after TPC-H numbers against `main`? Something like: > > ```shell > cargo run --release --bin tpch -- benchmark ballista \ > --host localhost --port 50050 \ > --iterations 3 --path <tpch parquet dir> --format parquet > ``` > > SF100 would be great, or the SF1000 setup where you saw q8. A second run with `-c ballista.scheduler.max_partitions_per_task=1` would help too, since that's where the build side gets copied into the most tasks. Could you also include the q8 stage 5 plan before and after, like the snippet in the description? Thanks for the review. I currently don't have the hardware to run SF 100, and the rewrite rule doesn't occur for sf10. I'll look to get a cluster sometime in the next week to better benchmark and document the new behavior of this branch + the shuffle-affinity PR I made. Marking as draft for the time being. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
