andygrove commented on PR #2657: URL: https://github.com/apache/datafusion-ballista/pull/2657#issuecomment-6439253246
## SF1000 results with size-aware join selection (34a1151a7) Following up on the slow-query analysis above: pipelined jobs now plan with hash joins and `PipelinedJoinSelection` demotes a partitioned hash join to a sort-merge join when its estimated build side per partition exceeds `ballista.hash_join_max_build_partition_bytes` (64 MiB default) or the size is unknown. `CollectLeft` joins are left alone, and a limit of 0 disables the check. Same setup as before: 2 executors x 8 vCPU, 48 GB pool, 1 TB spill cap, 16 partitions, local data, one iteration. All 22 queries pass. Total **2450s**, down from 3795s for pipelined before this change, and **0.73x** the 3374s blocking baseline. | Query | Blocking (s) | Pipelined before (s) | Pipelined + join selection (s) | vs blocking | |---|---:|---:|---:|---:| | 1 | 27.2 | 26.4 | 26.6 | 0.98x | | 2 | 51.2 | 98.4 | 26.3 | 0.51x | | 3 | 137.4 | 135.0 | 120.3 | 0.88x | | 4 | 51.4 | 46.8 | 16.4 | 0.32x | | 5 | 430.0 | 344.3 | 284.4 | 0.66x | | 6 | 12.6 | 8.6 | 9.2 | 0.73x | | 7 | 374.4 | 360.8 | 248.4 | 0.66x | | 8 | 45.0 | 361.2 | 151.4 | 3.37x | | 9 | 199.6 | 598.5 | 246.5 | 1.24x | | 10 | 136.6 | 109.5 | 94.5 | 0.69x | | 11 | 55.5 | 85.5 | 23.5 | 0.42x | | 12 | 73.4 | 53.6 | 49.1 | 0.67x | | 13 | 69.9 | 41.5 | 41.5 | 0.59x | | 14 | 19.0 | 15.0 | 14.9 | 0.78x | | 15 | 43.0 | 32.2 | 26.1 | 0.61x | | 16 | 23.8 | 16.0 | 10.0 | 0.42x | | 17 | 225.9 | 374.8 | 157.4 | 0.70x | | 18 | 459.7 | 409.2 | 360.7 | 0.78x | | 19 | 26.2 | 22.6 | 22.1 | 0.84x | | 20 | 63.8 | 57.6 | 49.3 | 0.77x | | 21 | 817.1 | 575.4 | 423.2 | 0.52x | | 22 | 31.6 | 22.1 | 48.1 | 1.52x | | **Total** | **3374** | **3795** | **2450** | **0.73x** | Notes: - Q9 no longer OOMs the way it did with a global `prefer_hash_join=true`. The large `orders` build is demoted to SMJ, and the small dimension joins stay as hash joins. - Q8 is still 3.4x blocking. In blocking mode AQE sees the actual size of the filtered `part` side and broadcasts it. The static estimate is too large to broadcast, so `lineitem` is shuffled over the network. Better static estimates, or runtime join adaptation within pipelined mode, would be needed to close this gap. - The Q22 plan is unchanged from the previous pipelined run, so the 22s to 48s change looks like run-to-run noise and not an effect of this change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
