andygrove commented on PR #2657: URL: https://github.com/apache/datafusion-ballista/pull/2657#issuecomment-6432248414
## TPC-H SF1000: pipelined vs blocking I ran TPC-H SF1000 with `ballista.execution.mode=pipelined` at 83edc876f, after rebasing onto main so that #2659 is included. All 22 queries complete. The total is 3795 s, 1.12x the blocking total. Most queries are faster, but a few regress badly: Q8 is 8x slower and Q9 3x. **Setup:** 2 executors on 2 nodes, each with 8 vCPU, a 56 GiB pod limit, `--memory-pool-size=48GB` and `--max-temp-directory-size=1TB`. `target_partitions=16`, Parquet read from node-local disk, 1 iteration. **Caveat:** the blocking column is an earlier run on the same cluster, built from the #2659 branch rather than this PR's image, so treat the comparison as approximate. An earlier pipelined run, on this branch before the rebase, failed Q5, Q8, Q9, Q17, Q18 and Q21. Tasks exceeded DataFusion's default 100 GB spill limit, which executors could not raise before #2659. Pipelined mode holds every stage's state at the same time, so it spills more than blocking does. That makes `--max-temp-directory-size` effectively required for pipelined runs at this scale. | Query | Blocking (s) | Pipelined (s) | Pipelined / blocking | |---|---:|---:|---:| | 1 | 27.2 | 26.4 | 0.97x | | 2 | 51.2 | 98.4 | **1.92x** | | 3 | 137.4 | 135.0 | 0.98x | | 4 | 51.4 | 46.8 | 0.91x | | 5 | 430.0 | 344.3 | 0.80x | | 6 | 12.6 | 8.6 | 0.69x | | 7 | 374.4 | 360.8 | 0.96x | | 8 | 45.0 | 361.2 | **8.03x** | | 9 | 199.6 | 598.5 | **3.00x** | | 10 | 136.6 | 109.5 | 0.80x | | 11 | 55.5 | 85.5 | **1.54x** | | 12 | 73.4 | 53.6 | 0.73x | | 13 | 69.9 | 41.5 | 0.59x | | 14 | 19.0 | 15.0 | 0.79x | | 15 | 43.0 | 32.2 | 0.75x | | 16 | 23.8 | 16.0 | 0.67x | | 17 | 225.9 | 374.8 | **1.66x** | | 18 | 459.7 | 409.2 | 0.89x | | 19 | 26.2 | 22.6 | 0.86x | | 20 | 63.8 | 57.6 | 0.90x | | 21 | 817.1 | 575.4 | 0.70x | | 22 | 31.6 | 22.1 | 0.70x | | **Total** | **3374** | **3795** | **1.12x** | 17 of 22 queries run at 0.59–0.98x of blocking. Q8 and Q9 alone add about 715 s, so they are the ones to look at next. I have scheduler event logs for every query in this run. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
