andygrove opened a new issue, #1959: URL: https://github.com/apache/datafusion-ballista/issues/1959
Follow-up to #1944, adding the dimension that turns out to explain most of the Ballista↔Comet gap: **join strategy (SortMergeJoin vs ShuffledHashJoin)**. #1944 compared the three engines at their *defaults* — where Ballista uses SMJ and Comet uses SHJ — so it was partly comparing plans, not engines. This snapshot runs **both** strategies on **all three** engines. > Informational snapshot, not a bug report. Supersedes #1944. ## Setup - **Cluster:** 2-node k3s, each 32 cores / 128 GiB. **2 executors × 8 cores (16 task slots)**, sequential runs (idle engines scaled to 0 — no I/O contention). - **Data:** TPC-H **SF100** Parquet in MinIO (S3); `s3://` for Ballista, `s3a://` for Spark/Comet. Identical 22-query set; all row counts match. - **Ballista:** `main` + in-flight PRs (shuffle in-flight governor #1951, broadcast-join promotion, exchange reuse #1953). DataFusion 54. **Static planner.** `target_partitions=32`. - **Comet:** 1.0.0-SNAPSHOT on Spark 3.5. **Vanilla Spark:** 3.5 with Comet disabled (`spark.comet.enabled=false`, default sort shuffle manager). - **Join-strategy toggles:** Ballista `datafusion.optimizer.prefer_hash_join`; Comet `spark.comet.exec.replaceSortMergeJoin`; Spark `spark.sql.join.preferSortMergeJoin`. - **Method:** single iteration, no warmup. ## Totals — the headline | Engine | SortMergeJoin | ShuffledHashJoin | Δ from hash join | |---|---:|---:|---:| | **Ballista** | 560 s | **398 s** | **−29 %** | | **Comet** | 356 s | 347 s | −3 % | | **Vanilla Spark** | 586 s | 590 s | +1 % | **Matched strategy:** | | Ballista | Comet | Vanilla Spark | |---|---:|---:|---:| | SortMergeJoin | 560 | 356 | 586 | | ShuffledHashJoin | **398** | **347** | 590 | ## Observations 1. **Enabling hash join is a −29 % win for Ballista, but a no-op for Comet (−3 %) and Spark (+1 %).** Ballista's SortMergeJoin sorts both sides of every large join (~963 M rows — ~1.6× the fact table — for Q9 alone); those sorts dominate its time. Comet's columnar sorts and Spark's row-based sorts are comparatively cheap, so the strategy toggle barely moves them. 2. **At matched strategy, Ballista is within ~15 % of Comet** (398 vs 347 s, 1.15×) and **~1.5× faster than vanilla Spark** (398 vs 590 s). Most of the headline Ballista-vs-Comet gap reported in #1944 was Ballista running its SMJ default against Comet's SHJ default — a config difference, not an engine difference. 3. **Ballista SHJ ran all 22 queries at SF100 with no OOM** (2×8, 28 GB pool). `prefer_hash_join=false` is a conservative default because DataFusion's hash join cannot spill (#1648); at this scale that default costs ~29 %. 4. The residual matched-SHJ gap to Comet is **query-dependent**: Ballista is *faster* on several (Q1 7.1 vs 10.4, Q4, Q8, Q17, Q22) and Comet is 2×+ on a few (**Q7, Q14, Q19**). Those outliers — not the average — are where the remaining execution/shuffle difference lives. Attributing them needs shuffle-read metrics that Ballista does not yet expose (#1958). 5. **Vanilla Spark is the slowest of the three** regardless of strategy (590–586 s); both Ballista and Comet are comfortably ahead. ## Per-query (seconds) | Q | Bal SMJ | Bal SHJ | Comet SMJ | Comet SHJ | Spark SMJ | Spark SHJ | |---|---:|---:|---:|---:|---:|---:| | Q1 | 6.6 | 7.1 | 11.0 | 10.4 | 60.6 | 60.0 | | Q2 | 10.1 | 7.8 | 8.6 | 6.9 | 10.8 | 10.2 | | Q3 | 17.7 | 13.3 | 14.9 | 13.5 | 23.9 | 23.2 | | Q4 | 12.1 | 6.4 | 7.5 | 7.7 | 17.0 | 17.2 | | Q5 | 45.0 | 30.4 | 28.6 | 24.1 | 37.2 | 33.7 | | Q6 | 6.2 | 6.0 | 3.7 | 3.7 | 11.1 | 11.2 | | Q7 | 43.8 | 32.5 | 17.2 | 15.7 | 24.7 | 25.3 | | Q8 | 53.8 | 30.3 | 32.3 | 32.8 | 32.3 | 32.2 | | Q9 | 77.1 | 45.7 | 45.5 | 42.7 | 56.0 | 57.8 | | Q10 | 17.2 | 14.8 | 14.8 | 13.6 | 22.3 | 22.1 | | Q11 | 8.3 | 7.3 | 7.7 | 7.1 | 10.2 | 10.1 | | Q12 | 10.5 | 8.3 | 8.8 | 8.0 | 15.9 | 15.9 | | Q13 | 8.4 | 7.7 | 7.4 | 7.1 | 14.1 | 16.9 | | Q14 | 14.9 | 14.8 | 7.1 | 6.5 | 12.6 | 12.6 | | Q15 | 11.1 | 11.2 | 11.3 | 11.2 | 25.9 | 26.9 | | Q16 | 3.3 | 3.0 | 3.2 | 3.2 | 5.5 | 5.5 | | Q17 | 48.6 | 35.4 | 38.4 | 39.3 | 60.7 | 60.4 | | Q18 | 48.4 | 28.8 | 23.6 | 24.7 | 47.0 | 46.6 | | Q19 | 16.9 | 17.3 | 8.0 | 8.0 | 15.1 | 15.0 | | Q20 | 17.6 | 15.7 | 9.6 | 9.5 | 16.7 | 16.5 | | Q21 | 77.9 | 50.9 | 42.5 | 47.2 | 58.2 | 62.1 | | Q22 | 4.4 | 3.5 | 4.3 | 3.9 | 8.2 | 8.8 | | **Total** | **560** | **398** | **356** | **347** | **586** | **590** | ## Caveats Single iteration on a shared homelab cluster carries ~5–10 % run-to-run noise (disk/warm-up effects). The −29 % Ballista hash-join effect and the matched-strategy ordering are well above that; the per-query 2× outliers (Q7/Q14/Q19) deserve multi-iteration confirmation before drawing conclusions from them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
