andygrove opened a new issue, #1944:
URL: https://github.com/apache/datafusion-ballista/issues/1944
Sharing a performance **snapshot** of Ballista vs Spark and Comet on TPC-H @
SF100, run on a
small homelab Kubernetes cluster with data in MinIO (S3). This is an
informational data point,
**not a bug report or a request for changes** — hopefully a useful reference
on where Ballista
stands today. A few observations at the end.
## Setup
- **Cluster:** 2-node k3s, each node 32 cores / 128 GiB. Ballista: 2
executors × 8 cores
(16 task slots), scheduler + executors in-cluster.
- **Data:** TPC-H SF100, Parquet, in MinIO (S3-compatible), read over
`s3://` (Ballista) /
`s3a://` (Spark/Comet).
- **Engines:**
- **Ballista** from `main` (DataFusion 54). S3 read support for the `tpch`
benchmark came
from an in-flight PR that lets `--path` accept `s3://`.
- **Comet** 1.0.0-SNAPSHOT on Spark 3.5.3.
- **Vanilla Spark** 3.5.3 (Comet disabled) as a reference point.
- **Queries:** the SQLBench-H TPC-H queries from
`apache/datafusion-benchmarks` — the
**identical** set across all three engines (verified: all 22 row counts
match).
- **Method:** single iteration, no warmup; each engine run **sequentially**
(no I/O
contention); identical 2×8 shape.
## Engine comparison — SF100 total
| Engine | Total (s) |
|---|---:|
| **Comet** (Spark 3.5 + Comet) | **340** |
| **Ballista** @ `target_partitions=32` (best) | **552** |
| Vanilla Spark 3.5 | 593 |
| Ballista @ `target_partitions=16` | 651 |
| Ballista @ `target_partitions=2` (tpch default) | 1954 |
Per-query (seconds):
| Q | Comet | Spark | Ballista@32 |
|---|------:|------:|------------:|
| Q1 | 10.6 | 68.8 | 6.3 |
| Q2 | 6.7 | 11.1 | 10.4 |
| Q3 | 13.5 | 24.1 | 22.1 |
| Q4 | 7.6 | 16.1 | 10.2 |
| Q5 | 25.2 | 37.0 | 40.5 |
| Q6 | 3.6 | 11.1 | 5.9 |
| Q7 | 15.8 | 24.9 | 46.1 |
| Q8 | 30.9 | 31.8 | 49.7 |
| Q9 | 41.0 | 55.8 | 60.5 |
| Q10 | 13.6 | 22.5 | 25.6 |
| Q11 | 6.8 | 10.2 | 10.7 |
| Q12 | 7.8 | 15.7 | 9.5 |
| Q13 | 6.8 | 13.6 | 12.4 |
| Q14 | 6.6 | 12.4 | 17.1 |
| Q15 | 11.4 | 26.3 | 16.0 |
| Q16 | 3.1 | 5.6 | 8.0 |
| Q17 | 37.7 | 60.8 | 45.5 |
| Q18 | 22.6 | 48.0 | 42.1 |
| Q19 | 8.4 | 15.0 | 19.0 |
| Q20 | 9.8 | 16.4 | 19.3 |
| Q21 | 46.5 | 57.0 | 69.2 |
| Q22 | 3.9 | 8.8 | 5.6 |
| **Total** | **340** | **593** | **552** |
## `target_partitions` sweep (Ballista)
The `tpch` benchmark defaults to `--partitions 2`, which under-parallelizes
a 16-core cluster.
Sweeping it:
| target_partitions | Total (s) | note |
|---:|---:|---|
| 2 | 1954 | tpch default — severe under-parallelization |
| 16 | 651 | = task slots |
| **32** | **552** | **best (2× task slots)** |
| 64 | 577 | past the knee |
| 128 | failed | shuffle-fetch port exhaustion |
| 256 | failed | shuffle-fetch port exhaustion |
The 128/256 failures are filed separately as #1943 (shuffle opens a
connection per fetch when
client caching is off, which is the default).
## Observations
1. **`target_partitions` is the dominant knob here.** The tpch default of 2
costs ~3.5× vs the
sweet spot on this cluster; matching or slightly exceeding total task
slots (best at 2×
slots = 32) is what closes most of the gap.
2. **Once parallelized, Ballista is competitive with vanilla Spark** (552 vs
593 s overall),
and they trade wins per query — Ballista is notably faster on some (Q1
6.3 vs 68.8 s, also
Q6/Q12/Q15/Q22), Spark faster on several join/shuffle-heavy ones (Q7, Q8,
Q14).
3. **Comet's edge is mostly its ~1.7× acceleration over Spark** (vectorized
execution + native
shuffle), rather than a large engine-quality gap with Ballista.
4. **Caveats:** single iteration / no warmup; small homelab hardware; MinIO
network I/O; a
single scale factor (SF100); Ballista S3 read support via an in-flight
PR. Absolute numbers
should be taken with a grain of salt — the relative picture is the
interesting part.
Posting in case it's useful; happy to share more detail or re-run with
different parameters.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
