This is an automated email from the ASF dual-hosted git repository.
andygrove pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-ballista.git
The following commit(s) were added to refs/heads/main by this push:
new d738ca9bd docs: refresh benchmarking.md with current SF1000 results
(Ballista main @ 696ca29b, Spark 3.4 baseline) (#2161)
d738ca9bd is described below
commit d738ca9bd6b86aecaf9587af9025ae0dd12c5af1
Author: Andy Grove <[email protected]>
AuthorDate: Thu Jul 23 16:25:57 2026 -0600
docs: refresh benchmarking.md with current SF1000 results (Ballista main @
696ca29b, Spark 3.4 baseline) (#2161)
* docs: refresh benchmarking.md with current SF1000 Ballista results
Replace the homelab-era result set with a fresh TPC-H SF1000 run captured
on a 32-executor Kubernetes cluster with data on S3, and trim the
surrounding narrative.
- Pin the commit under test (696ca29b) and record versions, environment,
and Ballista/DataFusion configuration together.
- Publish Q1-Q19 mean times (2 iterations); mark Q20-Q22 as FAIL on this
configuration.
- Drop the stale Spark/Comet comparison columns; a fresh comparison at
this cluster shape will be added in a follow-up.
Signed-off-by: Andy Grove <[email protected]>
* docs: add Spark 3.4 baseline to benchmarking.md
Include the Spark 3.4 (vanilla, no acceleration) column in the results
table and a Spark configuration highlights section, so the same-cluster
baseline is documented alongside the Ballista numbers.
Signed-off-by: Andy Grove <[email protected]>
* docs: note Spark task/CPU oversubscription in benchmarking.md
spark.executor.cores=16 is task parallelism, not CPU allocation. The
executor pods get 8 physical vCPU via spark.kubernetes.executor.limit.cores,
so Spark schedules 16 concurrent tasks on 8 physical cores. Call this out
alongside Ballistas 1:1 (--concurrent-tasks=8) sizing.
Signed-off-by: Andy Grove <[email protected]>
* docs: prettier format benchmarking.md
Signed-off-by: Andy Grove <[email protected]>
* docs: refresh benchmarking.md with 64 GiB executor run
Refresh the SF1000 results with a new run captured on the same commit
(696ca29b), same cluster shape, but with executor pod memory raised
32 -> 64 GiB (DataFusion memory pool auto-scales to ~44.8 GiB, still 70 %
of container). --concurrent-tasks stays at 8.
Trade-off vs the previous 32 GiB run:
- Q1-Q17 sum 1980 -> 868 s (-44 %). Big wins on the join-heavy queries:
Q5/Q7/Q9/Q17 each -60 % to -67 %; Q8 -43 %.
- Q18 and Q19, which completed at 32 GiB, do not complete at this sizing
and are recorded as FAIL alongside Q20-Q22.
Signed-off-by: Andy Grove <[email protected]>
* docs: refresh benchmarking.md with clean 22-query run
Ballista now completes the full TPC-H SF1000 suite at this commit and
cluster shape, so drop the FAIL rows and publish a full Q1-Q22 result set
plus a Q1-Q22 total.
Also record that the Python benchmark client (BallistaSessionContext) needs
a 64 GiB pod memory limit to run the whole suite in one job: with a smaller
limit the client OOMs midway through, even though the executor cluster is
healthy. This is a client-side sizing note, not an executor tuning knob.
Suite total: Ballista 1353 s vs Spark 3.4 845 s (SF1000, 32 executors x
8 vCPU x 64 GiB, AQE on, 1 iteration).
Signed-off-by: Andy Grove <[email protected]>
---------
Signed-off-by: Andy Grove <[email protected]>
---
docs/source/contributors-guide/benchmarking.md | 476 +++++++------------------
1 file changed, 134 insertions(+), 342 deletions(-)
diff --git a/docs/source/contributors-guide/benchmarking.md
b/docs/source/contributors-guide/benchmarking.md
index 94b608623..fb3b911bf 100644
--- a/docs/source/contributors-guide/benchmarking.md
+++ b/docs/source/contributors-guide/benchmarking.md
@@ -19,384 +19,176 @@
# Benchmarking
-This page describes how Ballista is benchmarked at scale, and records the
current
-results. It is aimed at contributors who want to reproduce a number,
understand why
-a comparison is set up the way it is, or add a result of their own.
+Current TPC-H **SF1000** results for Ballista, compared against a vanilla
+**Spark 3.4** baseline running on the same cluster shape.
-The benchmark used here is derived from TPC-H. For running TPC-H locally at
small
-scale factors, see [`benchmarks/README.md`][bench-readme] in the repository;
this
-page covers the multi-node, large-scale-factor setup and the cross-engine
-comparisons.
+## Versions under test
-[bench-readme]:
https://github.com/apache/datafusion-ballista/blob/main/benchmarks/README.md
-
-## What we measure, and why
-
-Ballista's central performance question is **how evenly work is spread across
-executor tasks**. A distributed query is only as fast as its slowest stage,
and a
-stage is only as fast as its slowest task, so a query whose work concentrates
onto
-a few partitions will underperform no matter how fast the underlying operators
are.
-Benchmarks are therefore run at a scale where that imbalance actually shows up.
-
-These benchmarks are run with **adaptive query execution (AQE) enabled**
-(`ballista.planner.adaptive.enabled=true`). This is the configuration Ballista
is
-being actively developed against, so it is the one measured here.
-
-AQE selects the adaptive planner, which can re-plan stages using runtime
statistics —
-coalescing partitions, re-optimising joins, and promoting a small join side to
a
-broadcast at runtime. The alternative, AQE off, selects the static
-`DefaultDistributedPlanner`; it is a different planner with materially
different join
-behaviour, and it is no longer benchmarked on this page.
+| Engine | Version
|
+| -------- |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|
+| Ballista |
[`696ca29b`](https://github.com/apache/datafusion-ballista/commit/696ca29b3c1fe3013238822b7f5d8cdb918fdaa3)
(`main`, 2026-07-23), Cargo pkg `54.0.0`, DataFusion `54.1.0` |
+| Spark | 3.4 (vanilla, no acceleration plugin)
|
## Environment
-The results on this page currently come from a **small homelab cluster**: two
-bare-metal nodes running Kubernetes, with the TPC-H data staged on node-local
disk.
-This is a deliberate starting point rather than a final destination. A two-node
-cluster with local disk removes as many confounds as possible — no object-store
-latency, no cloud network variance, no noisy neighbours — so that when a query
is
-slow, the cause is Ballista and not the environment.
-
-The intent is to **move these benchmarks to AWS with data in S3** once the
results
-here are good. That environment is the one users actually run, and it exercises
-things a homelab cannot: object-store reads instead of local disk, higher and
more
-variable network latency between executors, and larger executor counts. Some of
-Ballista's behaviour is expected to change there — a shuffle that is cheap
over a
-local link is not cheap over a cloud network, and object-store reads make scan
-parallelism matter differently.
-
-So results on this page should be read as **relative comparisons on controlled
-hardware**, useful for "did this change help", not as absolute throughput
numbers
-for a cloud deployment.
-
-## Reference cluster
-
-All results on this page use the following shape. It is a reference point, not
a
-requirement — the commands below work on any cluster, but numbers are only
-comparable when the shape matches.
-
-| | |
-| ------------------- | ------------------------------------------- |
-| Executors | 2, one per physical node |
-| Per executor | 16 cores, 56 GiB, `--memory-pool-size=48GB` |
-| Per task slot | 16 concurrent tasks → 3 GB pool each |
-| Scheduler | 1 |
-| Data | TPC-H SF1000 Parquet, node-local disk |
-| `target_partitions` | 64 |
-
-Two details matter more than they look:
-
-- **Executors are spread one per node.** Packing two executors onto one node
makes
- them contend for the same disk and memory bandwidth, which is measuring the
host,
- not the engine.
-- **Data is on node-local disk, not object storage.** With a modest network
between
- nodes, reading from object storage makes the interconnect the bottleneck and
the
- engine comparison becomes an I/O comparison.
-
-The memory pool is deliberately set below the container limit. Ballista splits
-`--memory-pool-size` into one `FairSpillPool` per task slot, and that
accounting
-does not cover every allocation, so leaving headroom between the pool and the
-container limit avoids the container being killed outright instead of
reporting a
-graceful resource error.
+- **Cluster:** Kubernetes on AWS (`us-west-2`); one driver/scheduler pod and
+ 32 executor pods for each engine, launched on the same node pool.
+- **Executor pod (Ballista):** x86_64, 8 vCPU, 64 GiB memory.
+- **Executor pod (Spark):** x86_64, 8 vCPU, 64 GiB + 10 GiB overhead.
+- **Client pod (Ballista):** the Python benchmark runner submits SQL to
+ Ballista via `BallistaSessionContext` and collects results locally.
+ Requires 64 GiB memory limit to complete the full 22-query suite —
+ smaller limits (16 GiB) OOM the client partway through, even though the
+ executor cluster is healthy.
+- **Data:** TPC-H SF1000 Parquet on S3 (`us-west-2`), ZSTD compression,
+ ~512 MiB row groups, one directory per table.
+
+## Ballista configuration
+
+| Flag / config key | Value
|
+| ------------------------------------------------------------- |
------------- |
+| `--concurrent-tasks` | `8`
|
+| `--memory-pool-size` (bytes; ≈70 % of the 64 GiB container) |
`48103633715` |
+| `datafusion.execution.target_partitions` | `256`
|
+| `datafusion.execution.collect_statistics` | `true`
|
+| `datafusion.execution.listing_table_factory_infer_partitions` | `false`
|
+| `datafusion.catalog.information_schema` | `true`
|
+| `ballista.planner.adaptive.enabled` | `true` (AQE)
|
+| `ballista.shuffle.sort_based.memory_limit_per_task_bytes` | `0`
|
+
+`datafusion.optimizer.prefer_hash_join` is left at its default; under AQE the
+join strategy is selected at runtime by `DelayJoinSelectionRule` /
+`DynamicJoinSelectionExec` from runtime statistics and the broadcast /
+`ballista.optimizer.hash_join_max_build_partition_bytes` thresholds.
+
+## Spark configuration (highlights)
+
+Vanilla Spark 3.4 — no Comet plugin, stock `SortShuffleManager`.
+
+| Key | Value
|
+| ---------------------------------------------------------- |
------------------------------------ |
+| `spark.executor.instances` | `32`
|
+| `spark.executor.cores` | `16` (task
parallelism per executor) |
+| `spark.kubernetes.executor.limit.cores` / `.request.cores` | `8` (physical
vCPU) |
+| `spark.executor.memory` | `64G`
|
+| `spark.executor.memoryOverhead` | `10G`
|
+| `spark.memory.fraction` | `0.6`
|
+| `spark.memory.storageFraction` | `0.2`
|
+| `spark.sql.shuffle.partitions` | `512`
|
+| `spark.sql.broadcastTimeout` | `900`
|
+| `spark.serializer` |
`KryoSerializer` |
+| `spark.io.compression.codec` | `zstd`
|
+
+Spark AQE is left at its Spark 3.4 defaults. Shuffle spills to a `gp3`-backed
+per-executor volume (`spark.kubernetes.executor.volumes...spark-local-dir-1`).
+
+Note that `spark.executor.cores=16` is Spark's **task parallelism** setting,
+not a CPU allocation — each executor pod is given only **8 physical vCPU**
+via `spark.kubernetes.executor.limit.cores` / `.request.cores`, so Spark
+schedules 16 concurrent tasks onto 8 physical cores (2× oversubscription).
+The matching Ballista executor runs `--concurrent-tasks=8` on the same
+8 physical vCPU (1:1).
## Queries
-All engines run the **same SQL**, which is what makes a cross-engine comparison
-apples-to-apples.
+The SQLBench-H phrasing of the 22 TPC-H queries from
+[apache/datafusion-benchmarks](https://github.com/apache/datafusion-benchmarks).
-The shared set is the TPC-H queries from [apache/datafusion-benchmarks][dfb]
-(SQLBench-H). Ballista's bundled queries in `benchmarks/queries/` are the
classic
-TPC-H phrasing and differ in places, so the shared set is overlaid over them
for a
-comparison run.
-
-Spark and Comet are driven through Comet's benchmark harness, which reads its
own
-bundled copy of the queries (labelled CometBench-H). That copy is textually
-identical to the SQLBench-H set — all 22 queries match once the licence header
-comment is ignored — so the engines are executing the same statements even
though
-they load them from different paths. Worth re-checking if either set is ever
-regenerated.
-
-[dfb]: https://github.com/apache/datafusion-benchmarks
+## Results
-## Running the benchmark
+**Times in seconds; lower is better.** Ballista: single iteration. Spark:
+mean of 3 iterations (cold iteration dropped by the harness).
+
+| Query | Ballista (s) | Spark 3.4 (s) |
+| -----------------: | -----------: | ------------: |
+| 1 | 21.16 | 67.58 |
+| 2 | 32.37 | 29.80 |
+| 3 | 32.69 | 25.13 |
+| 4 | 27.75 | 21.19 |
+| 5 | 49.37 | 54.12 |
+| 6 | 16.17 | 1.23 |
+| 7 | 48.24 | 19.57 |
+| 8 | 189.30 | 48.60 |
+| 9 | 154.88 | 69.38 |
+| 10 | 67.72 | 35.92 |
+| 11 | 26.72 | 30.88 |
+| 12 | 24.46 | 10.78 |
+| 13 | 18.36 | 20.45 |
+| 14 | 19.93 | 7.00 |
+| 15 | 27.56 | 23.75 |
+| 16 | 18.03 | 23.41 |
+| 17 | 60.46 | 82.30 |
+| 18 | 282.79 | 129.40 |
+| 19 | 23.28 | 11.26 |
+| 20 | 88.93 | 19.22 |
+| 21 | 110.87 | 101.53 |
+| 22 | 12.17 | 12.71 |
+| **Total (Q1–Q22)** | **1353.21** | **845.21** |
+
+Row counts agree across engines for every query. Ballista completed the
+full 22-query suite in a single continuous run.
+
+## Reproducing
### Ballista
-Start a scheduler and one executor per node:
+Bring up the cluster (one scheduler, N executors), then run the suite from a
+client. Executor sizing on each node:
```sh
-# scheduler
-ballista-scheduler --bind-host 0.0.0.0 --bind-port 50050
-
-# executor (one per node; --concurrent-tasks defaults to the detected core
count)
ballista-executor \
- --bind-host 0.0.0.0 \
+ --bind-host 0.0.0.0 --bind-port 50051 \
--scheduler-host <scheduler> --scheduler-port 50050 \
- --memory-pool-size=48GB \
- --work-dir /work \
- --client-ttl=60
+ --concurrent-tasks 8 \
+ --memory-pool-size 48103633715
```
-`--client-ttl=60` enables shuffle-client connection caching. Without it every
-shuffle fetch opens a new connection, and a high `target_partitions` can
exhaust
-ephemeral ports.
-
-Run a query:
+Run all 22 queries:
```sh
tpch benchmark ballista \
--host <scheduler> --port 50050 \
- --query 18 \
- --path /mnt/bigdata/tpch/sf1000 --format parquet \
- --partitions 64 --iterations 1 \
- -c ballista.planner.adaptive.enabled=true
+ --path s3://<bucket>/tpch/sf1000 --format parquet \
+ --partitions 256 --iterations 2 \
+ -c ballista.planner.adaptive.enabled=true \
+ -c datafusion.execution.collect_statistics=true \
+ -c ballista.shuffle.sort_based.memory_limit_per_task_bytes=0
```
-Omit `--query` to run all 22, and add `--skip <n>` (repeatable) to leave
individual
-queries out of that run — `--skip 18` is what produced the Ballista column
below.
-`ballista.planner.adaptive.enabled=true` is the AQE-on configuration these
results
-use.
-
-Note that `datafusion.optimizer.prefer_hash_join` is deliberately left at its
default.
-Under AQE it does not select the join strategy: `DelayJoinSelectionRule` folds
both
-`HashJoinExec` and `SortMergeJoinExec` into a single
`DynamicJoinSelectionExec`, and
-the strategy is then chosen from runtime statistics — the broadcast
thresholds, and
-`ballista.optimizer.hash_join_max_build_partition_bytes` (64 MiB by default),
which
-lowers a Partitioned hash join to the spillable `SortMergeJoin` when the build
side's
-largest partition would not fit. Setting `prefer_hash_join=false` does not
force
-sort-merge; it only changes which conversion path the plan takes, and the
-`SortMergeJoinExec` path discards the join's projection, so it measures a
slower plan
-rather than a different join strategy.
-
### Spark
-Spark runs the same queries via `tpcbench.py` from
-[apache/datafusion-benchmarks][dfb]:
+Runs the same queries via `tpcbench.py` from
+[apache/datafusion-benchmarks](https://github.com/apache/datafusion-benchmarks),
+with the highlights above and stock Spark 3.4 defaults for everything else:
```sh
spark-submit \
--master <master> \
- --conf spark.executor.instances=2 \
+ --conf spark.executor.instances=32 \
--conf spark.executor.cores=16 \
- --conf spark.executor.memory=32G \
- --conf spark.executor.memoryOverhead=8G \
- --conf spark.memory.offHeap.enabled=true \
- --conf spark.memory.offHeap.size=16g \
- --conf spark.comet.enabled=false \
- --conf
spark.shuffle.manager=org.apache.spark.shuffle.sort.SortShuffleManager \
+ --conf spark.executor.memory=64G \
+ --conf spark.executor.memoryOverhead=10G \
+ --conf spark.sql.shuffle.partitions=512 \
tpcbench.py \
--benchmark tpch \
- --data /mnt/bigdata/tpch/sf1000 \
+ --data s3a://<bucket>/tpch/sf1000 \
--format parquet \
- --iterations 1 \
- --query 18
+ --iterations 3
```
-`spark.comet.enabled=false` and the stock `SortShuffleManager` are what make
this a
-**vanilla** Spark baseline, in case the image being used ships Comet.
-
-### Comet
-
-[Apache DataFusion Comet][comet] accelerates Spark by translating supported
-operators to DataFusion. Same queries, same sizing; the difference is the
plugin,
-the Comet shuffle manager, and the jar on the classpath:
-
-```sh
-spark-submit \
- --master <master> \
- --jars $COMET_JAR --driver-class-path $COMET_JAR \
- --conf spark.executor.extraClassPath=$COMET_JAR \
- --conf spark.plugins=org.apache.spark.CometPlugin \
- --conf
spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager
\
- --conf spark.comet.exec.replaceSortMergeJoin=false \
- --conf spark.comet.exec.memoryPool=fair_unified \
- --conf spark.comet.exec.memoryPool.fraction=0.8 \
- --conf spark.executor.instances=2 \
- --conf spark.executor.cores=16 \
- --conf spark.executor.memory=32G \
- --conf spark.executor.memoryOverhead=8G \
- --conf spark.memory.offHeap.enabled=true \
- --conf spark.memory.offHeap.size=16g \
- tpcbench.py \
- --benchmark tpch \
- --data /mnt/bigdata/tpch/sf1000 \
- --format parquet \
- --iterations 1 \
- --query 18
-```
-
-`spark.comet.exec.replaceSortMergeJoin=false` keeps Comet on Spark's
`SortMergeJoin`
-instead of converting it to a shuffled hash join.
-
-This no longer pins the two engines to the same join strategy. Ballista under
AQE
-chooses per join at runtime — broadcast, Partitioned hash, or sort-merge — and
there
-is no supported way to force it to sort-merge throughout (see the note in the
Ballista
-section above). So a Ballista-vs-Comet gap may partly reflect different join
-strategies, not only different execution. Read the comparison with that in
mind.
-
-[comet]: https://datafusion.apache.org/comet/
-
-## Why compare against Comet
-
-Comet is the most informative comparison available to Ballista, because
**Comet and
-Ballista execute DataFusion physical plans using the same DataFusion
operators**.
-The scan, filter, join, and aggregate implementations doing the work are
largely
-shared code. When Ballista and Comet diverge on a query, the difference
therefore
-points at what is _not_ shared — how work is distributed, scheduled, shuffled,
and
-bounded by memory — rather than at the speed of the operators themselves. That
is
-precisely the surface Ballista is trying to improve, which makes the comparison
-diagnostic rather than merely competitive.
-
-That said, **the two do not necessarily run the same plan shape**, and the
numbers
-should not be read as an operator-level A/B:
-
-- **Different planners produce the plan.** Comet accelerates a plan that
_Spark's_
- optimizer produced: Spark chooses the join order and join strategies, and
- **Spark's AQE** coalesces shuffle partitions, converts joins, and splits
skewed
- partitions at runtime. Ballista plans with DataFusion's optimizer and, when
- enabled, its own experimental AQE. Spark's AQE is a mature implementation and
- Ballista's is not, so the same SQL can arrive at execution with materially
- different plans.
-- **Comet falls back to Spark.** Operators and expressions Comet does not
support
- stay on the JVM, so a Comet run is generally a mix of DataFusion and Spark
- execution rather than an all-DataFusion one.
-- **The distribution models differ.** Comet executes within Spark's task model
and
- shuffle service; Ballista has its own scheduler, stage/task model, and
shuffle.
-
-So a Comet-vs-Ballista gap is best read as a question — _what is Spark's
planner or
-execution model doing here that Ballista's is not?_ — rather than as a verdict
on
-DataFusion. Vanilla Spark is included as the third data point, since it
isolates
-how much of any Comet result comes from DataFusion acceleration versus from
Spark's
-planner.
-
-## Results
-
-TPC-H **SF1000**, reference cluster above, **AQE on**, 1 iteration,
-`target_partitions=64`, all Ballista join settings at their defaults. Ballista
picks
-each join's strategy at runtime; Spark and Comet are held to `SortMergeJoin`.
Times in
-seconds; lower is better.
-
-Versions under test:
-
-| Engine | Version |
-| -------- | ----------------------------------------------- |
-| Ballista | `#2124` @ `5290436d` |
-| Spark | 3.5.3 (vanilla, Comet disabled) |
-| Comet | `main` @ `b0165552` (1.0.0-SNAPSHOT, Spark 3.5) |
-
-Pin the **exact commit** the numbers came from, not "main": `main` moves, and
a row
-that mixes numbers from different commits silently misattributes a regression.
-
-The Ballista column comes from a **single suite run of 21 queries** on freshly
-started executors, using `--skip 18` to leave out the one query that exhausts
the
-memory pool and OOM-kills the executors (see below), which would otherwise
wedge
-the rest of the suite. Comet and Spark each completed the full suite in one
run.
-
-Keep the whole column to one continuous run wherever possible. Driving queries
as
-individual single-query jobs measures something different: each job starts
against
-a cold page cache, which on this cluster cost the two join-free queries (Q1,
Q6)
-roughly 20 s and 20 s respectively when the two methods were compared
directly. Use
-`--skip` rather than a loop of `--query` runs.
-
-Single iteration means **run-to-run variance is not quantified here**.
Comparing the
-same commit across the two methods above showed spreads of up to ~20% on
individual
-queries, so treat a per-query difference under about 20% as noise and read the
total
-row instead.
-
-| Query | Spark | Comet | Ballista (AQE on) | Rows
|
-| --------------------: | ---------: | ---------: | ----------------: | -----:
|
-| 1 | 427.1 | 46.6 | 54.2 | 4
|
-| 2 | 75.9 | 39.2 | 60.3 | 100
|
-| 3 | 154.1 | 195.4 | 140.0 | 10
|
-| 4 | 82.0 | 42.6 | 46.3 | 5
|
-| 5 | 336.4 | 493.4 | 327.2 | 5
|
-| 6 | 28.9 | 14.6 | 27.2 | 1
|
-| 7 | 180.6 | 149.5 | 364.9 | 4
|
-| 8 | 391.8 | 642.1 | 481.1 | 2
|
-| 9 | 509.8 | 843.5 | 606.4 | 175
|
-| 10 | 151.1 | 104.8 | 148.4 | 20
|
-| 11 | 44.7 | 44.1 | 62.0 | 0 [1]
|
-| 12 | 74.1 | 49.9 | 58.3 | 2
|
-| 13 | 98.7 | 58.3 | 100.4 | 30
|
-| 14 | 43.0 | 27.4 | 71.1 | 1
|
-| 15 | 121.5 | 65.6 | 53.8 | 1 [2]
|
-| 16 | 24.5 | 17.5 | 34.3 | 27840
|
-| 17 | 406.7 | 285.8 | 417.8 | 1
|
-| 18 | 428.8 | 370.5 | TBD | 100
|
-| 19 | 58.9 | 35.5 | 68.5 | 1
|
-| 20 | 105.9 | 67.6 | 93.7 | 110759
|
-| 21 | 562.2 | 460.1 | 698.0 | 100
|
-| 22 | 36.8 | 21.5 | 22.9 | 7
|
-| **Total (excl. Q18)** | **3914.7** | **3705.0** | **3936.8** |
|
-
-The **Total** row sums the 21 queries **excluding Q18**, because Q18 has no
Ballista
-figure at this commit (below), so a 22-query total would not be comparable
across
-engines. Spark's and Comet's Q18 times are in its row.
-
-Row counts agree across all three engines on every query for which Ballista
produced
-a result.
-
-The **AQE-off** Ballista column has been removed pending a re-run at a matched
core
-count; the numbers above for AQE off were taken at a different executor size
and are
-not comparable to a fresh AQE-off run, so they were dropped rather than left
stale.
-
-[1] Q11 returns 0 rows for every engine at this scale factor: the query's
threshold
-constant is tuned for SF1.
-
-[2] Q15 is a multi-statement query (`CREATE VIEW` / `SELECT` / `DROP VIEW`);
all three
-engines report 1 row here. (A previous result set saw Spark report 0 for Q15
depending
-on which statement the harness took as the result; it does not recur in this
run.)
-
-**Q18 is not measured at this commit.** Q18's hash-join build side is
`Partitioned`
-and does not spill; with 16 task slots sharing the 48 GB pool (3 GB per slot),
the
-per-task build side exceeds the container limit and the executor is OOM-killed
-([#2025](https://github.com/apache/datafusion-ballista/issues/2025)). At the
previous
-8-slot sizing (6 GB per slot) Q18 completed, so this is a direct consequence
of the
-denser packing.
-
-It was excluded from the suite run above with `--skip 18` so that one query
could not
-wedge the other 21, and it has not since been run to completion at this commit
— hence
-`TBD` rather than `OOM`. An earlier attempt on this branch did OOM-kill an
executor,
-but under a different configuration (`prefer_hash_join=false`, which sends the
plan
-down a conversion path that discards the join's projection and so materialises
more
-columns than the configuration documented above). That result does not
transfer, so it
-is not recorded here.
-
-The `Rows` column is the row count the query returned, recorded so a time is
never
-read without the answer it produced.
-
-This table records **one current result set**. When results are refreshed, the
-table and the pinned versions above are replaced together — a row must never
mix
-numbers from different commits, because a stale row silently misattributes a
-regression.
-
-`TBD` means not yet measured on this cluster at this commit; a query that ran
but
-did not produce an answer is recorded as `FAIL`, or `OOM` where the failure is
a
-known memory exhaustion.
-
-### Recording a result
+## Recording a new result set
- Pin the **exact commit** the numbers came from, not a branch name.
-- Report the **AQE-on** number (`ballista.planner.adaptive.enabled=true`) — the
- configuration this page measures — from the same commit and cluster.
-- Prefer the figure from a **full suite run**, not a standalone single-query
run: a
- long-lived executor deep into a suite is not in the same state as a freshly
started
- one, and a single-query job pays a cold-cache penalty that is large enough
to swamp
- the effect being measured. When a query cannot complete in-suite (e.g. Q18's
OOM
- currently wedges the run), exclude it with `--skip <n>` so the rest of the
column
- still comes from one continuous run, and measure that query separately.
-- Note the **row count** each query returned. A fast wrong answer is not a
result,
- and distributed execution has produced silently wrong row counts before.
-- Flag any stage whose runtime is dominated by a few partitions — that is the
- imbalance this page exists to surface.
-
-## Known issues found by benchmarking
-
-Benchmarking at SF1000 is how most of the following were found. They are worth
-knowing about before interpreting a number:
-
-| Issue | Summary
|
-| ------------------------------------------------------------------ |
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|
-| [#1359](https://github.com/apache/datafusion-ballista/issues/1359) |
Umbrella issue for adaptive (AQE) query execution.
|
-| [#2025](https://github.com/apache/datafusion-ballista/issues/2025) | Q18's
hash-join build side exhausts the memory pool at SF1000. DataFusion's hash-join
build side does not spill, so a per-partition build side larger than one task
slot's pool fails the task outright. |
-| [#2063](https://github.com/apache/datafusion-ballista/issues/2063) | AQE can
hang when a re-plan cancels an in-flight stage; the job never reports
completion.
|
+- Replace the version, environment, config, and results tables together — a
+ row that mixes numbers from different commits silently misattributes a
+ regression.
+- Prefer a single continuous suite run: a long-lived executor deep into a
+ suite is not in the same state as a freshly started one.
+- Report `FAIL` for a query that ran but did not produce an answer, and
+ `OOM` when the failure is a known memory exhaustion. See the tracker for
+ open issues found by benchmarking: [#1359][aqe],
+ [#2025][q18], [#2063][aqe-hang].
+
+[aqe]: https://github.com/apache/datafusion-ballista/issues/1359
+[q18]: https://github.com/apache/datafusion-ballista/issues/2025
+[aqe-hang]: https://github.com/apache/datafusion-ballista/issues/2063
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
