(datafusion-ballista) branch asf-site updated: Publish built docs triggered by d738ca9bd6b86aecaf9587af9025ae0dd12c5af1

github-bot Thu, 23 Jul 2026 15:26:52 -0700

This is an automated email from the ASF dual-hosted git repository.

github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-ballista.git



The following commit(s) were added to refs/heads/asf-site by this push:
     new 5c05df167 Publish built docs triggered by 
d738ca9bd6b86aecaf9587af9025ae0dd12c5af1
5c05df167 is described below

commit 5c05df16764ad823b765777a9360b2cce31c90bb
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Thu Jul 23 22:26:23 2026 +0000

    Publish built docs triggered by d738ca9bd6b86aecaf9587af9025ae0dd12c5af1
---
 _sources/contributors-guide/benchmarking.md.txt | 476 +++++------------
 contributors-guide/benchmarking.html            | 661 ++++++++----------------
 searchindex.js                                  |   2 +-
 3 files changed, 361 insertions(+), 778 deletions(-)

diff --git a/_sources/contributors-guide/benchmarking.md.txt 
b/_sources/contributors-guide/benchmarking.md.txt
index 94b608623..fb3b911bf 100644
--- a/_sources/contributors-guide/benchmarking.md.txt
+++ b/_sources/contributors-guide/benchmarking.md.txt
@@ -19,384 +19,176 @@
 
 # Benchmarking
 
-This page describes how Ballista is benchmarked at scale, and records the 
current
-results. It is aimed at contributors who want to reproduce a number, 
understand why
-a comparison is set up the way it is, or add a result of their own.
+Current TPC-H **SF1000** results for Ballista, compared against a vanilla
+**Spark 3.4** baseline running on the same cluster shape.
 
-The benchmark used here is derived from TPC-H. For running TPC-H locally at 
small
-scale factors, see [`benchmarks/README.md`][bench-readme] in the repository; 
this
-page covers the multi-node, large-scale-factor setup and the cross-engine
-comparisons.
+## Versions under test
 
-[bench-readme]: 
https://github.com/apache/datafusion-ballista/blob/main/benchmarks/README.md
-
-## What we measure, and why
-
-Ballista's central performance question is **how evenly work is spread across
-executor tasks**. A distributed query is only as fast as its slowest stage, 
and a
-stage is only as fast as its slowest task, so a query whose work concentrates 
onto
-a few partitions will underperform no matter how fast the underlying operators 
are.
-Benchmarks are therefore run at a scale where that imbalance actually shows up.
-
-These benchmarks are run with **adaptive query execution (AQE) enabled**
-(`ballista.planner.adaptive.enabled=true`). This is the configuration Ballista 
is
-being actively developed against, so it is the one measured here.
-
-AQE selects the adaptive planner, which can re-plan stages using runtime 
statistics —
-coalescing partitions, re-optimising joins, and promoting a small join side to 
a
-broadcast at runtime. The alternative, AQE off, selects the static
-`DefaultDistributedPlanner`; it is a different planner with materially 
different join
-behaviour, and it is no longer benchmarked on this page.
+| Engine   | Version                                                           
                                                                                
                        |
+| -------- | 
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 |
+| Ballista | 
[`696ca29b`](https://github.com/apache/datafusion-ballista/commit/696ca29b3c1fe3013238822b7f5d8cdb918fdaa3)
 (`main`, 2026-07-23), Cargo pkg `54.0.0`, DataFusion `54.1.0` |
+| Spark    | 3.4 (vanilla, no acceleration plugin)                             
                                                                                
                        |
 
 ## Environment
 
-The results on this page currently come from a **small homelab cluster**: two
-bare-metal nodes running Kubernetes, with the TPC-H data staged on node-local 
disk.
-This is a deliberate starting point rather than a final destination. A two-node
-cluster with local disk removes as many confounds as possible — no object-store
-latency, no cloud network variance, no noisy neighbours — so that when a query 
is
-slow, the cause is Ballista and not the environment.
-
-The intent is to **move these benchmarks to AWS with data in S3** once the 
results
-here are good. That environment is the one users actually run, and it exercises
-things a homelab cannot: object-store reads instead of local disk, higher and 
more
-variable network latency between executors, and larger executor counts. Some of
-Ballista's behaviour is expected to change there — a shuffle that is cheap 
over a
-local link is not cheap over a cloud network, and object-store reads make scan
-parallelism matter differently.
-
-So results on this page should be read as **relative comparisons on controlled
-hardware**, useful for "did this change help", not as absolute throughput 
numbers
-for a cloud deployment.
-
-## Reference cluster
-
-All results on this page use the following shape. It is a reference point, not 
a
-requirement — the commands below work on any cluster, but numbers are only
-comparable when the shape matches.
-
-|                     |                                             |
-| ------------------- | ------------------------------------------- |
-| Executors           | 2, one per physical node                    |
-| Per executor        | 16 cores, 56 GiB, `--memory-pool-size=48GB` |
-| Per task slot       | 16 concurrent tasks → 3 GB pool each        |
-| Scheduler           | 1                                           |
-| Data                | TPC-H SF1000 Parquet, node-local disk       |
-| `target_partitions` | 64                                          |
-
-Two details matter more than they look:
-
-- **Executors are spread one per node.** Packing two executors onto one node 
makes
-  them contend for the same disk and memory bandwidth, which is measuring the 
host,
-  not the engine.
-- **Data is on node-local disk, not object storage.** With a modest network 
between
-  nodes, reading from object storage makes the interconnect the bottleneck and 
the
-  engine comparison becomes an I/O comparison.
-
-The memory pool is deliberately set below the container limit. Ballista splits
-`--memory-pool-size` into one `FairSpillPool` per task slot, and that 
accounting
-does not cover every allocation, so leaving headroom between the pool and the
-container limit avoids the container being killed outright instead of 
reporting a
-graceful resource error.
+- **Cluster:** Kubernetes on AWS (`us-west-2`); one driver/scheduler pod and
+  32 executor pods for each engine, launched on the same node pool.
+- **Executor pod (Ballista):** x86_64, 8 vCPU, 64 GiB memory.
+- **Executor pod (Spark):** x86_64, 8 vCPU, 64 GiB + 10 GiB overhead.
+- **Client pod (Ballista):** the Python benchmark runner submits SQL to
+  Ballista via `BallistaSessionContext` and collects results locally.
+  Requires 64 GiB memory limit to complete the full 22-query suite —
+  smaller limits (16 GiB) OOM the client partway through, even though the
+  executor cluster is healthy.
+- **Data:** TPC-H SF1000 Parquet on S3 (`us-west-2`), ZSTD compression,
+  ~512 MiB row groups, one directory per table.
+
+## Ballista configuration
+
+| Flag / config key                                             | Value        
 |
+| ------------------------------------------------------------- | 
------------- |
+| `--concurrent-tasks`                                          | `8`          
 |
+| `--memory-pool-size` (bytes; ≈70 % of the 64 GiB container)   | 
`48103633715` |
+| `datafusion.execution.target_partitions`                      | `256`        
 |
+| `datafusion.execution.collect_statistics`                     | `true`       
 |
+| `datafusion.execution.listing_table_factory_infer_partitions` | `false`      
 |
+| `datafusion.catalog.information_schema`                       | `true`       
 |
+| `ballista.planner.adaptive.enabled`                           | `true` (AQE) 
 |
+| `ballista.shuffle.sort_based.memory_limit_per_task_bytes`     | `0`          
 |
+
+`datafusion.optimizer.prefer_hash_join` is left at its default; under AQE the
+join strategy is selected at runtime by `DelayJoinSelectionRule` /
+`DynamicJoinSelectionExec` from runtime statistics and the broadcast /
+`ballista.optimizer.hash_join_max_build_partition_bytes` thresholds.
+
+## Spark configuration (highlights)
+
+Vanilla Spark 3.4 — no Comet plugin, stock `SortShuffleManager`.
+
+| Key                                                        | Value           
                     |
+| ---------------------------------------------------------- | 
------------------------------------ |
+| `spark.executor.instances`                                 | `32`            
                     |
+| `spark.executor.cores`                                     | `16` (task 
parallelism per executor) |
+| `spark.kubernetes.executor.limit.cores` / `.request.cores` | `8` (physical 
vCPU)                  |
+| `spark.executor.memory`                                    | `64G`           
                     |
+| `spark.executor.memoryOverhead`                            | `10G`           
                     |
+| `spark.memory.fraction`                                    | `0.6`           
                     |
+| `spark.memory.storageFraction`                             | `0.2`           
                     |
+| `spark.sql.shuffle.partitions`                             | `512`           
                     |
+| `spark.sql.broadcastTimeout`                               | `900`           
                     |
+| `spark.serializer`                                         | 
`KryoSerializer`                     |
+| `spark.io.compression.codec`                               | `zstd`          
                     |
+
+Spark AQE is left at its Spark 3.4 defaults. Shuffle spills to a `gp3`-backed
+per-executor volume (`spark.kubernetes.executor.volumes...spark-local-dir-1`).
+
+Note that `spark.executor.cores=16` is Spark's **task parallelism** setting,
+not a CPU allocation — each executor pod is given only **8 physical vCPU**
+via `spark.kubernetes.executor.limit.cores` / `.request.cores`, so Spark
+schedules 16 concurrent tasks onto 8 physical cores (2× oversubscription).
+The matching Ballista executor runs `--concurrent-tasks=8` on the same
+8 physical vCPU (1:1).
 
 ## Queries
 
-All engines run the **same SQL**, which is what makes a cross-engine comparison
-apples-to-apples.
+The SQLBench-H phrasing of the 22 TPC-H queries from
+[apache/datafusion-benchmarks](https://github.com/apache/datafusion-benchmarks).
 
-The shared set is the TPC-H queries from [apache/datafusion-benchmarks][dfb]
-(SQLBench-H). Ballista's bundled queries in `benchmarks/queries/` are the 
classic
-TPC-H phrasing and differ in places, so the shared set is overlaid over them 
for a
-comparison run.
-
-Spark and Comet are driven through Comet's benchmark harness, which reads its 
own
-bundled copy of the queries (labelled CometBench-H). That copy is textually
-identical to the SQLBench-H set — all 22 queries match once the licence header
-comment is ignored — so the engines are executing the same statements even 
though
-they load them from different paths. Worth re-checking if either set is ever
-regenerated.
-
-[dfb]: https://github.com/apache/datafusion-benchmarks
+## Results
 
-## Running the benchmark
+**Times in seconds; lower is better.** Ballista: single iteration. Spark:
+mean of 3 iterations (cold iteration dropped by the harness).
+
+|              Query | Ballista (s) | Spark 3.4 (s) |
+| -----------------: | -----------: | ------------: |
+|                  1 |        21.16 |         67.58 |
+|                  2 |        32.37 |         29.80 |
+|                  3 |        32.69 |         25.13 |
+|                  4 |        27.75 |         21.19 |
+|                  5 |        49.37 |         54.12 |
+|                  6 |        16.17 |          1.23 |
+|                  7 |        48.24 |         19.57 |
+|                  8 |       189.30 |         48.60 |
+|                  9 |       154.88 |         69.38 |
+|                 10 |        67.72 |         35.92 |
+|                 11 |        26.72 |         30.88 |
+|                 12 |        24.46 |         10.78 |
+|                 13 |        18.36 |         20.45 |
+|                 14 |        19.93 |          7.00 |
+|                 15 |        27.56 |         23.75 |
+|                 16 |        18.03 |         23.41 |
+|                 17 |        60.46 |         82.30 |
+|                 18 |       282.79 |        129.40 |
+|                 19 |        23.28 |         11.26 |
+|                 20 |        88.93 |         19.22 |
+|                 21 |       110.87 |        101.53 |
+|                 22 |        12.17 |         12.71 |
+| **Total (Q1–Q22)** |  **1353.21** |    **845.21** |
+
+Row counts agree across engines for every query. Ballista completed the
+full 22-query suite in a single continuous run.
+
+## Reproducing
 
 ### Ballista
 
-Start a scheduler and one executor per node:
+Bring up the cluster (one scheduler, N executors), then run the suite from a
+client. Executor sizing on each node:
 
 ```sh
-# scheduler
-ballista-scheduler --bind-host 0.0.0.0 --bind-port 50050
-
-# executor (one per node; --concurrent-tasks defaults to the detected core 
count)
 ballista-executor \
-  --bind-host 0.0.0.0 \
+  --bind-host 0.0.0.0 --bind-port 50051 \
   --scheduler-host <scheduler> --scheduler-port 50050 \
-  --memory-pool-size=48GB \
-  --work-dir /work \
-  --client-ttl=60
+  --concurrent-tasks 8 \
+  --memory-pool-size 48103633715
 ```
 
-`--client-ttl=60` enables shuffle-client connection caching. Without it every
-shuffle fetch opens a new connection, and a high `target_partitions` can 
exhaust
-ephemeral ports.
-
-Run a query:
+Run all 22 queries:
 
 ```sh
 tpch benchmark ballista \
   --host <scheduler> --port 50050 \
-  --query 18 \
-  --path /mnt/bigdata/tpch/sf1000 --format parquet \
-  --partitions 64 --iterations 1 \
-  -c ballista.planner.adaptive.enabled=true
+  --path s3://<bucket>/tpch/sf1000 --format parquet \
+  --partitions 256 --iterations 2 \
+  -c ballista.planner.adaptive.enabled=true \
+  -c datafusion.execution.collect_statistics=true \
+  -c ballista.shuffle.sort_based.memory_limit_per_task_bytes=0
 ```
 
-Omit `--query` to run all 22, and add `--skip <n>` (repeatable) to leave 
individual
-queries out of that run — `--skip 18` is what produced the Ballista column 
below.
-`ballista.planner.adaptive.enabled=true` is the AQE-on configuration these 
results
-use.
-
-Note that `datafusion.optimizer.prefer_hash_join` is deliberately left at its 
default.
-Under AQE it does not select the join strategy: `DelayJoinSelectionRule` folds 
both
-`HashJoinExec` and `SortMergeJoinExec` into a single 
`DynamicJoinSelectionExec`, and
-the strategy is then chosen from runtime statistics — the broadcast 
thresholds, and
-`ballista.optimizer.hash_join_max_build_partition_bytes` (64 MiB by default), 
which
-lowers a Partitioned hash join to the spillable `SortMergeJoin` when the build 
side's
-largest partition would not fit. Setting `prefer_hash_join=false` does not 
force
-sort-merge; it only changes which conversion path the plan takes, and the
-`SortMergeJoinExec` path discards the join's projection, so it measures a 
slower plan
-rather than a different join strategy.
-
 ### Spark
 
-Spark runs the same queries via `tpcbench.py` from
-[apache/datafusion-benchmarks][dfb]:
+Runs the same queries via `tpcbench.py` from
+[apache/datafusion-benchmarks](https://github.com/apache/datafusion-benchmarks),
+with the highlights above and stock Spark 3.4 defaults for everything else:
 
 ```sh
 spark-submit \
   --master <master> \
-  --conf spark.executor.instances=2 \
+  --conf spark.executor.instances=32 \
   --conf spark.executor.cores=16 \
-  --conf spark.executor.memory=32G \
-  --conf spark.executor.memoryOverhead=8G \
-  --conf spark.memory.offHeap.enabled=true \
-  --conf spark.memory.offHeap.size=16g \
-  --conf spark.comet.enabled=false \
-  --conf 
spark.shuffle.manager=org.apache.spark.shuffle.sort.SortShuffleManager \
+  --conf spark.executor.memory=64G \
+  --conf spark.executor.memoryOverhead=10G \
+  --conf spark.sql.shuffle.partitions=512 \
   tpcbench.py \
     --benchmark tpch \
-    --data /mnt/bigdata/tpch/sf1000 \
+    --data s3a://<bucket>/tpch/sf1000 \
     --format parquet \
-    --iterations 1 \
-    --query 18
+    --iterations 3
 ```
 
-`spark.comet.enabled=false` and the stock `SortShuffleManager` are what make 
this a
-**vanilla** Spark baseline, in case the image being used ships Comet.
-
-### Comet
-
-[Apache DataFusion Comet][comet] accelerates Spark by translating supported
-operators to DataFusion. Same queries, same sizing; the difference is the 
plugin,
-the Comet shuffle manager, and the jar on the classpath:
-
-```sh
-spark-submit \
-  --master <master> \
-  --jars $COMET_JAR --driver-class-path $COMET_JAR \
-  --conf spark.executor.extraClassPath=$COMET_JAR \
-  --conf spark.plugins=org.apache.spark.CometPlugin \
-  --conf 
spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager
 \
-  --conf spark.comet.exec.replaceSortMergeJoin=false \
-  --conf spark.comet.exec.memoryPool=fair_unified \
-  --conf spark.comet.exec.memoryPool.fraction=0.8 \
-  --conf spark.executor.instances=2 \
-  --conf spark.executor.cores=16 \
-  --conf spark.executor.memory=32G \
-  --conf spark.executor.memoryOverhead=8G \
-  --conf spark.memory.offHeap.enabled=true \
-  --conf spark.memory.offHeap.size=16g \
-  tpcbench.py \
-    --benchmark tpch \
-    --data /mnt/bigdata/tpch/sf1000 \
-    --format parquet \
-    --iterations 1 \
-    --query 18
-```
-
-`spark.comet.exec.replaceSortMergeJoin=false` keeps Comet on Spark's 
`SortMergeJoin`
-instead of converting it to a shuffled hash join.
-
-This no longer pins the two engines to the same join strategy. Ballista under 
AQE
-chooses per join at runtime — broadcast, Partitioned hash, or sort-merge — and 
there
-is no supported way to force it to sort-merge throughout (see the note in the 
Ballista
-section above). So a Ballista-vs-Comet gap may partly reflect different join
-strategies, not only different execution. Read the comparison with that in 
mind.
-
-[comet]: https://datafusion.apache.org/comet/
-
-## Why compare against Comet
-
-Comet is the most informative comparison available to Ballista, because 
**Comet and
-Ballista execute DataFusion physical plans using the same DataFusion 
operators**.
-The scan, filter, join, and aggregate implementations doing the work are 
largely
-shared code. When Ballista and Comet diverge on a query, the difference 
therefore
-points at what is _not_ shared — how work is distributed, scheduled, shuffled, 
and
-bounded by memory — rather than at the speed of the operators themselves. That 
is
-precisely the surface Ballista is trying to improve, which makes the comparison
-diagnostic rather than merely competitive.
-
-That said, **the two do not necessarily run the same plan shape**, and the 
numbers
-should not be read as an operator-level A/B:
-
-- **Different planners produce the plan.** Comet accelerates a plan that 
_Spark's_
-  optimizer produced: Spark chooses the join order and join strategies, and
-  **Spark's AQE** coalesces shuffle partitions, converts joins, and splits 
skewed
-  partitions at runtime. Ballista plans with DataFusion's optimizer and, when
-  enabled, its own experimental AQE. Spark's AQE is a mature implementation and
-  Ballista's is not, so the same SQL can arrive at execution with materially
-  different plans.
-- **Comet falls back to Spark.** Operators and expressions Comet does not 
support
-  stay on the JVM, so a Comet run is generally a mix of DataFusion and Spark
-  execution rather than an all-DataFusion one.
-- **The distribution models differ.** Comet executes within Spark's task model 
and
-  shuffle service; Ballista has its own scheduler, stage/task model, and 
shuffle.
-
-So a Comet-vs-Ballista gap is best read as a question — _what is Spark's 
planner or
-execution model doing here that Ballista's is not?_ — rather than as a verdict 
on
-DataFusion. Vanilla Spark is included as the third data point, since it 
isolates
-how much of any Comet result comes from DataFusion acceleration versus from 
Spark's
-planner.
-
-## Results
-
-TPC-H **SF1000**, reference cluster above, **AQE on**, 1 iteration,
-`target_partitions=64`, all Ballista join settings at their defaults. Ballista 
picks
-each join's strategy at runtime; Spark and Comet are held to `SortMergeJoin`. 
Times in
-seconds; lower is better.
-
-Versions under test:
-
-| Engine   | Version                                         |
-| -------- | ----------------------------------------------- |
-| Ballista | `#2124` @ `5290436d`                            |
-| Spark    | 3.5.3 (vanilla, Comet disabled)                 |
-| Comet    | `main` @ `b0165552` (1.0.0-SNAPSHOT, Spark 3.5) |
-
-Pin the **exact commit** the numbers came from, not "main": `main` moves, and 
a row
-that mixes numbers from different commits silently misattributes a regression.
-
-The Ballista column comes from a **single suite run of 21 queries** on freshly
-started executors, using `--skip 18` to leave out the one query that exhausts 
the
-memory pool and OOM-kills the executors (see below), which would otherwise 
wedge
-the rest of the suite. Comet and Spark each completed the full suite in one 
run.
-
-Keep the whole column to one continuous run wherever possible. Driving queries 
as
-individual single-query jobs measures something different: each job starts 
against
-a cold page cache, which on this cluster cost the two join-free queries (Q1, 
Q6)
-roughly 20 s and 20 s respectively when the two methods were compared 
directly. Use
-`--skip` rather than a loop of `--query` runs.
-
-Single iteration means **run-to-run variance is not quantified here**. 
Comparing the
-same commit across the two methods above showed spreads of up to ~20% on 
individual
-queries, so treat a per-query difference under about 20% as noise and read the 
total
-row instead.
-
-|                 Query |      Spark |      Comet | Ballista (AQE on) |   Rows 
|
-| --------------------: | ---------: | ---------: | ----------------: | -----: 
|
-|                     1 |      427.1 |       46.6 |              54.2 |      4 
|
-|                     2 |       75.9 |       39.2 |              60.3 |    100 
|
-|                     3 |      154.1 |      195.4 |             140.0 |     10 
|
-|                     4 |       82.0 |       42.6 |              46.3 |      5 
|
-|                     5 |      336.4 |      493.4 |             327.2 |      5 
|
-|                     6 |       28.9 |       14.6 |              27.2 |      1 
|
-|                     7 |      180.6 |      149.5 |             364.9 |      4 
|
-|                     8 |      391.8 |      642.1 |             481.1 |      2 
|
-|                     9 |      509.8 |      843.5 |             606.4 |    175 
|
-|                    10 |      151.1 |      104.8 |             148.4 |     20 
|
-|                    11 |       44.7 |       44.1 |              62.0 |  0 [1] 
|
-|                    12 |       74.1 |       49.9 |              58.3 |      2 
|
-|                    13 |       98.7 |       58.3 |             100.4 |     30 
|
-|                    14 |       43.0 |       27.4 |              71.1 |      1 
|
-|                    15 |      121.5 |       65.6 |              53.8 |  1 [2] 
|
-|                    16 |       24.5 |       17.5 |              34.3 |  27840 
|
-|                    17 |      406.7 |      285.8 |             417.8 |      1 
|
-|                    18 |      428.8 |      370.5 |               TBD |    100 
|
-|                    19 |       58.9 |       35.5 |              68.5 |      1 
|
-|                    20 |      105.9 |       67.6 |              93.7 | 110759 
|
-|                    21 |      562.2 |      460.1 |             698.0 |    100 
|
-|                    22 |       36.8 |       21.5 |              22.9 |      7 
|
-| **Total (excl. Q18)** | **3914.7** | **3705.0** |        **3936.8** |        
|
-
-The **Total** row sums the 21 queries **excluding Q18**, because Q18 has no 
Ballista
-figure at this commit (below), so a 22-query total would not be comparable 
across
-engines. Spark's and Comet's Q18 times are in its row.
-
-Row counts agree across all three engines on every query for which Ballista 
produced
-a result.
-
-The **AQE-off** Ballista column has been removed pending a re-run at a matched 
core
-count; the numbers above for AQE off were taken at a different executor size 
and are
-not comparable to a fresh AQE-off run, so they were dropped rather than left 
stale.
-
-[1] Q11 returns 0 rows for every engine at this scale factor: the query's 
threshold
-constant is tuned for SF1.
-
-[2] Q15 is a multi-statement query (`CREATE VIEW` / `SELECT` / `DROP VIEW`); 
all three
-engines report 1 row here. (A previous result set saw Spark report 0 for Q15 
depending
-on which statement the harness took as the result; it does not recur in this 
run.)
-
-**Q18 is not measured at this commit.** Q18's hash-join build side is 
`Partitioned`
-and does not spill; with 16 task slots sharing the 48 GB pool (3 GB per slot), 
the
-per-task build side exceeds the container limit and the executor is OOM-killed
-([#2025](https://github.com/apache/datafusion-ballista/issues/2025)). At the 
previous
-8-slot sizing (6 GB per slot) Q18 completed, so this is a direct consequence 
of the
-denser packing.
-
-It was excluded from the suite run above with `--skip 18` so that one query 
could not
-wedge the other 21, and it has not since been run to completion at this commit 
— hence
-`TBD` rather than `OOM`. An earlier attempt on this branch did OOM-kill an 
executor,
-but under a different configuration (`prefer_hash_join=false`, which sends the 
plan
-down a conversion path that discards the join's projection and so materialises 
more
-columns than the configuration documented above). That result does not 
transfer, so it
-is not recorded here.
-
-The `Rows` column is the row count the query returned, recorded so a time is 
never
-read without the answer it produced.
-
-This table records **one current result set**. When results are refreshed, the
-table and the pinned versions above are replaced together — a row must never 
mix
-numbers from different commits, because a stale row silently misattributes a
-regression.
-
-`TBD` means not yet measured on this cluster at this commit; a query that ran 
but
-did not produce an answer is recorded as `FAIL`, or `OOM` where the failure is 
a
-known memory exhaustion.
-
-### Recording a result
+## Recording a new result set
 
 - Pin the **exact commit** the numbers came from, not a branch name.
-- Report the **AQE-on** number (`ballista.planner.adaptive.enabled=true`) — the
-  configuration this page measures — from the same commit and cluster.
-- Prefer the figure from a **full suite run**, not a standalone single-query 
run: a
-  long-lived executor deep into a suite is not in the same state as a freshly 
started
-  one, and a single-query job pays a cold-cache penalty that is large enough 
to swamp
-  the effect being measured. When a query cannot complete in-suite (e.g. Q18's 
OOM
-  currently wedges the run), exclude it with `--skip <n>` so the rest of the 
column
-  still comes from one continuous run, and measure that query separately.
-- Note the **row count** each query returned. A fast wrong answer is not a 
result,
-  and distributed execution has produced silently wrong row counts before.
-- Flag any stage whose runtime is dominated by a few partitions — that is the
-  imbalance this page exists to surface.
-
-## Known issues found by benchmarking
-
-Benchmarking at SF1000 is how most of the following were found. They are worth
-knowing about before interpreting a number:
-
-| Issue                                                              | Summary 
                                                                                
                                                                                
                                 |
-| ------------------------------------------------------------------ | 
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 |
-| [#1359](https://github.com/apache/datafusion-ballista/issues/1359) | 
Umbrella issue for adaptive (AQE) query execution.                              
                                                                                
                                         |
-| [#2025](https://github.com/apache/datafusion-ballista/issues/2025) | Q18's 
hash-join build side exhausts the memory pool at SF1000. DataFusion's hash-join 
build side does not spill, so a per-partition build side larger than one task 
slot's pool fails the task outright. |
-| [#2063](https://github.com/apache/datafusion-ballista/issues/2063) | AQE can 
hang when a re-plan cancels an in-flight stage; the job never reports 
completion.                                                                     
                                           |
+- Replace the version, environment, config, and results tables together — a
+  row that mixes numbers from different commits silently misattributes a
+  regression.
+- Prefer a single continuous suite run: a long-lived executor deep into a
+  suite is not in the same state as a freshly started one.
+- Report `FAIL` for a query that ran but did not produce an answer, and
+  `OOM` when the failure is a known memory exhaustion. See the tracker for
+  open issues found by benchmarking: [#1359][aqe],
+  [#2025][q18], [#2063][aqe-hang].
+
+[aqe]: https://github.com/apache/datafusion-ballista/issues/1359
+[q18]: https://github.com/apache/datafusion-ballista/issues/2025
+[aqe-hang]: https://github.com/apache/datafusion-ballista/issues/2063
diff --git a/contributors-guide/benchmarking.html 
b/contributors-guide/benchmarking.html
index 384390bfe..d3f16c2b0 100644
--- a/contributors-guide/benchmarking.html
+++ b/contributors-guide/benchmarking.html
@@ -390,519 +390,314 @@
 -->
 <section id="benchmarking">
 <h1>Benchmarking<a class="headerlink" href="#benchmarking" title="Link to this 
heading">#</a></h1>
-<p>This page describes how Ballista is benchmarked at scale, and records the 
current
-results. It is aimed at contributors who want to reproduce a number, 
understand why
-a comparison is set up the way it is, or add a result of their own.</p>
-<p>The benchmark used here is derived from TPC-H. For running TPC-H locally at 
small
-scale factors, see <a class="reference external" 
href="https://github.com/apache/datafusion-ballista/blob/main/benchmarks/README.md";><code
 class="docutils literal notranslate"><span 
class="pre">benchmarks/README.md</span></code></a> in the repository; this
-page covers the multi-node, large-scale-factor setup and the cross-engine
-comparisons.</p>
-<section id="what-we-measure-and-why">
-<h2>What we measure, and why<a class="headerlink" 
href="#what-we-measure-and-why" title="Link to this heading">#</a></h2>
-<p>Ballista’s central performance question is <strong>how evenly work is 
spread across
-executor tasks</strong>. A distributed query is only as fast as its slowest 
stage, and a
-stage is only as fast as its slowest task, so a query whose work concentrates 
onto
-a few partitions will underperform no matter how fast the underlying operators 
are.
-Benchmarks are therefore run at a scale where that imbalance actually shows 
up.</p>
-<p>These benchmarks are run with <strong>adaptive query execution (AQE) 
enabled</strong>
-(<code class="docutils literal notranslate"><span 
class="pre">ballista.planner.adaptive.enabled=true</span></code>). This is the 
configuration Ballista is
-being actively developed against, so it is the one measured here.</p>
-<p>AQE selects the adaptive planner, which can re-plan stages using runtime 
statistics —
-coalescing partitions, re-optimising joins, and promoting a small join side to 
a
-broadcast at runtime. The alternative, AQE off, selects the static
-<code class="docutils literal notranslate"><span 
class="pre">DefaultDistributedPlanner</span></code>; it is a different planner 
with materially different join
-behaviour, and it is no longer benchmarked on this page.</p>
+<p>Current TPC-H <strong>SF1000</strong> results for Ballista, compared 
against a vanilla
+<strong>Spark 3.4</strong> baseline running on the same cluster shape.</p>
+<section id="versions-under-test">
+<h2>Versions under test<a class="headerlink" href="#versions-under-test" 
title="Link to this heading">#</a></h2>
+<div class="pst-scrollable-table-container"><table class="table">
+<thead>
+<tr class="row-odd"><th class="head"><p>Engine</p></th>
+<th class="head"><p>Version</p></th>
+</tr>
+</thead>
+<tbody>
+<tr class="row-even"><td><p>Ballista</p></td>
+<td><p><a class="reference external" 
href="https://github.com/apache/datafusion-ballista/commit/696ca29b3c1fe3013238822b7f5d8cdb918fdaa3";><code
 class="docutils literal notranslate"><span 
class="pre">696ca29b</span></code></a> (<code class="docutils literal 
notranslate"><span class="pre">main</span></code>, 2026-07-23), Cargo pkg <code 
class="docutils literal notranslate"><span class="pre">54.0.0</span></code>, 
DataFusion <code class="docutils literal notranslate"><span class="pre">54.1.0 
[...]
+</tr>
+<tr class="row-odd"><td><p>Spark</p></td>
+<td><p>3.4 (vanilla, no acceleration plugin)</p></td>
+</tr>
+</tbody>
+</table>
+</div>
 </section>
 <section id="environment">
 <h2>Environment<a class="headerlink" href="#environment" title="Link to this 
heading">#</a></h2>
-<p>The results on this page currently come from a <strong>small homelab 
cluster</strong>: two
-bare-metal nodes running Kubernetes, with the TPC-H data staged on node-local 
disk.
-This is a deliberate starting point rather than a final destination. A two-node
-cluster with local disk removes as many confounds as possible — no object-store
-latency, no cloud network variance, no noisy neighbours — so that when a query 
is
-slow, the cause is Ballista and not the environment.</p>
-<p>The intent is to <strong>move these benchmarks to AWS with data in 
S3</strong> once the results
-here are good. That environment is the one users actually run, and it exercises
-things a homelab cannot: object-store reads instead of local disk, higher and 
more
-variable network latency between executors, and larger executor counts. Some of
-Ballista’s behaviour is expected to change there — a shuffle that is cheap 
over a
-local link is not cheap over a cloud network, and object-store reads make scan
-parallelism matter differently.</p>
-<p>So results on this page should be read as <strong>relative comparisons on 
controlled
-hardware</strong>, useful for “did this change help”, not as absolute 
throughput numbers
-for a cloud deployment.</p>
+<ul class="simple">
+<li><p><strong>Cluster:</strong> Kubernetes on AWS (<code class="docutils 
literal notranslate"><span class="pre">us-west-2</span></code>); one 
driver/scheduler pod and
+32 executor pods for each engine, launched on the same node pool.</p></li>
+<li><p><strong>Executor pod (Ballista):</strong> x86_64, 8 vCPU, 64 GiB 
memory.</p></li>
+<li><p><strong>Executor pod (Spark):</strong> x86_64, 8 vCPU, 64 GiB + 10 GiB 
overhead.</p></li>
+<li><p><strong>Client pod (Ballista):</strong> the Python benchmark runner 
submits SQL to
+Ballista via <code class="docutils literal notranslate"><span 
class="pre">BallistaSessionContext</span></code> and collects results locally.
+Requires 64 GiB memory limit to complete the full 22-query suite —
+smaller limits (16 GiB) OOM the client partway through, even though the
+executor cluster is healthy.</p></li>
+<li><p><strong>Data:</strong> TPC-H SF1000 Parquet on S3 (<code 
class="docutils literal notranslate"><span 
class="pre">us-west-2</span></code>), ZSTD compression,
+~512 MiB row groups, one directory per table.</p></li>
+</ul>
 </section>
-<section id="reference-cluster">
-<h2>Reference cluster<a class="headerlink" href="#reference-cluster" 
title="Link to this heading">#</a></h2>
-<p>All results on this page use the following shape. It is a reference point, 
not a
-requirement — the commands below work on any cluster, but numbers are only
-comparable when the shape matches.</p>
+<section id="ballista-configuration">
+<h2>Ballista configuration<a class="headerlink" href="#ballista-configuration" 
title="Link to this heading">#</a></h2>
 <div class="pst-scrollable-table-container"><table class="table">
 <thead>
-<tr class="row-odd"><th class="head"><p></p></th>
-<th class="head"><p></p></th>
+<tr class="row-odd"><th class="head"><p>Flag / config key</p></th>
+<th class="head"><p>Value</p></th>
 </tr>
 </thead>
 <tbody>
-<tr class="row-even"><td><p>Executors</p></td>
-<td><p>2, one per physical node</p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span 
class="pre">--concurrent-tasks</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span 
class="pre">8</span></code></p></td>
+</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span 
class="pre">--memory-pool-size</span></code> (bytes; ≈70 % of the 64 GiB 
container)</p></td>
+<td><p><code class="docutils literal notranslate"><span 
class="pre">48103633715</span></code></p></td>
 </tr>
-<tr class="row-odd"><td><p>Per executor</p></td>
-<td><p>16 cores, 56 GiB, <code class="docutils literal notranslate"><span 
class="pre">--memory-pool-size=48GB</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span 
class="pre">datafusion.execution.target_partitions</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span 
class="pre">256</span></code></p></td>
 </tr>
-<tr class="row-even"><td><p>Per task slot</p></td>
-<td><p>16 concurrent tasks → 3 GB pool each</p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span 
class="pre">datafusion.execution.collect_statistics</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span 
class="pre">true</span></code></p></td>
 </tr>
-<tr class="row-odd"><td><p>Scheduler</p></td>
-<td><p>1</p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span 
class="pre">datafusion.execution.listing_table_factory_infer_partitions</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span 
class="pre">false</span></code></p></td>
 </tr>
-<tr class="row-even"><td><p>Data</p></td>
-<td><p>TPC-H SF1000 Parquet, node-local disk</p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span 
class="pre">datafusion.catalog.information_schema</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span 
class="pre">true</span></code></p></td>
 </tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span 
class="pre">target_partitions</span></code></p></td>
-<td><p>64</p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span 
class="pre">ballista.planner.adaptive.enabled</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span 
class="pre">true</span></code> (AQE)</p></td>
+</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span 
class="pre">ballista.shuffle.sort_based.memory_limit_per_task_bytes</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span 
class="pre">0</span></code></p></td>
 </tr>
 </tbody>
 </table>
 </div>
-<p>Two details matter more than they look:</p>
-<ul class="simple">
-<li><p><strong>Executors are spread one per node.</strong> Packing two 
executors onto one node makes
-them contend for the same disk and memory bandwidth, which is measuring the 
host,
-not the engine.</p></li>
-<li><p><strong>Data is on node-local disk, not object storage.</strong> With a 
modest network between
-nodes, reading from object storage makes the interconnect the bottleneck and 
the
-engine comparison becomes an I/O comparison.</p></li>
-</ul>
-<p>The memory pool is deliberately set below the container limit. Ballista 
splits
-<code class="docutils literal notranslate"><span 
class="pre">--memory-pool-size</span></code> into one <code class="docutils 
literal notranslate"><span class="pre">FairSpillPool</span></code> per task 
slot, and that accounting
-does not cover every allocation, so leaving headroom between the pool and the
-container limit avoids the container being killed outright instead of 
reporting a
-graceful resource error.</p>
-</section>
-<section id="queries">
-<h2>Queries<a class="headerlink" href="#queries" title="Link to this 
heading">#</a></h2>
-<p>All engines run the <strong>same SQL</strong>, which is what makes a 
cross-engine comparison
-apples-to-apples.</p>
-<p>The shared set is the TPC-H queries from <a class="reference external" 
href="https://github.com/apache/datafusion-benchmarks";>apache/datafusion-benchmarks</a>
-(SQLBench-H). Ballista’s bundled queries in <code class="docutils literal 
notranslate"><span class="pre">benchmarks/queries/</span></code> are the classic
-TPC-H phrasing and differ in places, so the shared set is overlaid over them 
for a
-comparison run.</p>
-<p>Spark and Comet are driven through Comet’s benchmark harness, which reads 
its own
-bundled copy of the queries (labelled CometBench-H). That copy is textually
-identical to the SQLBench-H set — all 22 queries match once the licence header
-comment is ignored — so the engines are executing the same statements even 
though
-they load them from different paths. Worth re-checking if either set is ever
-regenerated.</p>
-</section>
-<section id="running-the-benchmark">
-<h2>Running the benchmark<a class="headerlink" href="#running-the-benchmark" 
title="Link to this heading">#</a></h2>
-<section id="ballista">
-<h3>Ballista<a class="headerlink" href="#ballista" title="Link to this 
heading">#</a></h3>
-<p>Start a scheduler and one executor per node:</p>
-<div class="highlight-sh notranslate"><div 
class="highlight"><pre><span></span><span class="c1"># scheduler</span>
-ballista-scheduler<span class="w"> </span>--bind-host<span class="w"> 
</span><span class="m">0</span>.0.0.0<span class="w"> </span>--bind-port<span 
class="w"> </span><span class="m">50050</span>
-
-<span class="c1"># executor (one per node; --concurrent-tasks defaults to the 
detected core count)</span>
-ballista-executor<span class="w"> </span><span class="se">\</span>
-<span class="w">  </span>--bind-host<span class="w"> </span><span 
class="m">0</span>.0.0.0<span class="w"> </span><span class="se">\</span>
-<span class="w">  </span>--scheduler-host<span class="w"> 
</span>&lt;scheduler&gt;<span class="w"> </span>--scheduler-port<span 
class="w"> </span><span class="m">50050</span><span class="w"> </span><span 
class="se">\</span>
-<span class="w">  </span>--memory-pool-size<span class="o">=</span>48GB<span 
class="w"> </span><span class="se">\</span>
-<span class="w">  </span>--work-dir<span class="w"> </span>/work<span 
class="w"> </span><span class="se">\</span>
-<span class="w">  </span>--client-ttl<span class="o">=</span><span 
class="m">60</span>
-</pre></div>
-</div>
-<p><code class="docutils literal notranslate"><span 
class="pre">--client-ttl=60</span></code> enables shuffle-client connection 
caching. Without it every
-shuffle fetch opens a new connection, and a high <code class="docutils literal 
notranslate"><span class="pre">target_partitions</span></code> can exhaust
-ephemeral ports.</p>
-<p>Run a query:</p>
-<div class="highlight-sh notranslate"><div 
class="highlight"><pre><span></span>tpch<span class="w"> </span>benchmark<span 
class="w"> </span>ballista<span class="w"> </span><span class="se">\</span>
-<span class="w">  </span>--host<span class="w"> </span>&lt;scheduler&gt;<span 
class="w"> </span>--port<span class="w"> </span><span 
class="m">50050</span><span class="w"> </span><span class="se">\</span>
-<span class="w">  </span>--query<span class="w"> </span><span 
class="m">18</span><span class="w"> </span><span class="se">\</span>
-<span class="w">  </span>--path<span class="w"> 
</span>/mnt/bigdata/tpch/sf1000<span class="w"> </span>--format<span class="w"> 
</span>parquet<span class="w"> </span><span class="se">\</span>
-<span class="w">  </span>--partitions<span class="w"> </span><span 
class="m">64</span><span class="w"> </span>--iterations<span class="w"> 
</span><span class="m">1</span><span class="w"> </span><span class="se">\</span>
-<span class="w">  </span>-c<span class="w"> 
</span>ballista.planner.adaptive.enabled<span class="o">=</span><span 
class="nb">true</span>
-</pre></div>
-</div>
-<p>Omit <code class="docutils literal notranslate"><span 
class="pre">--query</span></code> to run all 22, and add <code class="docutils 
literal notranslate"><span class="pre">--skip</span> <span 
class="pre">&lt;n&gt;</span></code> (repeatable) to leave individual
-queries out of that run — <code class="docutils literal notranslate"><span 
class="pre">--skip</span> <span class="pre">18</span></code> is what produced 
the Ballista column below.
-<code class="docutils literal notranslate"><span 
class="pre">ballista.planner.adaptive.enabled=true</span></code> is the AQE-on 
configuration these results
-use.</p>
-<p>Note that <code class="docutils literal notranslate"><span 
class="pre">datafusion.optimizer.prefer_hash_join</span></code> is deliberately 
left at its default.
-Under AQE it does not select the join strategy: <code class="docutils literal 
notranslate"><span class="pre">DelayJoinSelectionRule</span></code> folds both
-<code class="docutils literal notranslate"><span 
class="pre">HashJoinExec</span></code> and <code class="docutils literal 
notranslate"><span class="pre">SortMergeJoinExec</span></code> into a single 
<code class="docutils literal notranslate"><span 
class="pre">DynamicJoinSelectionExec</span></code>, and
-the strategy is then chosen from runtime statistics — the broadcast 
thresholds, and
-<code class="docutils literal notranslate"><span 
class="pre">ballista.optimizer.hash_join_max_build_partition_bytes</span></code>
 (64 MiB by default), which
-lowers a Partitioned hash join to the spillable <code class="docutils literal 
notranslate"><span class="pre">SortMergeJoin</span></code> when the build side’s
-largest partition would not fit. Setting <code class="docutils literal 
notranslate"><span class="pre">prefer_hash_join=false</span></code> does not 
force
-sort-merge; it only changes which conversion path the plan takes, and the
-<code class="docutils literal notranslate"><span 
class="pre">SortMergeJoinExec</span></code> path discards the join’s 
projection, so it measures a slower plan
-rather than a different join strategy.</p>
-</section>
-<section id="spark">
-<h3>Spark<a class="headerlink" href="#spark" title="Link to this 
heading">#</a></h3>
-<p>Spark runs the same queries via <code class="docutils literal 
notranslate"><span class="pre">tpcbench.py</span></code> from
-<a class="reference external" 
href="https://github.com/apache/datafusion-benchmarks";>apache/datafusion-benchmarks</a>:</p>
-<div class="highlight-sh notranslate"><div 
class="highlight"><pre><span></span>spark-submit<span class="w"> </span><span 
class="se">\</span>
-<span class="w">  </span>--master<span class="w"> </span>&lt;master&gt;<span 
class="w"> </span><span class="se">\</span>
-<span class="w">  </span>--conf<span class="w"> 
</span>spark.executor.instances<span class="o">=</span><span 
class="m">2</span><span class="w"> </span><span class="se">\</span>
-<span class="w">  </span>--conf<span class="w"> 
</span>spark.executor.cores<span class="o">=</span><span 
class="m">16</span><span class="w"> </span><span class="se">\</span>
-<span class="w">  </span>--conf<span class="w"> 
</span>spark.executor.memory<span class="o">=</span>32G<span class="w"> 
</span><span class="se">\</span>
-<span class="w">  </span>--conf<span class="w"> 
</span>spark.executor.memoryOverhead<span class="o">=</span>8G<span class="w"> 
</span><span class="se">\</span>
-<span class="w">  </span>--conf<span class="w"> 
</span>spark.memory.offHeap.enabled<span class="o">=</span><span 
class="nb">true</span><span class="w"> </span><span class="se">\</span>
-<span class="w">  </span>--conf<span class="w"> 
</span>spark.memory.offHeap.size<span class="o">=</span>16g<span class="w"> 
</span><span class="se">\</span>
-<span class="w">  </span>--conf<span class="w"> 
</span>spark.comet.enabled<span class="o">=</span><span 
class="nb">false</span><span class="w"> </span><span class="se">\</span>
-<span class="w">  </span>--conf<span class="w"> 
</span>spark.shuffle.manager<span 
class="o">=</span>org.apache.spark.shuffle.sort.SortShuffleManager<span 
class="w"> </span><span class="se">\</span>
-<span class="w">  </span>tpcbench.py<span class="w"> </span><span 
class="se">\</span>
-<span class="w">    </span>--benchmark<span class="w"> </span>tpch<span 
class="w"> </span><span class="se">\</span>
-<span class="w">    </span>--data<span class="w"> 
</span>/mnt/bigdata/tpch/sf1000<span class="w"> </span><span class="se">\</span>
-<span class="w">    </span>--format<span class="w"> </span>parquet<span 
class="w"> </span><span class="se">\</span>
-<span class="w">    </span>--iterations<span class="w"> </span><span 
class="m">1</span><span class="w"> </span><span class="se">\</span>
-<span class="w">    </span>--query<span class="w"> </span><span 
class="m">18</span>
-</pre></div>
-</div>
-<p><code class="docutils literal notranslate"><span 
class="pre">spark.comet.enabled=false</span></code> and the stock <code 
class="docutils literal notranslate"><span 
class="pre">SortShuffleManager</span></code> are what make this a
-<strong>vanilla</strong> Spark baseline, in case the image being used ships 
Comet.</p>
-</section>
-<section id="comet">
-<h3>Comet<a class="headerlink" href="#comet" title="Link to this 
heading">#</a></h3>
-<p><a class="reference external" 
href="https://datafusion.apache.org/comet/";>Apache DataFusion Comet</a> 
accelerates Spark by translating supported
-operators to DataFusion. Same queries, same sizing; the difference is the 
plugin,
-the Comet shuffle manager, and the jar on the classpath:</p>
-<div class="highlight-sh notranslate"><div 
class="highlight"><pre><span></span>spark-submit<span class="w"> </span><span 
class="se">\</span>
-<span class="w">  </span>--master<span class="w"> </span>&lt;master&gt;<span 
class="w"> </span><span class="se">\</span>
-<span class="w">  </span>--jars<span class="w"> </span><span 
class="nv">$COMET_JAR</span><span class="w"> </span>--driver-class-path<span 
class="w"> </span><span class="nv">$COMET_JAR</span><span class="w"> 
</span><span class="se">\</span>
-<span class="w">  </span>--conf<span class="w"> 
</span>spark.executor.extraClassPath<span class="o">=</span><span 
class="nv">$COMET_JAR</span><span class="w"> </span><span class="se">\</span>
-<span class="w">  </span>--conf<span class="w"> </span>spark.plugins<span 
class="o">=</span>org.apache.spark.CometPlugin<span class="w"> </span><span 
class="se">\</span>
-<span class="w">  </span>--conf<span class="w"> 
</span>spark.shuffle.manager<span 
class="o">=</span>org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager<span
 class="w"> </span><span class="se">\</span>
-<span class="w">  </span>--conf<span class="w"> 
</span>spark.comet.exec.replaceSortMergeJoin<span class="o">=</span><span 
class="nb">false</span><span class="w"> </span><span class="se">\</span>
-<span class="w">  </span>--conf<span class="w"> 
</span>spark.comet.exec.memoryPool<span class="o">=</span>fair_unified<span 
class="w"> </span><span class="se">\</span>
-<span class="w">  </span>--conf<span class="w"> 
</span>spark.comet.exec.memoryPool.fraction<span class="o">=</span><span 
class="m">0</span>.8<span class="w"> </span><span class="se">\</span>
-<span class="w">  </span>--conf<span class="w"> 
</span>spark.executor.instances<span class="o">=</span><span 
class="m">2</span><span class="w"> </span><span class="se">\</span>
-<span class="w">  </span>--conf<span class="w"> 
</span>spark.executor.cores<span class="o">=</span><span 
class="m">16</span><span class="w"> </span><span class="se">\</span>
-<span class="w">  </span>--conf<span class="w"> 
</span>spark.executor.memory<span class="o">=</span>32G<span class="w"> 
</span><span class="se">\</span>
-<span class="w">  </span>--conf<span class="w"> 
</span>spark.executor.memoryOverhead<span class="o">=</span>8G<span class="w"> 
</span><span class="se">\</span>
-<span class="w">  </span>--conf<span class="w"> 
</span>spark.memory.offHeap.enabled<span class="o">=</span><span 
class="nb">true</span><span class="w"> </span><span class="se">\</span>
-<span class="w">  </span>--conf<span class="w"> 
</span>spark.memory.offHeap.size<span class="o">=</span>16g<span class="w"> 
</span><span class="se">\</span>
-<span class="w">  </span>tpcbench.py<span class="w"> </span><span 
class="se">\</span>
-<span class="w">    </span>--benchmark<span class="w"> </span>tpch<span 
class="w"> </span><span class="se">\</span>
-<span class="w">    </span>--data<span class="w"> 
</span>/mnt/bigdata/tpch/sf1000<span class="w"> </span><span class="se">\</span>
-<span class="w">    </span>--format<span class="w"> </span>parquet<span 
class="w"> </span><span class="se">\</span>
-<span class="w">    </span>--iterations<span class="w"> </span><span 
class="m">1</span><span class="w"> </span><span class="se">\</span>
-<span class="w">    </span>--query<span class="w"> </span><span 
class="m">18</span>
-</pre></div>
-</div>
-<p><code class="docutils literal notranslate"><span 
class="pre">spark.comet.exec.replaceSortMergeJoin=false</span></code> keeps 
Comet on Spark’s <code class="docutils literal notranslate"><span 
class="pre">SortMergeJoin</span></code>
-instead of converting it to a shuffled hash join.</p>
-<p>This no longer pins the two engines to the same join strategy. Ballista 
under AQE
-chooses per join at runtime — broadcast, Partitioned hash, or sort-merge — and 
there
-is no supported way to force it to sort-merge throughout (see the note in the 
Ballista
-section above). So a Ballista-vs-Comet gap may partly reflect different join
-strategies, not only different execution. Read the comparison with that in 
mind.</p>
-</section>
-</section>
-<section id="why-compare-against-comet">
-<h2>Why compare against Comet<a class="headerlink" 
href="#why-compare-against-comet" title="Link to this heading">#</a></h2>
-<p>Comet is the most informative comparison available to Ballista, because 
<strong>Comet and
-Ballista execute DataFusion physical plans using the same DataFusion 
operators</strong>.
-The scan, filter, join, and aggregate implementations doing the work are 
largely
-shared code. When Ballista and Comet diverge on a query, the difference 
therefore
-points at what is <em>not</em> shared — how work is distributed, scheduled, 
shuffled, and
-bounded by memory — rather than at the speed of the operators themselves. That 
is
-precisely the surface Ballista is trying to improve, which makes the comparison
-diagnostic rather than merely competitive.</p>
-<p>That said, <strong>the two do not necessarily run the same plan 
shape</strong>, and the numbers
-should not be read as an operator-level A/B:</p>
-<ul class="simple">
-<li><p><strong>Different planners produce the plan.</strong> Comet accelerates 
a plan that <em>Spark’s</em>
-optimizer produced: Spark chooses the join order and join strategies, and
-<strong>Spark’s AQE</strong> coalesces shuffle partitions, converts joins, and 
splits skewed
-partitions at runtime. Ballista plans with DataFusion’s optimizer and, when
-enabled, its own experimental AQE. Spark’s AQE is a mature implementation and
-Ballista’s is not, so the same SQL can arrive at execution with materially
-different plans.</p></li>
-<li><p><strong>Comet falls back to Spark.</strong> Operators and expressions 
Comet does not support
-stay on the JVM, so a Comet run is generally a mix of DataFusion and Spark
-execution rather than an all-DataFusion one.</p></li>
-<li><p><strong>The distribution models differ.</strong> Comet executes within 
Spark’s task model and
-shuffle service; Ballista has its own scheduler, stage/task model, and 
shuffle.</p></li>
-</ul>
-<p>So a Comet-vs-Ballista gap is best read as a question — <em>what is Spark’s 
planner or
-execution model doing here that Ballista’s is not?</em> — rather than as a 
verdict on
-DataFusion. Vanilla Spark is included as the third data point, since it 
isolates
-how much of any Comet result comes from DataFusion acceleration versus from 
Spark’s
-planner.</p>
+<p><code class="docutils literal notranslate"><span 
class="pre">datafusion.optimizer.prefer_hash_join</span></code> is left at its 
default; under AQE the
+join strategy is selected at runtime by <code class="docutils literal 
notranslate"><span class="pre">DelayJoinSelectionRule</span></code> /
+<code class="docutils literal notranslate"><span 
class="pre">DynamicJoinSelectionExec</span></code> from runtime statistics and 
the broadcast /
+<code class="docutils literal notranslate"><span 
class="pre">ballista.optimizer.hash_join_max_build_partition_bytes</span></code>
 thresholds.</p>
 </section>
-<section id="results">
-<h2>Results<a class="headerlink" href="#results" title="Link to this 
heading">#</a></h2>
-<p>TPC-H <strong>SF1000</strong>, reference cluster above, <strong>AQE 
on</strong>, 1 iteration,
-<code class="docutils literal notranslate"><span 
class="pre">target_partitions=64</span></code>, all Ballista join settings at 
their defaults. Ballista picks
-each join’s strategy at runtime; Spark and Comet are held to <code 
class="docutils literal notranslate"><span 
class="pre">SortMergeJoin</span></code>. Times in
-seconds; lower is better.</p>
-<p>Versions under test:</p>
+<section id="spark-configuration-highlights">
+<h2>Spark configuration (highlights)<a class="headerlink" 
href="#spark-configuration-highlights" title="Link to this heading">#</a></h2>
+<p>Vanilla Spark 3.4 — no Comet plugin, stock <code class="docutils literal 
notranslate"><span class="pre">SortShuffleManager</span></code>.</p>
 <div class="pst-scrollable-table-container"><table class="table">
 <thead>
-<tr class="row-odd"><th class="head"><p>Engine</p></th>
-<th class="head"><p>Version</p></th>
+<tr class="row-odd"><th class="head"><p>Key</p></th>
+<th class="head"><p>Value</p></th>
 </tr>
 </thead>
 <tbody>
-<tr class="row-even"><td><p>Ballista</p></td>
-<td><p><code class="docutils literal notranslate"><span 
class="pre">#2124</span></code> &#64; <code class="docutils literal 
notranslate"><span class="pre">5290436d</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span 
class="pre">spark.executor.instances</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span 
class="pre">32</span></code></p></td>
 </tr>
-<tr class="row-odd"><td><p>Spark</p></td>
-<td><p>3.5.3 (vanilla, Comet disabled)</p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span 
class="pre">spark.executor.cores</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span 
class="pre">16</span></code> (task parallelism per executor)</p></td>
+</tr>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span 
class="pre">spark.kubernetes.executor.limit.cores</span></code> / <code 
class="docutils literal notranslate"><span 
class="pre">.request.cores</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span 
class="pre">8</span></code> (physical vCPU)</p></td>
+</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span 
class="pre">spark.executor.memory</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span 
class="pre">64G</span></code></p></td>
 </tr>
-<tr class="row-even"><td><p>Comet</p></td>
-<td><p><code class="docutils literal notranslate"><span 
class="pre">main</span></code> &#64; <code class="docutils literal 
notranslate"><span class="pre">b0165552</span></code> (1.0.0-SNAPSHOT, Spark 
3.5)</p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span 
class="pre">spark.executor.memoryOverhead</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span 
class="pre">10G</span></code></p></td>
+</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span 
class="pre">spark.memory.fraction</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span 
class="pre">0.6</span></code></p></td>
+</tr>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span 
class="pre">spark.memory.storageFraction</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span 
class="pre">0.2</span></code></p></td>
+</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span 
class="pre">spark.sql.shuffle.partitions</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span 
class="pre">512</span></code></p></td>
+</tr>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span 
class="pre">spark.sql.broadcastTimeout</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span 
class="pre">900</span></code></p></td>
+</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span 
class="pre">spark.serializer</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span 
class="pre">KryoSerializer</span></code></p></td>
+</tr>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span 
class="pre">spark.io.compression.codec</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span 
class="pre">zstd</span></code></p></td>
 </tr>
 </tbody>
 </table>
 </div>
-<p>Pin the <strong>exact commit</strong> the numbers came from, not “main”: 
<code class="docutils literal notranslate"><span class="pre">main</span></code> 
moves, and a row
-that mixes numbers from different commits silently misattributes a 
regression.</p>
-<p>The Ballista column comes from a <strong>single suite run of 21 
queries</strong> on freshly
-started executors, using <code class="docutils literal notranslate"><span 
class="pre">--skip</span> <span class="pre">18</span></code> to leave out the 
one query that exhausts the
-memory pool and OOM-kills the executors (see below), which would otherwise 
wedge
-the rest of the suite. Comet and Spark each completed the full suite in one 
run.</p>
-<p>Keep the whole column to one continuous run wherever possible. Driving 
queries as
-individual single-query jobs measures something different: each job starts 
against
-a cold page cache, which on this cluster cost the two join-free queries (Q1, 
Q6)
-roughly 20 s and 20 s respectively when the two methods were compared 
directly. Use
-<code class="docutils literal notranslate"><span 
class="pre">--skip</span></code> rather than a loop of <code class="docutils 
literal notranslate"><span class="pre">--query</span></code> runs.</p>
-<p>Single iteration means <strong>run-to-run variance is not quantified 
here</strong>. Comparing the
-same commit across the two methods above showed spreads of up to ~20% on 
individual
-queries, so treat a per-query difference under about 20% as noise and read the 
total
-row instead.</p>
+<p>Spark AQE is left at its Spark 3.4 defaults. Shuffle spills to a <code 
class="docutils literal notranslate"><span class="pre">gp3</span></code>-backed
+per-executor volume (<code class="docutils literal notranslate"><span 
class="pre">spark.kubernetes.executor.volumes...spark-local-dir-1</span></code>).</p>
+<p>Note that <code class="docutils literal notranslate"><span 
class="pre">spark.executor.cores=16</span></code> is Spark’s <strong>task 
parallelism</strong> setting,
+not a CPU allocation — each executor pod is given only <strong>8 physical 
vCPU</strong>
+via <code class="docutils literal notranslate"><span 
class="pre">spark.kubernetes.executor.limit.cores</span></code> / <code 
class="docutils literal notranslate"><span 
class="pre">.request.cores</span></code>, so Spark
+schedules 16 concurrent tasks onto 8 physical cores (2× oversubscription).
+The matching Ballista executor runs <code class="docutils literal 
notranslate"><span class="pre">--concurrent-tasks=8</span></code> on the same
+8 physical vCPU (1:1).</p>
+</section>
+<section id="queries">
+<h2>Queries<a class="headerlink" href="#queries" title="Link to this 
heading">#</a></h2>
+<p>The SQLBench-H phrasing of the 22 TPC-H queries from
+<a class="reference external" 
href="https://github.com/apache/datafusion-benchmarks";>apache/datafusion-benchmarks</a>.</p>
+</section>
+<section id="results">
+<h2>Results<a class="headerlink" href="#results" title="Link to this 
heading">#</a></h2>
+<p><strong>Times in seconds; lower is better.</strong> Ballista: single 
iteration. Spark:
+mean of 3 iterations (cold iteration dropped by the harness).</p>
 <div class="pst-scrollable-table-container"><table class="table">
 <thead>
 <tr class="row-odd"><th class="head text-right"><p>Query</p></th>
-<th class="head text-right"><p>Spark</p></th>
-<th class="head text-right"><p>Comet</p></th>
-<th class="head text-right"><p>Ballista (AQE on)</p></th>
-<th class="head text-right"><p>Rows</p></th>
+<th class="head text-right"><p>Ballista (s)</p></th>
+<th class="head text-right"><p>Spark 3.4 (s)</p></th>
 </tr>
 </thead>
 <tbody>
 <tr class="row-even"><td class="text-right"><p>1</p></td>
-<td class="text-right"><p>427.1</p></td>
-<td class="text-right"><p>46.6</p></td>
-<td class="text-right"><p>54.2</p></td>
-<td class="text-right"><p>4</p></td>
+<td class="text-right"><p>21.16</p></td>
+<td class="text-right"><p>67.58</p></td>
 </tr>
 <tr class="row-odd"><td class="text-right"><p>2</p></td>
-<td class="text-right"><p>75.9</p></td>
-<td class="text-right"><p>39.2</p></td>
-<td class="text-right"><p>60.3</p></td>
-<td class="text-right"><p>100</p></td>
+<td class="text-right"><p>32.37</p></td>
+<td class="text-right"><p>29.80</p></td>
 </tr>
 <tr class="row-even"><td class="text-right"><p>3</p></td>
-<td class="text-right"><p>154.1</p></td>
-<td class="text-right"><p>195.4</p></td>
-<td class="text-right"><p>140.0</p></td>
-<td class="text-right"><p>10</p></td>
+<td class="text-right"><p>32.69</p></td>
+<td class="text-right"><p>25.13</p></td>
 </tr>
 <tr class="row-odd"><td class="text-right"><p>4</p></td>
-<td class="text-right"><p>82.0</p></td>
-<td class="text-right"><p>42.6</p></td>
-<td class="text-right"><p>46.3</p></td>
-<td class="text-right"><p>5</p></td>
+<td class="text-right"><p>27.75</p></td>
+<td class="text-right"><p>21.19</p></td>
 </tr>
 <tr class="row-even"><td class="text-right"><p>5</p></td>
-<td class="text-right"><p>336.4</p></td>
-<td class="text-right"><p>493.4</p></td>
-<td class="text-right"><p>327.2</p></td>
-<td class="text-right"><p>5</p></td>
+<td class="text-right"><p>49.37</p></td>
+<td class="text-right"><p>54.12</p></td>
 </tr>
 <tr class="row-odd"><td class="text-right"><p>6</p></td>
-<td class="text-right"><p>28.9</p></td>
-<td class="text-right"><p>14.6</p></td>
-<td class="text-right"><p>27.2</p></td>
-<td class="text-right"><p>1</p></td>
+<td class="text-right"><p>16.17</p></td>
+<td class="text-right"><p>1.23</p></td>
 </tr>
 <tr class="row-even"><td class="text-right"><p>7</p></td>
-<td class="text-right"><p>180.6</p></td>
-<td class="text-right"><p>149.5</p></td>
-<td class="text-right"><p>364.9</p></td>
-<td class="text-right"><p>4</p></td>
+<td class="text-right"><p>48.24</p></td>
+<td class="text-right"><p>19.57</p></td>
 </tr>
 <tr class="row-odd"><td class="text-right"><p>8</p></td>
-<td class="text-right"><p>391.8</p></td>
-<td class="text-right"><p>642.1</p></td>
-<td class="text-right"><p>481.1</p></td>
-<td class="text-right"><p>2</p></td>
+<td class="text-right"><p>189.30</p></td>
+<td class="text-right"><p>48.60</p></td>
 </tr>
 <tr class="row-even"><td class="text-right"><p>9</p></td>
-<td class="text-right"><p>509.8</p></td>
-<td class="text-right"><p>843.5</p></td>
-<td class="text-right"><p>606.4</p></td>
-<td class="text-right"><p>175</p></td>
+<td class="text-right"><p>154.88</p></td>
+<td class="text-right"><p>69.38</p></td>
 </tr>
 <tr class="row-odd"><td class="text-right"><p>10</p></td>
-<td class="text-right"><p>151.1</p></td>
-<td class="text-right"><p>104.8</p></td>
-<td class="text-right"><p>148.4</p></td>
-<td class="text-right"><p>20</p></td>
+<td class="text-right"><p>67.72</p></td>
+<td class="text-right"><p>35.92</p></td>
 </tr>
 <tr class="row-even"><td class="text-right"><p>11</p></td>
-<td class="text-right"><p>44.7</p></td>
-<td class="text-right"><p>44.1</p></td>
-<td class="text-right"><p>62.0</p></td>
-<td class="text-right"><p>0 [1]</p></td>
+<td class="text-right"><p>26.72</p></td>
+<td class="text-right"><p>30.88</p></td>
 </tr>
 <tr class="row-odd"><td class="text-right"><p>12</p></td>
-<td class="text-right"><p>74.1</p></td>
-<td class="text-right"><p>49.9</p></td>
-<td class="text-right"><p>58.3</p></td>
-<td class="text-right"><p>2</p></td>
+<td class="text-right"><p>24.46</p></td>
+<td class="text-right"><p>10.78</p></td>
 </tr>
 <tr class="row-even"><td class="text-right"><p>13</p></td>
-<td class="text-right"><p>98.7</p></td>
-<td class="text-right"><p>58.3</p></td>
-<td class="text-right"><p>100.4</p></td>
-<td class="text-right"><p>30</p></td>
+<td class="text-right"><p>18.36</p></td>
+<td class="text-right"><p>20.45</p></td>
 </tr>
 <tr class="row-odd"><td class="text-right"><p>14</p></td>
-<td class="text-right"><p>43.0</p></td>
-<td class="text-right"><p>27.4</p></td>
-<td class="text-right"><p>71.1</p></td>
-<td class="text-right"><p>1</p></td>
+<td class="text-right"><p>19.93</p></td>
+<td class="text-right"><p>7.00</p></td>
 </tr>
 <tr class="row-even"><td class="text-right"><p>15</p></td>
-<td class="text-right"><p>121.5</p></td>
-<td class="text-right"><p>65.6</p></td>
-<td class="text-right"><p>53.8</p></td>
-<td class="text-right"><p>1 [2]</p></td>
+<td class="text-right"><p>27.56</p></td>
+<td class="text-right"><p>23.75</p></td>
 </tr>
 <tr class="row-odd"><td class="text-right"><p>16</p></td>
-<td class="text-right"><p>24.5</p></td>
-<td class="text-right"><p>17.5</p></td>
-<td class="text-right"><p>34.3</p></td>
-<td class="text-right"><p>27840</p></td>
+<td class="text-right"><p>18.03</p></td>
+<td class="text-right"><p>23.41</p></td>
 </tr>
 <tr class="row-even"><td class="text-right"><p>17</p></td>
-<td class="text-right"><p>406.7</p></td>
-<td class="text-right"><p>285.8</p></td>
-<td class="text-right"><p>417.8</p></td>
-<td class="text-right"><p>1</p></td>
+<td class="text-right"><p>60.46</p></td>
+<td class="text-right"><p>82.30</p></td>
 </tr>
 <tr class="row-odd"><td class="text-right"><p>18</p></td>
-<td class="text-right"><p>428.8</p></td>
-<td class="text-right"><p>370.5</p></td>
-<td class="text-right"><p>TBD</p></td>
-<td class="text-right"><p>100</p></td>
+<td class="text-right"><p>282.79</p></td>
+<td class="text-right"><p>129.40</p></td>
 </tr>
 <tr class="row-even"><td class="text-right"><p>19</p></td>
-<td class="text-right"><p>58.9</p></td>
-<td class="text-right"><p>35.5</p></td>
-<td class="text-right"><p>68.5</p></td>
-<td class="text-right"><p>1</p></td>
+<td class="text-right"><p>23.28</p></td>
+<td class="text-right"><p>11.26</p></td>
 </tr>
 <tr class="row-odd"><td class="text-right"><p>20</p></td>
-<td class="text-right"><p>105.9</p></td>
-<td class="text-right"><p>67.6</p></td>
-<td class="text-right"><p>93.7</p></td>
-<td class="text-right"><p>110759</p></td>
+<td class="text-right"><p>88.93</p></td>
+<td class="text-right"><p>19.22</p></td>
 </tr>
 <tr class="row-even"><td class="text-right"><p>21</p></td>
-<td class="text-right"><p>562.2</p></td>
-<td class="text-right"><p>460.1</p></td>
-<td class="text-right"><p>698.0</p></td>
-<td class="text-right"><p>100</p></td>
+<td class="text-right"><p>110.87</p></td>
+<td class="text-right"><p>101.53</p></td>
 </tr>
 <tr class="row-odd"><td class="text-right"><p>22</p></td>
-<td class="text-right"><p>36.8</p></td>
-<td class="text-right"><p>21.5</p></td>
-<td class="text-right"><p>22.9</p></td>
-<td class="text-right"><p>7</p></td>
+<td class="text-right"><p>12.17</p></td>
+<td class="text-right"><p>12.71</p></td>
 </tr>
-<tr class="row-even"><td class="text-right"><p><strong>Total (excl. 
Q18)</strong></p></td>
-<td class="text-right"><p><strong>3914.7</strong></p></td>
-<td class="text-right"><p><strong>3705.0</strong></p></td>
-<td class="text-right"><p><strong>3936.8</strong></p></td>
-<td class="text-right"><p></p></td>
+<tr class="row-even"><td class="text-right"><p><strong>Total 
(Q1–Q22)</strong></p></td>
+<td class="text-right"><p><strong>1353.21</strong></p></td>
+<td class="text-right"><p><strong>845.21</strong></p></td>
 </tr>
 </tbody>
 </table>
 </div>
-<p>The <strong>Total</strong> row sums the 21 queries <strong>excluding 
Q18</strong>, because Q18 has no Ballista
-figure at this commit (below), so a 22-query total would not be comparable 
across
-engines. Spark’s and Comet’s Q18 times are in its row.</p>
-<p>Row counts agree across all three engines on every query for which Ballista 
produced
-a result.</p>
-<p>The <strong>AQE-off</strong> Ballista column has been removed pending a 
re-run at a matched core
-count; the numbers above for AQE off were taken at a different executor size 
and are
-not comparable to a fresh AQE-off run, so they were dropped rather than left 
stale.</p>
-<p>[1] Q11 returns 0 rows for every engine at this scale factor: the query’s 
threshold
-constant is tuned for SF1.</p>
-<p>[2] Q15 is a multi-statement query (<code class="docutils literal 
notranslate"><span class="pre">CREATE</span> <span 
class="pre">VIEW</span></code> / <code class="docutils literal 
notranslate"><span class="pre">SELECT</span></code> / <code class="docutils 
literal notranslate"><span class="pre">DROP</span> <span 
class="pre">VIEW</span></code>); all three
-engines report 1 row here. (A previous result set saw Spark report 0 for Q15 
depending
-on which statement the harness took as the result; it does not recur in this 
run.)</p>
-<p><strong>Q18 is not measured at this commit.</strong> Q18’s hash-join build 
side is <code class="docutils literal notranslate"><span 
class="pre">Partitioned</span></code>
-and does not spill; with 16 task slots sharing the 48 GB pool (3 GB per slot), 
the
-per-task build side exceeds the container limit and the executor is OOM-killed
-(<a class="reference external" 
href="https://github.com/apache/datafusion-ballista/issues/2025";>#2025</a>). At 
the previous
-8-slot sizing (6 GB per slot) Q18 completed, so this is a direct consequence 
of the
-denser packing.</p>
-<p>It was excluded from the suite run above with <code class="docutils literal 
notranslate"><span class="pre">--skip</span> <span class="pre">18</span></code> 
so that one query could not
-wedge the other 21, and it has not since been run to completion at this commit 
— hence
-<code class="docutils literal notranslate"><span class="pre">TBD</span></code> 
rather than <code class="docutils literal notranslate"><span 
class="pre">OOM</span></code>. An earlier attempt on this branch did OOM-kill 
an executor,
-but under a different configuration (<code class="docutils literal 
notranslate"><span class="pre">prefer_hash_join=false</span></code>, which 
sends the plan
-down a conversion path that discards the join’s projection and so materialises 
more
-columns than the configuration documented above). That result does not 
transfer, so it
-is not recorded here.</p>
-<p>The <code class="docutils literal notranslate"><span 
class="pre">Rows</span></code> column is the row count the query returned, 
recorded so a time is never
-read without the answer it produced.</p>
-<p>This table records <strong>one current result set</strong>. When results 
are refreshed, the
-table and the pinned versions above are replaced together — a row must never 
mix
-numbers from different commits, because a stale row silently misattributes a
-regression.</p>
-<p><code class="docutils literal notranslate"><span 
class="pre">TBD</span></code> means not yet measured on this cluster at this 
commit; a query that ran but
-did not produce an answer is recorded as <code class="docutils literal 
notranslate"><span class="pre">FAIL</span></code>, or <code class="docutils 
literal notranslate"><span class="pre">OOM</span></code> where the failure is a
-known memory exhaustion.</p>
-<section id="recording-a-result">
-<h3>Recording a result<a class="headerlink" href="#recording-a-result" 
title="Link to this heading">#</a></h3>
-<ul class="simple">
-<li><p>Pin the <strong>exact commit</strong> the numbers came from, not a 
branch name.</p></li>
-<li><p>Report the <strong>AQE-on</strong> number (<code class="docutils 
literal notranslate"><span 
class="pre">ballista.planner.adaptive.enabled=true</span></code>) — the
-configuration this page measures — from the same commit and cluster.</p></li>
-<li><p>Prefer the figure from a <strong>full suite run</strong>, not a 
standalone single-query run: a
-long-lived executor deep into a suite is not in the same state as a freshly 
started
-one, and a single-query job pays a cold-cache penalty that is large enough to 
swamp
-the effect being measured. When a query cannot complete in-suite (e.g. Q18’s 
OOM
-currently wedges the run), exclude it with <code class="docutils literal 
notranslate"><span class="pre">--skip</span> <span 
class="pre">&lt;n&gt;</span></code> so the rest of the column
-still comes from one continuous run, and measure that query 
separately.</p></li>
-<li><p>Note the <strong>row count</strong> each query returned. A fast wrong 
answer is not a result,
-and distributed execution has produced silently wrong row counts 
before.</p></li>
-<li><p>Flag any stage whose runtime is dominated by a few partitions — that is 
the
-imbalance this page exists to surface.</p></li>
-</ul>
+<p>Row counts agree across engines for every query. Ballista completed the
+full 22-query suite in a single continuous run.</p>
 </section>
+<section id="reproducing">
+<h2>Reproducing<a class="headerlink" href="#reproducing" title="Link to this 
heading">#</a></h2>
+<section id="ballista">
+<h3>Ballista<a class="headerlink" href="#ballista" title="Link to this 
heading">#</a></h3>
+<p>Bring up the cluster (one scheduler, N executors), then run the suite from a
+client. Executor sizing on each node:</p>
+<div class="highlight-sh notranslate"><div 
class="highlight"><pre><span></span>ballista-executor<span class="w"> 
</span><span class="se">\</span>
+<span class="w">  </span>--bind-host<span class="w"> </span><span 
class="m">0</span>.0.0.0<span class="w"> </span>--bind-port<span class="w"> 
</span><span class="m">50051</span><span class="w"> </span><span 
class="se">\</span>
+<span class="w">  </span>--scheduler-host<span class="w"> 
</span>&lt;scheduler&gt;<span class="w"> </span>--scheduler-port<span 
class="w"> </span><span class="m">50050</span><span class="w"> </span><span 
class="se">\</span>
+<span class="w">  </span>--concurrent-tasks<span class="w"> </span><span 
class="m">8</span><span class="w"> </span><span class="se">\</span>
+<span class="w">  </span>--memory-pool-size<span class="w"> </span><span 
class="m">48103633715</span>
+</pre></div>
+</div>
+<p>Run all 22 queries:</p>
+<div class="highlight-sh notranslate"><div 
class="highlight"><pre><span></span>tpch<span class="w"> </span>benchmark<span 
class="w"> </span>ballista<span class="w"> </span><span class="se">\</span>
+<span class="w">  </span>--host<span class="w"> </span>&lt;scheduler&gt;<span 
class="w"> </span>--port<span class="w"> </span><span 
class="m">50050</span><span class="w"> </span><span class="se">\</span>
+<span class="w">  </span>--path<span class="w"> 
</span>s3://&lt;bucket&gt;/tpch/sf1000<span class="w"> </span>--format<span 
class="w"> </span>parquet<span class="w"> </span><span class="se">\</span>
+<span class="w">  </span>--partitions<span class="w"> </span><span 
class="m">256</span><span class="w"> </span>--iterations<span class="w"> 
</span><span class="m">2</span><span class="w"> </span><span class="se">\</span>
+<span class="w">  </span>-c<span class="w"> 
</span>ballista.planner.adaptive.enabled<span class="o">=</span><span 
class="nb">true</span><span class="w"> </span><span class="se">\</span>
+<span class="w">  </span>-c<span class="w"> 
</span>datafusion.execution.collect_statistics<span class="o">=</span><span 
class="nb">true</span><span class="w"> </span><span class="se">\</span>
+<span class="w">  </span>-c<span class="w"> 
</span>ballista.shuffle.sort_based.memory_limit_per_task_bytes<span 
class="o">=</span><span class="m">0</span>
+</pre></div>
+</div>
 </section>
-<section id="known-issues-found-by-benchmarking">
-<h2>Known issues found by benchmarking<a class="headerlink" 
href="#known-issues-found-by-benchmarking" title="Link to this 
heading">#</a></h2>
-<p>Benchmarking at SF1000 is how most of the following were found. They are 
worth
-knowing about before interpreting a number:</p>
-<div class="pst-scrollable-table-container"><table class="table">
-<thead>
-<tr class="row-odd"><th class="head"><p>Issue</p></th>
-<th class="head"><p>Summary</p></th>
-</tr>
-</thead>
-<tbody>
-<tr class="row-even"><td><p><a class="reference external" 
href="https://github.com/apache/datafusion-ballista/issues/1359";>#1359</a></p></td>
-<td><p>Umbrella issue for adaptive (AQE) query execution.</p></td>
-</tr>
-<tr class="row-odd"><td><p><a class="reference external" 
href="https://github.com/apache/datafusion-ballista/issues/2025";>#2025</a></p></td>
-<td><p>Q18’s hash-join build side exhausts the memory pool at SF1000. 
DataFusion’s hash-join build side does not spill, so a per-partition build side 
larger than one task slot’s pool fails the task outright.</p></td>
-</tr>
-<tr class="row-even"><td><p><a class="reference external" 
href="https://github.com/apache/datafusion-ballista/issues/2063";>#2063</a></p></td>
-<td><p>AQE can hang when a re-plan cancels an in-flight stage; the job never 
reports completion.</p></td>
-</tr>
-</tbody>
-</table>
+<section id="spark">
+<h3>Spark<a class="headerlink" href="#spark" title="Link to this 
heading">#</a></h3>
+<p>Runs the same queries via <code class="docutils literal notranslate"><span 
class="pre">tpcbench.py</span></code> from
+<a class="reference external" 
href="https://github.com/apache/datafusion-benchmarks";>apache/datafusion-benchmarks</a>,
+with the highlights above and stock Spark 3.4 defaults for everything else:</p>
+<div class="highlight-sh notranslate"><div 
class="highlight"><pre><span></span>spark-submit<span class="w"> </span><span 
class="se">\</span>
+<span class="w">  </span>--master<span class="w"> </span>&lt;master&gt;<span 
class="w"> </span><span class="se">\</span>
+<span class="w">  </span>--conf<span class="w"> 
</span>spark.executor.instances<span class="o">=</span><span 
class="m">32</span><span class="w"> </span><span class="se">\</span>
+<span class="w">  </span>--conf<span class="w"> 
</span>spark.executor.cores<span class="o">=</span><span 
class="m">16</span><span class="w"> </span><span class="se">\</span>
+<span class="w">  </span>--conf<span class="w"> 
</span>spark.executor.memory<span class="o">=</span>64G<span class="w"> 
</span><span class="se">\</span>
+<span class="w">  </span>--conf<span class="w"> 
</span>spark.executor.memoryOverhead<span class="o">=</span>10G<span class="w"> 
</span><span class="se">\</span>
+<span class="w">  </span>--conf<span class="w"> 
</span>spark.sql.shuffle.partitions<span class="o">=</span><span 
class="m">512</span><span class="w"> </span><span class="se">\</span>
+<span class="w">  </span>tpcbench.py<span class="w"> </span><span 
class="se">\</span>
+<span class="w">    </span>--benchmark<span class="w"> </span>tpch<span 
class="w"> </span><span class="se">\</span>
+<span class="w">    </span>--data<span class="w"> 
</span>s3a://&lt;bucket&gt;/tpch/sf1000<span class="w"> </span><span 
class="se">\</span>
+<span class="w">    </span>--format<span class="w"> </span>parquet<span 
class="w"> </span><span class="se">\</span>
+<span class="w">    </span>--iterations<span class="w"> </span><span 
class="m">3</span>
+</pre></div>
 </div>
 </section>
+</section>
+<section id="recording-a-new-result-set">
+<h2>Recording a new result set<a class="headerlink" 
href="#recording-a-new-result-set" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Pin the <strong>exact commit</strong> the numbers came from, not a 
branch name.</p></li>
+<li><p>Replace the version, environment, config, and results tables together — 
a
+row that mixes numbers from different commits silently misattributes a
+regression.</p></li>
+<li><p>Prefer a single continuous suite run: a long-lived executor deep into a
+suite is not in the same state as a freshly started one.</p></li>
+<li><p>Report <code class="docutils literal notranslate"><span 
class="pre">FAIL</span></code> for a query that ran but did not produce an 
answer, and
+<code class="docutils literal notranslate"><span class="pre">OOM</span></code> 
when the failure is a known memory exhaustion. See the tracker for
+open issues found by benchmarking: <a class="reference external" 
href="https://github.com/apache/datafusion-ballista/issues/1359";>#1359</a>,
+<a class="reference external" 
href="https://github.com/apache/datafusion-ballista/issues/2025";>#2025</a>, <a 
class="reference external" 
href="https://github.com/apache/datafusion-ballista/issues/2063";>#2063</a>.</p></li>
+</ul>
+</section>
 </section>
 
 
@@ -952,22 +747,18 @@ knowing about before interpreting a number:</p>
   </div>
   <nav id="pst-page-toc-nav" class="page-toc" 
aria-labelledby="pst-page-navigation-heading-2">
     <ul class="pst-show_toc_level nav section-nav flex-column">
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" 
href="#what-we-measure-and-why">What we measure, and why</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" 
href="#versions-under-test">Versions under test</a></li>
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" 
href="#environment">Environment</a></li>
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" 
href="#reference-cluster">Reference cluster</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" 
href="#ballista-configuration">Ballista configuration</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" 
href="#spark-configuration-highlights">Spark configuration (highlights)</a></li>
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" 
href="#queries">Queries</a></li>
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" 
href="#running-the-benchmark">Running the benchmark</a><ul class="nav 
section-nav flex-column">
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" 
href="#results">Results</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" 
href="#reproducing">Reproducing</a><ul class="nav section-nav flex-column">
 <li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" 
href="#ballista">Ballista</a></li>
 <li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" 
href="#spark">Spark</a></li>
-<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" 
href="#comet">Comet</a></li>
-</ul>
-</li>
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" 
href="#why-compare-against-comet">Why compare against Comet</a></li>
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" 
href="#results">Results</a><ul class="nav section-nav flex-column">
-<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" 
href="#recording-a-result">Recording a result</a></li>
 </ul>
 </li>
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" 
href="#known-issues-found-by-benchmarking">Known issues found by 
benchmarking</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" 
href="#recording-a-new-result-set">Recording a new result set</a></li>
 </ul>
   </nav></div>
 
diff --git a/searchindex.js b/searchindex.js
index 9c8447e7b..188213351 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles":{"AQE join behavior 
(opt-in)":[[27,"aqe-join-behavior-opt-in"]],"Adaptive Query Execution 
(AQE)":[[21,"adaptive-query-execution-aqe"]],"Adaptive Query Execution 
(Experimental)":[[50,"adaptive-query-execution-experimental"]],"Adding a 
persona":[[25,"adding-a-persona"]],"Aggregate 
Functions":[[49,"aggregate-functions"]],"Apache DataFusion 
Ballista":[[26,null]],"Apache DataFusion Ballista 0.10.0 
Changelog":[[0,null]],"Apache DataFusion Ballista 0.11.0 Changelog" [...]
\ No newline at end of file
+Search.setIndex({"alltitles":{"AQE join behavior 
(opt-in)":[[27,"aqe-join-behavior-opt-in"]],"Adaptive Query Execution 
(AQE)":[[21,"adaptive-query-execution-aqe"]],"Adaptive Query Execution 
(Experimental)":[[50,"adaptive-query-execution-experimental"]],"Adding a 
persona":[[25,"adding-a-persona"]],"Aggregate 
Functions":[[49,"aggregate-functions"]],"Apache DataFusion 
Ballista":[[26,null]],"Apache DataFusion Ballista 0.10.0 
Changelog":[[0,null]],"Apache DataFusion Ballista 0.11.0 Changelog" [...]
\ No newline at end of file


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

(datafusion-ballista) branch asf-site updated: Publish built docs triggered by d738ca9bd6b86aecaf9587af9025ae0dd12c5af1

Reply via email to