This is an automated email from the ASF dual-hosted git repository.
andygrove pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/main by this push:
new 73ef55f96f docs: add 1.0.0 TPC-DS benchmark results, remove TPC-H
(#5284)
73ef55f96f is described below
commit 73ef55f96fd0d5e0744cd66c60016c865b55a4d2
Author: Matt Butrovich <[email protected]>
AuthorDate: Thu Aug 6 17:22:44 2026 -0400
docs: add 1.0.0 TPC-DS benchmark results, remove TPC-H (#5284)
* add 1.0.0 tpc-ds benchmark results, remove tpc-h
* Update docs/source/contributor-guide/benchmark-results/tpc-ds.md
Co-authored-by: Andy Grove <[email protected]>
---------
Co-authored-by: Andy Grove <[email protected]>
---
README.md | 4 +-
benchmarks/results/1.0.0/comet-tpcds.json | 329 +++++++++++++++++++++
benchmarks/results/1.0.0/spark-tpcds.json | 326 ++++++++++++++++++++
.../benchmark-results/1.0.0/tpcds_allqueries.png | Bin 0 -> 25588 bytes
.../1.0.0/tpcds_queries_compare.png | Bin 0 -> 37471 bytes
.../1.0.0/tpcds_queries_speedup_abs.png | Bin 0 -> 63779 bytes
.../1.0.0/tpcds_queries_speedup_rel.png | Bin 0 -> 87987 bytes
.../contributor-guide/benchmark-results/tpc-ds.md | 10 +-
.../contributor-guide/benchmark-results/tpc-h.md | 79 -----
docs/source/contributor-guide/benchmarking.md | 2 -
docs/source/contributor-guide/roadmap.md | 5 +-
11 files changed, 664 insertions(+), 91 deletions(-)
diff --git a/README.md b/README.md
index 7725a45db6..6eccd07816 100644
--- a/README.md
+++ b/README.md
@@ -50,9 +50,9 @@ lower cloud bills, reduced on-prem capacity, and lower energy
usage, with no cha
DataFrame, or PySpark code. Comet runs on commodity hardware: no GPUs, FPGAs,
or other specialized accelerators are
required, so the savings come from better utilization of the infrastructure
you already run on.
-
+
-
+
See the [Comet Benchmarking
Guide](https://datafusion.apache.org/comet/contributor-guide/benchmarking.html)
for more details.
diff --git a/benchmarks/results/1.0.0/comet-tpcds.json
b/benchmarks/results/1.0.0/comet-tpcds.json
new file mode 100644
index 0000000000..6114f813cc
--- /dev/null
+++ b/benchmarks/results/1.0.0/comet-tpcds.json
@@ -0,0 +1,329 @@
+{
+ "engine": "comet",
+ "benchmark": "tpcds",
+ "format": "parquet",
+ "arch": "unknown",
+ "tags": {
+ "name": "tpcds",
+ "codec": "zstd",
+ "engine": "comet",
+ "format": "parquet",
+ "shuffle": "auto",
+ "comet_exec": "all except columnarToRow.native",
+ "scale_factor": "1000",
+ "spark_profile": "3.5_2.13"
+ },
+ "spark_conf": {
+ "spark.comet.exec.columnarToRow.native.enabled": "false",
+ "spark.comet.exec.memoryPool.fraction": "0.8"
+ },
+ "1": [
+ 6.3340000000000005
+ ],
+ "2": [
+ 7.683
+ ],
+ "3": [
+ 2.06
+ ],
+ "4": [
+ 18.740499999999997
+ ],
+ "5": [
+ 6.904999999999999
+ ],
+ "6": [
+ 1.6795
+ ],
+ "7": [
+ 2.8155
+ ],
+ "8": [
+ 1.814
+ ],
+ "9": [
+ 10.7955
+ ],
+ "10": [
+ 3.2039999999999997
+ ],
+ "11": [
+ 10.2075
+ ],
+ "12": [
+ 0.98
+ ],
+ "13": [
+ 2.596
+ ],
+ "15": [
+ 1.653
+ ],
+ "16": [
+ 5.874499999999999
+ ],
+ "17": [
+ 3.1105
+ ],
+ "18": [
+ 3.6715
+ ],
+ "19": [
+ 1.4205
+ ],
+ "20": [
+ 0.974
+ ],
+ "21": [
+ 0.8999999999999999
+ ],
+ "22": [
+ 3.02
+ ],
+ "25": [
+ 2.2935
+ ],
+ "26": [
+ 2.019
+ ],
+ "27": [
+ 3.33
+ ],
+ "28": [
+ 16.4965
+ ],
+ "29": [
+ 5.4030000000000005
+ ],
+ "30": [
+ 2.1115
+ ],
+ "31": [
+ 4.044
+ ],
+ "32": [
+ 1.0425
+ ],
+ "33": [
+ 1.6015
+ ],
+ "34": [
+ 1.9144999999999999
+ ],
+ "35": [
+ 4.3095
+ ],
+ "36": [
+ 2.814
+ ],
+ "37": [
+ 3.8335
+ ],
+ "38": [
+ 5.846500000000001
+ ],
+ "40": [
+ 3.5585
+ ],
+ "41": [
+ 0.5820000000000001
+ ],
+ "42": [
+ 0.9219999999999999
+ ],
+ "43": [
+ 2.391
+ ],
+ "44": [
+ 10.347000000000001
+ ],
+ "45": [
+ 1.8325
+ ],
+ "46": [
+ 2.8215000000000003
+ ],
+ "47": [
+ 7.792
+ ],
+ "48": [
+ 2.742
+ ],
+ "49": [
+ 4.2835
+ ],
+ "50": [
+ 14.4375
+ ],
+ "51": [
+ 5.838
+ ],
+ "52": [
+ 0.96
+ ],
+ "53": [
+ 2.3095
+ ],
+ "54": [
+ 7.2835
+ ],
+ "55": [
+ 1.3895
+ ],
+ "56": [
+ 1.6355
+ ],
+ "57": [
+ 3.5245
+ ],
+ "58": [
+ 1.371
+ ],
+ "59": [
+ 8.985
+ ],
+ "60": [
+ 1.626
+ ],
+ "61": [
+ 1.9115000000000002
+ ],
+ "62": [
+ 4.8315
+ ],
+ "63": [
+ 2.459
+ ],
+ "64": [
+ 11.4775
+ ],
+ "65": [
+ 4.9384999999999994
+ ],
+ "66": [
+ 2.6685
+ ],
+ "67": [
+ 29.274
+ ],
+ "68": [
+ 1.417
+ ],
+ "69": [
+ 2.1574999999999998
+ ],
+ "70": [
+ 5.5115
+ ],
+ "71": [
+ 1.4834999999999998
+ ],
+ "72": [
+ 10.6365
+ ],
+ "73": [
+ 0.9359999999999999
+ ],
+ "74": [
+ 7.454
+ ],
+ "75": [
+ 12.714500000000001
+ ],
+ "76": [
+ 8.5875
+ ],
+ "77": [
+ 1.419
+ ],
+ "78": [
+ 12.245000000000001
+ ],
+ "79": [
+ 2.1014999999999997
+ ],
+ "80": [
+ 5.086
+ ],
+ "81": [
+ 3.3535000000000004
+ ],
+ "82": [
+ 4.707
+ ],
+ "83": [
+ 1.032
+ ],
+ "84": [
+ 2.949
+ ],
+ "85": [
+ 3.436
+ ],
+ "86": [
+ 1.6745
+ ],
+ "87": [
+ 5.5794999999999995
+ ],
+ "88": [
+ 26.060499999999998
+ ],
+ "89": [
+ 2.2880000000000003
+ ],
+ "90": [
+ 4.696999999999999
+ ],
+ "91": [
+ 0.7825
+ ],
+ "92": [
+ 1.1075
+ ],
+ "93": [
+ 16.4155
+ ],
+ "94": [
+ 5.6240000000000006
+ ],
+ "95": [
+ 11.2075
+ ],
+ "96": [
+ 4.7940000000000005
+ ],
+ "97": [
+ 4.3919999999999995
+ ],
+ "98": [
+ 1.5605
+ ],
+ "99": [
+ 4.8545
+ ],
+ "14a": [
+ 24.436999999999998
+ ],
+ "14b": [
+ 22.16
+ ],
+ "23a": [
+ 29.295499999999997
+ ],
+ "23b": [
+ 37.251000000000005
+ ],
+ "24a": [
+ 22.159
+ ],
+ "24b": [
+ 20.985999999999997
+ ],
+ "39a": [
+ 2.8255
+ ],
+ "39b": [
+ 2.3885
+ ]
+}
\ No newline at end of file
diff --git a/benchmarks/results/1.0.0/spark-tpcds.json
b/benchmarks/results/1.0.0/spark-tpcds.json
new file mode 100644
index 0000000000..1cd03a595f
--- /dev/null
+++ b/benchmarks/results/1.0.0/spark-tpcds.json
@@ -0,0 +1,326 @@
+{
+ "engine": "comet",
+ "benchmark": "tpcds",
+ "format": "parquet",
+ "arch": "unknown",
+ "tags": {
+ "name": "tpcds",
+ "codec": "zstd",
+ "engine": "comet",
+ "format": "parquet",
+ "shuffle": "auto",
+ "comet_exec": "all",
+ "scale_factor": "1000",
+ "spark_profile": "3.5_2.13"
+ },
+ "spark_conf": {},
+ "1": [
+ 6.809
+ ],
+ "2": [
+ 12.9895
+ ],
+ "3": [
+ 4.234
+ ],
+ "4": [
+ 35.810500000000005
+ ],
+ "5": [
+ 15.868500000000001
+ ],
+ "6": [
+ 3.4825
+ ],
+ "7": [
+ 7.9645
+ ],
+ "8": [
+ 2.9074999999999998
+ ],
+ "9": [
+ 44.2675
+ ],
+ "10": [
+ 5.3785
+ ],
+ "11": [
+ 17.612499999999997
+ ],
+ "12": [
+ 2.0495
+ ],
+ "13": [
+ 10.3625
+ ],
+ "15": [
+ 2.38
+ ],
+ "16": [
+ 17.723
+ ],
+ "17": [
+ 4.9510000000000005
+ ],
+ "18": [
+ 5.947
+ ],
+ "19": [
+ 2.2720000000000002
+ ],
+ "20": [
+ 2.5325
+ ],
+ "21": [
+ 2.7560000000000002
+ ],
+ "22": [
+ 8.6765
+ ],
+ "25": [
+ 4.0285
+ ],
+ "26": [
+ 3.7569999999999997
+ ],
+ "27": [
+ 6.9325
+ ],
+ "28": [
+ 46.788
+ ],
+ "29": [
+ 12.363
+ ],
+ "30": [
+ 3.6425
+ ],
+ "31": [
+ 5.7010000000000005
+ ],
+ "32": [
+ 1.668
+ ],
+ "33": [
+ 3.1565000000000003
+ ],
+ "34": [
+ 4.8925
+ ],
+ "35": [
+ 6.4285
+ ],
+ "36": [
+ 6.4035
+ ],
+ "37": [
+ 7.8355
+ ],
+ "38": [
+ 12.9265
+ ],
+ "40": [
+ 6.959
+ ],
+ "41": [
+ 1.2125
+ ],
+ "42": [
+ 1.28
+ ],
+ "43": [
+ 4.897
+ ],
+ "44": [
+ 28.354
+ ],
+ "45": [
+ 2.6455
+ ],
+ "46": [
+ 6.6165
+ ],
+ "47": [
+ 8.8265
+ ],
+ "48": [
+ 7.9384999999999994
+ ],
+ "49": [
+ 10.405999999999999
+ ],
+ "50": [
+ 28.986
+ ],
+ "51": [
+ 10.7075
+ ],
+ "52": [
+ 1.8290000000000002
+ ],
+ "53": [
+ 6.7175
+ ],
+ "54": [
+ 3.589
+ ],
+ "55": [
+ 1.6265
+ ],
+ "56": [
+ 3.7504999999999997
+ ],
+ "57": [
+ 5.2855
+ ],
+ "58": [
+ 3.0045
+ ],
+ "59": [
+ 21.016
+ ],
+ "60": [
+ 3.622
+ ],
+ "61": [
+ 4.2
+ ],
+ "62": [
+ 10.746500000000001
+ ],
+ "63": [
+ 6.66
+ ],
+ "64": [
+ 36.2755
+ ],
+ "65": [
+ 11.690999999999999
+ ],
+ "66": [
+ 5.037
+ ],
+ "67": [
+ 48.193
+ ],
+ "68": [
+ 2.806
+ ],
+ "69": [
+ 4.5585
+ ],
+ "70": [
+ 9.5135
+ ],
+ "71": [
+ 2.178
+ ],
+ "72": [
+ 13.225999999999999
+ ],
+ "73": [
+ 2.2885
+ ],
+ "74": [
+ 15.521
+ ],
+ "75": [
+ 19.834
+ ],
+ "76": [
+ 20.462
+ ],
+ "77": [
+ 1.7915
+ ],
+ "78": [
+ 28.689999999999998
+ ],
+ "79": [
+ 4.1615
+ ],
+ "80": [
+ 9.7425
+ ],
+ "81": [
+ 4.734500000000001
+ ],
+ "82": [
+ 16.2755
+ ],
+ "83": [
+ 2.0785
+ ],
+ "84": [
+ 3.9640000000000004
+ ],
+ "85": [
+ 7.2575
+ ],
+ "86": [
+ 3.729
+ ],
+ "87": [
+ 11.461500000000001
+ ],
+ "88": [
+ 59.0305
+ ],
+ "89": [
+ 5.8895
+ ],
+ "90": [
+ 11.314
+ ],
+ "91": [
+ 1.4569999999999999
+ ],
+ "92": [
+ 1.646
+ ],
+ "93": [
+ 40.361000000000004
+ ],
+ "94": [
+ 15.7425
+ ],
+ "95": [
+ 30.113500000000002
+ ],
+ "96": [
+ 15.469
+ ],
+ "97": [
+ 11.235
+ ],
+ "98": [
+ 2.902
+ ],
+ "99": [
+ 11.688500000000001
+ ],
+ "14a": [
+ 51.202
+ ],
+ "14b": [
+ 43.394499999999994
+ ],
+ "23a": [
+ 83.11699999999999
+ ],
+ "23b": [
+ 98.0335
+ ],
+ "24a": [
+ 50.3245
+ ],
+ "24b": [
+ 46.4325
+ ],
+ "39a": [
+ 5.8175
+ ],
+ "39b": [
+ 5.401999999999999
+ ]
+}
\ No newline at end of file
diff --git
a/docs/source/_static/images/benchmark-results/1.0.0/tpcds_allqueries.png
b/docs/source/_static/images/benchmark-results/1.0.0/tpcds_allqueries.png
new file mode 100644
index 0000000000..9c12e3e439
Binary files /dev/null and
b/docs/source/_static/images/benchmark-results/1.0.0/tpcds_allqueries.png differ
diff --git
a/docs/source/_static/images/benchmark-results/1.0.0/tpcds_queries_compare.png
b/docs/source/_static/images/benchmark-results/1.0.0/tpcds_queries_compare.png
new file mode 100644
index 0000000000..d1bc3a917b
Binary files /dev/null and
b/docs/source/_static/images/benchmark-results/1.0.0/tpcds_queries_compare.png
differ
diff --git
a/docs/source/_static/images/benchmark-results/1.0.0/tpcds_queries_speedup_abs.png
b/docs/source/_static/images/benchmark-results/1.0.0/tpcds_queries_speedup_abs.png
new file mode 100644
index 0000000000..74cfc52eac
Binary files /dev/null and
b/docs/source/_static/images/benchmark-results/1.0.0/tpcds_queries_speedup_abs.png
differ
diff --git
a/docs/source/_static/images/benchmark-results/1.0.0/tpcds_queries_speedup_rel.png
b/docs/source/_static/images/benchmark-results/1.0.0/tpcds_queries_speedup_rel.png
new file mode 100644
index 0000000000..764badd455
Binary files /dev/null and
b/docs/source/_static/images/benchmark-results/1.0.0/tpcds_queries_speedup_rel.png
differ
diff --git a/docs/source/contributor-guide/benchmark-results/tpc-ds.md
b/docs/source/contributor-guide/benchmark-results/tpc-ds.md
index 0e96b977d0..c20d76abd1 100644
--- a/docs/source/contributor-guide/benchmark-results/tpc-ds.md
+++ b/docs/source/contributor-guide/benchmark-results/tpc-ds.md
@@ -17,7 +17,7 @@ specific language governing permissions and limitations
under the License.
-->
-# Apache DataFusion Comet: Benchmarks Derived From TPC-DS
+# TPC-DS
The following benchmarks were performed on an EKS cluster (`r6i.24xlarge`
instances with EBS storage) with data stored in S3.
@@ -25,19 +25,19 @@ The following benchmarks were performed on an EKS cluster
(`r6i.24xlarge` instan
Total time to run all queries (lower is better).
-
+
Per-query breakdown showing the relative performance of Spark and Comet.
-
+
How much Comet accelerates each query in relative terms.
-
+
How much Comet accelerates each query in absolute terms.
-
+
## Configuration
diff --git a/docs/source/contributor-guide/benchmark-results/tpc-h.md
b/docs/source/contributor-guide/benchmark-results/tpc-h.md
deleted file mode 100644
index cb6dc8873f..0000000000
--- a/docs/source/contributor-guide/benchmark-results/tpc-h.md
+++ /dev/null
@@ -1,79 +0,0 @@
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-# Apache DataFusion Comet: Benchmarks Derived From TPC-H
-
-The following benchmarks were performed on an EKS cluster (`r6i.24xlarge`
instances with EBS storage) with data stored in S3.
-
-## Benchmark Results
-
-Total time to run all queries (lower is better).
-
-
-
-The following charts are based on the tuned run using hash join.
-
-Per-query breakdown showing the relative performance of Spark and Comet.
-
-
-
-How much Comet accelerates each query in relative terms.
-
-
-
-How much Comet accelerates each query in absolute terms.
-
-
-
-## Configuration
-
-Common:
-
-```properties
-spark.executor.instances=32
-spark.executor.cores=16
-spark.memory.fraction=0.6
-spark.memory.storageFraction=0.2
-# Kubernetes CPU constraints
-spark.kubernetes.executor.request.cores=8
-spark.kubernetes.executor.limit.cores=8
-```
-
-Spark:
-
-```properties
-spark.executor.memory=64G
-spark.executor.memoryOverhead=10G
-```
-
-Comet:
-
-```properties
-spark.executor.memory=32G
-spark.executor.memoryOverhead=10G
-spark.memory.offHeap.enabled=true
-spark.memory.offHeap.size=32G
-```
-
-### Comet (Tuned)
-
-```properties
-spark.comet.exec.forceShuffledHashJoin=true
-spark.comet.exec.memoryPool.fraction=0.8
-```
diff --git a/docs/source/contributor-guide/benchmarking.md
b/docs/source/contributor-guide/benchmarking.md
index 4bfd5ae571..af6caf414d 100644
--- a/docs/source/contributor-guide/benchmarking.md
+++ b/docs/source/contributor-guide/benchmarking.md
@@ -27,7 +27,6 @@ Data generation scripts are available in the [DataFusion
Benchmarks](https://git
## Current Benchmark Results
-- [Benchmarks derived from TPC-H](benchmark-results/tpc-h)
- [Benchmarks derived from TPC-DS](benchmark-results/tpc-ds)
## Benchmarking Guides
@@ -43,7 +42,6 @@ We also have many micro benchmarks that can be run from an
IDE located [here](ht
```{toctree}
:hidden:
-benchmark-results/tpc-h
benchmark-results/tpc-ds
benchmarking_macos
benchmarking_aws_ec2
diff --git a/docs/source/contributor-guide/roadmap.md
b/docs/source/contributor-guide/roadmap.md
index 4fd9e16902..aa7b643232 100644
--- a/docs/source/contributor-guide/roadmap.md
+++ b/docs/source/contributor-guide/roadmap.md
@@ -82,13 +82,12 @@ scoping what that work would take.
## TPC-H and TPC-DS Performance
-Comet already delivers substantial speedups over vanilla Spark on both
benchmark suites; we publish per-query
-results for [TPC-H] and [TPC-DS] with each release. An independent [AWS Labs
benchmark] comparing Comet 0.16.0 with
+Comet already delivers substantial speedups over vanilla Spark on TPC-H and
TPC-DS; we publish per-query
+results for [TPC-DS] with each release. An independent [AWS Labs benchmark]
comparing Comet 0.16.0 with
Gluten 1.6.0 on a 3TB TPC-DS workload found that the two accelerators deliver
similar overall performance. Increasing
the speedup further and closing the remaining per-query gaps is an ongoing
focus, tracked under [#2004] (TPC-H) and
[#858] (TPC-DS).
-[TPC-H]: benchmark-results/tpc-h.md
[TPC-DS]: benchmark-results/tpc-ds.md
[AWS Labs benchmark]:
https://awslabs.github.io/data-on-eks/docs/benchmarks/spark-gluten-velox-comet-benchmark
[#858]: https://github.com/apache/datafusion-comet/issues/858
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
