This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 133bd7795 Publish built docs triggered by
f65226a7d92cc5280336c06e3869d0f23297829e
133bd7795 is described below
commit 133bd77959a196a8a087839c92360deca38946e0
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Tue May 12 14:04:44 2026 +0000
Publish built docs triggered by f65226a7d92cc5280336c06e3869d0f23297829e
---
_images/tpcds_allqueries.png | Bin 24170 -> 25524 bytes
_images/tpcds_queries_compare.png | Bin 36249 -> 37679 bytes
_images/tpcds_queries_speedup_abs.png | Bin 59986 -> 64879 bytes
_images/tpcds_queries_speedup_rel.png | Bin 80451 -> 92557 bytes
_images/tpch_queries_compare.png | Bin 26102 -> 25488 bytes
...eries_compare.png => tpch_queries_compare1.png} | Bin
_images/tpch_queries_speedup_abs.png | Bin 31955 -> 32108 bytes
_images/tpch_queries_speedup_rel.png | Bin 39743 -> 36118 bytes
.../benchmark-results/tpc-ds.md.txt | 34 +++++++--------
.../benchmark-results/tpc-h.md.txt | 48 +++++++++------------
_sources/user-guide/latest/datasources.md.txt | 2 +-
_sources/user-guide/latest/iceberg.md.txt | 6 +--
_sources/user-guide/latest/index.rst.txt | 8 ++--
_sources/user-guide/latest/installation.md.txt | 8 ++--
_sources/user-guide/latest/kubernetes.md.txt | 6 +--
.../benchmark-results/0.16.0/tpcds_allqueries.png | Bin 0 -> 25524 bytes
.../0.16.0/tpcds_queries_compare.png | Bin 0 -> 37679 bytes
.../0.16.0/tpcds_queries_speedup_abs.png | Bin 0 -> 64879 bytes
.../0.16.0/tpcds_queries_speedup_rel.png | Bin 0 -> 92557 bytes
.../benchmark-results/0.16.0/tpch_allqueries.png | Bin 0 -> 23705 bytes
.../0.16.0/tpch_queries_compare.png | Bin 0 -> 25488 bytes
.../0.16.0/tpch_queries_speedup_abs.png | Bin 0 -> 32108 bytes
.../0.16.0/tpch_queries_speedup_rel.png | Bin 0 -> 36118 bytes
contributor-guide/benchmark-results/tpc-ds.html | 22 +++++-----
contributor-guide/benchmark-results/tpc-h.html | 35 +++++++--------
index.html | 2 +-
objects.inv | Bin 1952 -> 1953 bytes
searchindex.js | 2 +-
user-guide/index.html | 4 +-
.../compatibility/expressions/aggregate.html | 2 +-
.../latest/compatibility/expressions/array.html | 2 +-
.../latest/compatibility/expressions/cast.html | 2 +-
.../latest/compatibility/expressions/datetime.html | 2 +-
.../latest/compatibility/expressions/index.html | 2 +-
.../latest/compatibility/expressions/map.html | 2 +-
.../latest/compatibility/expressions/math.html | 2 +-
.../latest/compatibility/expressions/misc.html | 2 +-
.../latest/compatibility/expressions/string.html | 2 +-
.../latest/compatibility/expressions/struct.html | 2 +-
.../latest/compatibility/floating-point.html | 2 +-
user-guide/latest/compatibility/index.html | 2 +-
user-guide/latest/compatibility/operators.html | 2 +-
user-guide/latest/compatibility/regex.html | 2 +-
user-guide/latest/compatibility/scans.html | 2 +-
.../latest/compatibility/spark-versions.html | 2 +-
user-guide/latest/configs.html | 2 +-
user-guide/latest/datasources.html | 4 +-
user-guide/latest/datatypes.html | 2 +-
user-guide/latest/expressions.html | 2 +-
user-guide/latest/iceberg.html | 8 ++--
user-guide/latest/index.html | 12 +++---
user-guide/latest/installation.html | 14 +++---
user-guide/latest/kubernetes.html | 8 ++--
user-guide/latest/metrics.html | 2 +-
user-guide/latest/operators.html | 2 +-
user-guide/latest/source.html | 2 +-
user-guide/latest/tuning.html | 2 +-
user-guide/latest/understanding-comet-plans.html | 2 +-
58 files changed, 130 insertions(+), 141 deletions(-)
diff --git a/_images/tpcds_allqueries.png b/_images/tpcds_allqueries.png
index 7f7853789..211309a76 100644
Binary files a/_images/tpcds_allqueries.png and b/_images/tpcds_allqueries.png
differ
diff --git a/_images/tpcds_queries_compare.png
b/_images/tpcds_queries_compare.png
index 841b4565f..22820b81e 100644
Binary files a/_images/tpcds_queries_compare.png and
b/_images/tpcds_queries_compare.png differ
diff --git a/_images/tpcds_queries_speedup_abs.png
b/_images/tpcds_queries_speedup_abs.png
index 81367f9ce..6e49ae4a2 100644
Binary files a/_images/tpcds_queries_speedup_abs.png and
b/_images/tpcds_queries_speedup_abs.png differ
diff --git a/_images/tpcds_queries_speedup_rel.png
b/_images/tpcds_queries_speedup_rel.png
index 6ef088acb..4f25fa863 100644
Binary files a/_images/tpcds_queries_speedup_rel.png and
b/_images/tpcds_queries_speedup_rel.png differ
diff --git a/_images/tpch_queries_compare.png b/_images/tpch_queries_compare.png
index 81f19707e..e86d43ab3 100644
Binary files a/_images/tpch_queries_compare.png and
b/_images/tpch_queries_compare.png differ
diff --git a/_images/tpch_queries_compare.png
b/_images/tpch_queries_compare1.png
similarity index 100%
copy from _images/tpch_queries_compare.png
copy to _images/tpch_queries_compare1.png
diff --git a/_images/tpch_queries_speedup_abs.png
b/_images/tpch_queries_speedup_abs.png
index 917a84bc6..016e7226e 100644
Binary files a/_images/tpch_queries_speedup_abs.png and
b/_images/tpch_queries_speedup_abs.png differ
diff --git a/_images/tpch_queries_speedup_rel.png
b/_images/tpch_queries_speedup_rel.png
index 69ba4c8ce..bf4ba2f3b 100644
Binary files a/_images/tpch_queries_speedup_rel.png and
b/_images/tpch_queries_speedup_rel.png differ
diff --git a/_sources/contributor-guide/benchmark-results/tpc-ds.md.txt
b/_sources/contributor-guide/benchmark-results/tpc-ds.md.txt
index 23e54612d..7a988a508 100644
--- a/_sources/contributor-guide/benchmark-results/tpc-ds.md.txt
+++ b/_sources/contributor-guide/benchmark-results/tpc-ds.md.txt
@@ -21,7 +21,23 @@ under the License.
The following benchmarks were performed on an EKS cluster (`r6i.24xlarge`
instances with EBS storage) with data stored in S3.
-The tracking issue for improving TPC-DS performance is
[#858](https://github.com/apache/datafusion-comet/issues/858).
+## Benchmark Results
+
+Total time to run all queries (lower is better).
+
+
+
+Per-query breakdown showing the relative performance of Spark and Comet.
+
+
+
+How much Comet accelerates each query in relative terms.
+
+
+
+How much Comet accelerates each query in absolute terms.
+
+
## Configuration
@@ -53,19 +69,3 @@ spark.memory.offHeap.enabled=true
spark.memory.offHeap.size=32G
spark.comet.memoryPool.fraction=0.8
```
-
-## Benchmark Results
-
-
-
-Here is a breakdown showing relative performance of Spark and Comet for each
query.
-
-
-
-The following chart shows how much Comet currently accelerates each query from
the benchmark in relative terms.
-
-
-
-The following chart shows how much Comet currently accelerates each query from
the benchmark in absolute terms.
-
-
diff --git a/_sources/contributor-guide/benchmark-results/tpc-h.md.txt
b/_sources/contributor-guide/benchmark-results/tpc-h.md.txt
index 09ef122e4..5a3ca2141 100644
--- a/_sources/contributor-guide/benchmark-results/tpc-h.md.txt
+++ b/_sources/contributor-guide/benchmark-results/tpc-h.md.txt
@@ -21,6 +21,26 @@ under the License.
The following benchmarks were performed on an EKS cluster (`r6i.24xlarge`
instances with EBS storage) with data stored in S3.
+## Benchmark Results
+
+Total time to run all queries (lower is better).
+
+
+
+The following charts are based on the tuned run using hash join.
+
+Per-query breakdown showing the relative performance of Spark and Comet.
+
+
+
+How much Comet accelerates each query in relative terms.
+
+
+
+How much Comet accelerates each query in absolute terms.
+
+
+
## Configuration
Common:
@@ -51,35 +71,9 @@ spark.memory.offHeap.enabled=true
spark.memory.offHeap.size=32G
```
-Comet (Tuned):
+### Comet (Tuned)
```properties
-spark.executor.memory=32G
-spark.executor.memoryOverhead=10G
-spark.memory.offHeap.enabled=true
-spark.memory.offHeap.size=32G
spark.comet.exec.replaceSortMergeJoin=true
spark.comet.memoryPool.fraction=0.8
```
-
-## Benchmark Results
-
-The following chart shows benchmark results comparing Spark to Comet, both
with Comet's default settings, and with Hash Join enabled in Comet.
-
-Comet's Hash Join does not support spilling yet, so it isn't suitable for all
workloads.
-
-
-
-## Comet (with Hash Join enabled)
-
-Here is a breakdown showing relative performance of Spark and Comet for each
query.
-
-
-
-The following chart shows how much Comet currently accelerates each query from
the benchmark in relative terms.
-
-
-
-The following chart shows how much Comet currently accelerates each query from
the benchmark in absolute terms.
-
-
diff --git a/_sources/user-guide/latest/datasources.md.txt
b/_sources/user-guide/latest/datasources.md.txt
index a887e339e..8808bce6d 100644
--- a/_sources/user-guide/latest/datasources.md.txt
+++ b/_sources/user-guide/latest/datasources.md.txt
@@ -99,7 +99,7 @@ root
| |-- lastName: string (nullable = true)
| |-- ageInYears: integer (nullable = true)
-25/01/30 16:50:43 INFO core/src/lib.rs: Comet native library version
0.16.0-SNAPSHOT initialized
+25/01/30 16:50:43 INFO core/src/lib.rs: Comet native library version
0.17.0-SNAPSHOT initialized
== Physical Plan ==
* CometColumnarToRow (2)
+- CometNativeScan: (1)
diff --git a/_sources/user-guide/latest/iceberg.md.txt
b/_sources/user-guide/latest/iceberg.md.txt
index aa150177a..862fa9b6b 100644
--- a/_sources/user-guide/latest/iceberg.md.txt
+++ b/_sources/user-guide/latest/iceberg.md.txt
@@ -25,7 +25,7 @@ Comet's native Iceberg reader relies on reflection to extract
`FileScanTask`s fr
then serialized to Comet's native execution engine (see
[PR #2528](https://github.com/apache/datafusion-comet/pull/2528)).
-The example below uses Spark's package downloader to retrieve Comet
0.16.0-SNAPSHOT and Iceberg
+The example below uses Spark's package downloader to retrieve Comet
0.17.0-SNAPSHOT and Iceberg
1.8.1, but Comet has been tested with Iceberg 1.5, 1.7, 1.8, 1.9, and 1.10.
The native Iceberg
reader is enabled by default. To disable it, set
`spark.comet.scan.icebergNative.enabled=false`.
@@ -35,7 +35,7 @@ matching your Spark and Scala versions (Comet also ships
Spark 3.5 / Scala 2.13
```shell
$SPARK_HOME/bin/spark-shell \
- --packages
org.apache.datafusion:comet-spark-spark3.5_2.12:0.16.0-SNAPSHOT,org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.8.1,org.apache.iceberg:iceberg-core:1.8.1
\
+ --packages
org.apache.datafusion:comet-spark-spark3.5_2.12:0.17.0-SNAPSHOT,org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.8.1,org.apache.iceberg:iceberg-core:1.8.1
\
--repositories https://repo1.maven.org/maven2/ \
--conf
spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
\
--conf
spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkCatalog \
@@ -109,7 +109,7 @@ configure Spark to use a REST catalog with Comet's native
Iceberg scan:
```shell
$SPARK_HOME/bin/spark-shell \
- --packages
org.apache.datafusion:comet-spark-spark3.5_2.12:0.16.0-SNAPSHOT,org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.8.1,org.apache.iceberg:iceberg-core:1.8.1
\
+ --packages
org.apache.datafusion:comet-spark-spark3.5_2.12:0.17.0-SNAPSHOT,org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.8.1,org.apache.iceberg:iceberg-core:1.8.1
\
--repositories https://repo1.maven.org/maven2/ \
--conf
spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
\
--conf spark.sql.catalog.rest_cat=org.apache.iceberg.spark.SparkCatalog \
diff --git a/_sources/user-guide/latest/index.rst.txt
b/_sources/user-guide/latest/index.rst.txt
index f7e48ec22..7af4b96b5 100644
--- a/_sources/user-guide/latest/index.rst.txt
+++ b/_sources/user-guide/latest/index.rst.txt
@@ -19,10 +19,10 @@
:alt: DataFusion Comet Logo
================================
-Comet 0.16.0-SNAPSHOT User Guide
+Comet 0.17.0-SNAPSHOT User Guide
================================
-This guide covers Comet 0.16.0-SNAPSHOT: how to install it, build it from
source, configure it for
+This guide covers Comet 0.17.0-SNAPSHOT: how to install it, build it from
source, configure it for
your Spark deployment, and get the best results from it. It also documents the
data sources, data
types, operators, and expressions that Comet supports, along with a
compatibility guide describing
known differences from Apache Spark.
@@ -31,10 +31,10 @@ Operational topics include reading and understanding Comet
query plans, tuning,
and integration guides for Apache Iceberg and Kubernetes. Select a topic from
the navigation menu
to read more.
-.. _toc.user-guide-links-0.16.0-SNAPSHOT:
+.. _toc.user-guide-links-0.17.0-SNAPSHOT:
.. toctree::
:maxdepth: 1
- :caption: Comet 0.16.0-SNAPSHOT User Guide
+ :caption: Comet 0.17.0-SNAPSHOT User Guide
:hidden:
Installing Comet <installation>
diff --git a/_sources/user-guide/latest/installation.md.txt
b/_sources/user-guide/latest/installation.md.txt
index e4ecda0de..8d8e06cb0 100644
--- a/_sources/user-guide/latest/installation.md.txt
+++ b/_sources/user-guide/latest/installation.md.txt
@@ -36,7 +36,7 @@ users must [build from source](source.md).
### Supported Spark Versions
-Comet 0.16.0-SNAPSHOT supports the following versions of Apache Spark. Refer
to the [Spark Version Compatibility] page
+Comet 0.17.0-SNAPSHOT supports the following versions of Apache Spark. Refer
to the [Spark Version Compatibility] page
in the [Compatibility Guide] for more information, such as known limitations
per Spark version.
[Spark Version Compatibility]: compatibility/spark-versions.md
@@ -88,7 +88,7 @@ See the [Comet Kubernetes Guide](kubernetes.md) guide.
Make sure `SPARK_HOME` points to the same Spark version as Comet was built for.
```shell
-export COMET_JAR=spark/target/comet-spark-spark4.1_2.13-0.16.0-SNAPSHOT.jar
+export COMET_JAR=spark/target/comet-spark-spark4.1_2.13-0.17.0-SNAPSHOT.jar
$SPARK_HOME/bin/spark-shell \
--jars $COMET_JAR \
@@ -112,7 +112,7 @@ scala> (0 until
10).toDF("a").write.mode("overwrite").parquet("/tmp/test")
Comet will log output similar to:
```shell
-INFO core/src/lib.rs: Comet native library version 0.16.0-SNAPSHOT initialized
+INFO core/src/lib.rs: Comet native library version 0.17.0-SNAPSHOT initialized
WARN CometExecRule: Comet cannot execute some parts of this plan natively (set
spark.comet.explainFallback.enabled=false to disable this logging):
Execute InsertIntoHadoopFsRelationCommand [COMET: Native support for
operator DataWritingCommandExec is disabled. Set
spark.comet.parquet.write.enabled=true to enable it.]
+- WriteFiles
@@ -145,7 +145,7 @@ explicitly contain Comet otherwise Spark may use a
different class-loader for th
components which will then fail at runtime. For example:
```shell
---driver-class-path spark/target/comet-spark-spark4.1_2.13-0.16.0-SNAPSHOT.jar
+--driver-class-path spark/target/comet-spark-spark4.1_2.13-0.17.0-SNAPSHOT.jar
```
Some cluster managers may require additional configuration, see
<https://spark.apache.org/docs/latest/cluster-overview.html>
diff --git a/_sources/user-guide/latest/kubernetes.md.txt
b/_sources/user-guide/latest/kubernetes.md.txt
index a58291ed0..65eb1527b 100644
--- a/_sources/user-guide/latest/kubernetes.md.txt
+++ b/_sources/user-guide/latest/kubernetes.md.txt
@@ -69,13 +69,13 @@ metadata:
spec:
type: Scala
mode: cluster
- image: apache/datafusion-comet:0.16.0-SNAPSHOT-spark3.5.5-scala2.12-java11
+ image: apache/datafusion-comet:0.17.0-SNAPSHOT-spark3.5.5-scala2.12-java11
imagePullPolicy: IfNotPresent
mainClass: org.apache.spark.examples.SparkPi
mainApplicationFile:
local:///opt/spark/examples/jars/spark-examples_2.13-4.1.1.jar
sparkConf:
- "spark.executor.extraClassPath":
"/opt/spark/jars/comet-spark-spark3.5_2.12-0.16.0-SNAPSHOT.jar"
- "spark.driver.extraClassPath":
"/opt/spark/jars/comet-spark-spark3.5_2.12-0.16.0-SNAPSHOT.jar"
+ "spark.executor.extraClassPath":
"/opt/spark/jars/comet-spark-spark3.5_2.12-0.17.0-SNAPSHOT.jar"
+ "spark.driver.extraClassPath":
"/opt/spark/jars/comet-spark-spark3.5_2.12-0.17.0-SNAPSHOT.jar"
"spark.plugins": "org.apache.spark.CometPlugin"
"spark.comet.enabled": "true"
"spark.comet.exec.enabled": "true"
diff --git a/_static/images/benchmark-results/0.16.0/tpcds_allqueries.png
b/_static/images/benchmark-results/0.16.0/tpcds_allqueries.png
new file mode 100644
index 000000000..211309a76
Binary files /dev/null and
b/_static/images/benchmark-results/0.16.0/tpcds_allqueries.png differ
diff --git a/_static/images/benchmark-results/0.16.0/tpcds_queries_compare.png
b/_static/images/benchmark-results/0.16.0/tpcds_queries_compare.png
new file mode 100644
index 000000000..22820b81e
Binary files /dev/null and
b/_static/images/benchmark-results/0.16.0/tpcds_queries_compare.png differ
diff --git
a/_static/images/benchmark-results/0.16.0/tpcds_queries_speedup_abs.png
b/_static/images/benchmark-results/0.16.0/tpcds_queries_speedup_abs.png
new file mode 100644
index 000000000..6e49ae4a2
Binary files /dev/null and
b/_static/images/benchmark-results/0.16.0/tpcds_queries_speedup_abs.png differ
diff --git
a/_static/images/benchmark-results/0.16.0/tpcds_queries_speedup_rel.png
b/_static/images/benchmark-results/0.16.0/tpcds_queries_speedup_rel.png
new file mode 100644
index 000000000..4f25fa863
Binary files /dev/null and
b/_static/images/benchmark-results/0.16.0/tpcds_queries_speedup_rel.png differ
diff --git a/_static/images/benchmark-results/0.16.0/tpch_allqueries.png
b/_static/images/benchmark-results/0.16.0/tpch_allqueries.png
new file mode 100644
index 000000000..ff4d79b6a
Binary files /dev/null and
b/_static/images/benchmark-results/0.16.0/tpch_allqueries.png differ
diff --git a/_static/images/benchmark-results/0.16.0/tpch_queries_compare.png
b/_static/images/benchmark-results/0.16.0/tpch_queries_compare.png
new file mode 100644
index 000000000..e86d43ab3
Binary files /dev/null and
b/_static/images/benchmark-results/0.16.0/tpch_queries_compare.png differ
diff --git
a/_static/images/benchmark-results/0.16.0/tpch_queries_speedup_abs.png
b/_static/images/benchmark-results/0.16.0/tpch_queries_speedup_abs.png
new file mode 100644
index 000000000..016e7226e
Binary files /dev/null and
b/_static/images/benchmark-results/0.16.0/tpch_queries_speedup_abs.png differ
diff --git
a/_static/images/benchmark-results/0.16.0/tpch_queries_speedup_rel.png
b/_static/images/benchmark-results/0.16.0/tpch_queries_speedup_rel.png
new file mode 100644
index 000000000..bf4ba2f3b
Binary files /dev/null and
b/_static/images/benchmark-results/0.16.0/tpch_queries_speedup_rel.png differ
diff --git a/contributor-guide/benchmark-results/tpc-ds.html
b/contributor-guide/benchmark-results/tpc-ds.html
index 7543bb9fd..949483baa 100644
--- a/contributor-guide/benchmark-results/tpc-ds.html
+++ b/contributor-guide/benchmark-results/tpc-ds.html
@@ -471,7 +471,17 @@ under the License.
<section id="apache-datafusion-comet-benchmarks-derived-from-tpc-ds">
<h1>Apache DataFusion Comet: Benchmarks Derived From TPC-DS<a
class="headerlink"
href="#apache-datafusion-comet-benchmarks-derived-from-tpc-ds" title="Link to
this heading">#</a></h1>
<p>The following benchmarks were performed on an EKS cluster (<code
class="docutils literal notranslate"><span
class="pre">r6i.24xlarge</span></code> instances with EBS storage) with data
stored in S3.</p>
-<p>The tracking issue for improving TPC-DS performance is <a class="reference
external"
href="https://github.com/apache/datafusion-comet/issues/858";>#858</a>.</p>
+<section id="benchmark-results">
+<h2>Benchmark Results<a class="headerlink" href="#benchmark-results"
title="Link to this heading">#</a></h2>
+<p>Total time to run all queries (lower is better).</p>
+<p><img alt="" src="../../_images/tpcds_allqueries.png" /></p>
+<p>Per-query breakdown showing the relative performance of Spark and Comet.</p>
+<p><img alt="" src="../../_images/tpcds_queries_compare.png" /></p>
+<p>How much Comet accelerates each query in relative terms.</p>
+<p><img alt="" src="../../_images/tpcds_queries_speedup_rel.png" /></p>
+<p>How much Comet accelerates each query in absolute terms.</p>
+<p><img alt="" src="../../_images/tpcds_queries_speedup_abs.png" /></p>
+</section>
<section id="configuration">
<h2>Configuration<a class="headerlink" href="#configuration" title="Link to
this heading">#</a></h2>
<p>Common:</p>
@@ -498,16 +508,6 @@ under the License.
</pre></div>
</div>
</section>
-<section id="benchmark-results">
-<h2>Benchmark Results<a class="headerlink" href="#benchmark-results"
title="Link to this heading">#</a></h2>
-<p><img alt="" src="../../_images/tpcds_allqueries.png" /></p>
-<p>Here is a breakdown showing relative performance of Spark and Comet for
each query.</p>
-<p><img alt="" src="../../_images/tpcds_queries_compare.png" /></p>
-<p>The following chart shows how much Comet currently accelerates each query
from the benchmark in relative terms.</p>
-<p><img alt="" src="../../_images/tpcds_queries_speedup_rel.png" /></p>
-<p>The following chart shows how much Comet currently accelerates each query
from the benchmark in absolute terms.</p>
-<p><img alt="" src="../../_images/tpcds_queries_speedup_abs.png" /></p>
-</section>
</section>
diff --git a/contributor-guide/benchmark-results/tpc-h.html
b/contributor-guide/benchmark-results/tpc-h.html
index 754b49118..d55c993c8 100644
--- a/contributor-guide/benchmark-results/tpc-h.html
+++ b/contributor-guide/benchmark-results/tpc-h.html
@@ -471,6 +471,18 @@ under the License.
<section id="apache-datafusion-comet-benchmarks-derived-from-tpc-h">
<h1>Apache DataFusion Comet: Benchmarks Derived From TPC-H<a
class="headerlink"
href="#apache-datafusion-comet-benchmarks-derived-from-tpc-h" title="Link to
this heading">#</a></h1>
<p>The following benchmarks were performed on an EKS cluster (<code
class="docutils literal notranslate"><span
class="pre">r6i.24xlarge</span></code> instances with EBS storage) with data
stored in S3.</p>
+<section id="benchmark-results">
+<h2>Benchmark Results<a class="headerlink" href="#benchmark-results"
title="Link to this heading">#</a></h2>
+<p>Total time to run all queries (lower is better).</p>
+<p><img alt="" src="../../_images/tpch_allqueries_with_tuned.png" /></p>
+<p>The following charts are based on the tuned run using hash join.</p>
+<p>Per-query breakdown showing the relative performance of Spark and Comet.</p>
+<p><img alt="" src="../../_images/tpch_queries_compare.png" /></p>
+<p>How much Comet accelerates each query in relative terms.</p>
+<p><img alt="" src="../../_images/tpch_queries_speedup_rel.png" /></p>
+<p>How much Comet accelerates each query in absolute terms.</p>
+<p><img alt="" src="../../_images/tpch_queries_speedup_abs.png" /></p>
+</section>
<section id="configuration">
<h2>Configuration<a class="headerlink" href="#configuration" title="Link to
this heading">#</a></h2>
<p>Common:</p>
@@ -495,30 +507,13 @@ under the License.
<span class="na">spark.memory.offHeap.size</span><span class="o">=</span><span
class="s">32G</span>
</pre></div>
</div>
-<p>Comet (Tuned):</p>
-<div class="highlight-properties notranslate"><div
class="highlight"><pre><span></span><span
class="na">spark.executor.memory</span><span class="o">=</span><span
class="s">32G</span>
-<span class="na">spark.executor.memoryOverhead</span><span
class="o">=</span><span class="s">10G</span>
-<span class="na">spark.memory.offHeap.enabled</span><span
class="o">=</span><span class="s">true</span>
-<span class="na">spark.memory.offHeap.size</span><span class="o">=</span><span
class="s">32G</span>
-<span class="na">spark.comet.exec.replaceSortMergeJoin</span><span
class="o">=</span><span class="s">true</span>
+<section id="comet-tuned">
+<h3>Comet (Tuned)<a class="headerlink" href="#comet-tuned" title="Link to this
heading">#</a></h3>
+<div class="highlight-properties notranslate"><div
class="highlight"><pre><span></span><span
class="na">spark.comet.exec.replaceSortMergeJoin</span><span
class="o">=</span><span class="s">true</span>
<span class="na">spark.comet.memoryPool.fraction</span><span
class="o">=</span><span class="s">0.8</span>
</pre></div>
</div>
</section>
-<section id="benchmark-results">
-<h2>Benchmark Results<a class="headerlink" href="#benchmark-results"
title="Link to this heading">#</a></h2>
-<p>The following chart shows benchmark results comparing Spark to Comet, both
with Comet’s default settings, and with Hash Join enabled in Comet.</p>
-<p>Comet’s Hash Join does not support spilling yet, so it isn’t suitable for
all workloads.</p>
-<p><img alt="" src="../../_images/tpch_allqueries_with_tuned.png" /></p>
-</section>
-<section id="comet-with-hash-join-enabled">
-<h2>Comet (with Hash Join enabled)<a class="headerlink"
href="#comet-with-hash-join-enabled" title="Link to this heading">#</a></h2>
-<p>Here is a breakdown showing relative performance of Spark and Comet for
each query.</p>
-<p><img alt="" src="../../_images/tpch_queries_compare.png" /></p>
-<p>The following chart shows how much Comet currently accelerates each query
from the benchmark in relative terms.</p>
-<p><img alt="" src="../../_images/tpch_queries_speedup_rel.png" /></p>
-<p>The following chart shows how much Comet currently accelerates each query
from the benchmark in absolute terms.</p>
-<p><img alt="" src="../../_images/tpch_queries_speedup_abs.png" /></p>
</section>
</section>
diff --git a/index.html b/index.html
index 74d58217c..4a2bd7ad9 100644
--- a/index.html
+++ b/index.html
@@ -429,7 +429,7 @@ Spark ecosystem without requiring any code changes.</p>
for details.</p>
<p><img alt="" src="_images/tpch_allqueries.png" /></p>
<p>Here is a breakdown showing relative performance of Spark and Comet for
each TPC-H query.</p>
-<p><img alt="" src="_images/tpch_queries_compare.png" /></p>
+<p><img alt="" src="_images/tpch_queries_compare1.png" /></p>
</section>
<section id="use-commodity-hardware">
<h2>Use Commodity Hardware<a class="headerlink" href="#use-commodity-hardware"
title="Link to this heading">#</a></h2>
diff --git a/objects.inv b/objects.inv
index 3e3926991..464f502ef 100644
Binary files a/objects.inv and b/objects.inv differ
diff --git a/searchindex.js b/searchindex.js
index 692888929..5cda76cd9 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"1. Format Your Code": [[13,
"format-your-code"]], "1. Install Comet": [[15, "install-comet"], [23,
"install-comet"]], "1. Native Operators (nativeExecs map)": [[3,
"native-operators-nativeexecs-map"]], "2. Build and Verify": [[13,
"build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[15,
"clone-iceberg-and-apply-diff"]], "2. Clone Spark and Apply Diff": [[23,
"clone-spark-and-apply-diff"]], "2. Sink Operators (sinks map)": [[3,
"sink-operators-sinks-m [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"1. Format Your Code": [[13,
"format-your-code"]], "1. Install Comet": [[15, "install-comet"], [23,
"install-comet"]], "1. Native Operators (nativeExecs map)": [[3,
"native-operators-nativeexecs-map"]], "2. Build and Verify": [[13,
"build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[15,
"clone-iceberg-and-apply-diff"]], "2. Clone Spark and Apply Diff": [[23,
"clone-spark-and-apply-diff"]], "2. Sink Operators (sinks map)": [[3,
"sink-operators-sinks-m [...]
\ No newline at end of file
diff --git a/user-guide/index.html b/user-guide/index.html
index ad40c90c0..d537ae034 100644
--- a/user-guide/index.html
+++ b/user-guide/index.html
@@ -65,7 +65,7 @@ under the License.
<script async="true" defer="true"
src="https://buttons.github.io/buttons.js";></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
- <link rel="next" title="Comet 0.16.0-SNAPSHOT User Guide"
href="latest/index.html" />
+ <link rel="next" title="Comet 0.17.0-SNAPSHOT User Guide"
href="latest/index.html" />
<link rel="prev" title="Comet Accelerator for Apache Spark and Apache
Iceberg" href="../index.html" />
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<meta name="docsearch:language" content="en"/>
@@ -473,7 +473,7 @@ navigation menu to view its guide.</p>
title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
- <p class="prev-next-title">Comet 0.16.0-SNAPSHOT User Guide</p>
+ <p class="prev-next-title">Comet 0.17.0-SNAPSHOT User Guide</p>
</div>
<i class="fa-solid fa-angle-right"></i>
</a>
diff --git a/user-guide/latest/compatibility/expressions/aggregate.html
b/user-guide/latest/compatibility/expressions/aggregate.html
index 9079e863a..414888b76 100644
--- a/user-guide/latest/compatibility/expressions/aggregate.html
+++ b/user-guide/latest/compatibility/expressions/aggregate.html
@@ -430,7 +430,7 @@ under the License.
<li class="breadcrumb-item"><a href="../../../index.html"
class="nav-link">Comet User Guide</a></li>
- <li class="breadcrumb-item"><a href="../../index.html"
class="nav-link">Comet 0.16.0-SNAPSHOT User Guide</a></li>
+ <li class="breadcrumb-item"><a href="../../index.html"
class="nav-link">Comet 0.17.0-SNAPSHOT User Guide</a></li>
<li class="breadcrumb-item"><a href="../index.html"
class="nav-link">Compatibility Guide</a></li>
diff --git a/user-guide/latest/compatibility/expressions/array.html
b/user-guide/latest/compatibility/expressions/array.html
index 88b6c3a2b..c90989b5e 100644
--- a/user-guide/latest/compatibility/expressions/array.html
+++ b/user-guide/latest/compatibility/expressions/array.html
@@ -430,7 +430,7 @@ under the License.
<li class="breadcrumb-item"><a href="../../../index.html"
class="nav-link">Comet User Guide</a></li>
- <li class="breadcrumb-item"><a href="../../index.html"
class="nav-link">Comet 0.16.0-SNAPSHOT User Guide</a></li>
+ <li class="breadcrumb-item"><a href="../../index.html"
class="nav-link">Comet 0.17.0-SNAPSHOT User Guide</a></li>
<li class="breadcrumb-item"><a href="../index.html"
class="nav-link">Compatibility Guide</a></li>
diff --git a/user-guide/latest/compatibility/expressions/cast.html
b/user-guide/latest/compatibility/expressions/cast.html
index 2d1ac2f85..a7224ed1b 100644
--- a/user-guide/latest/compatibility/expressions/cast.html
+++ b/user-guide/latest/compatibility/expressions/cast.html
@@ -430,7 +430,7 @@ under the License.
<li class="breadcrumb-item"><a href="../../../index.html"
class="nav-link">Comet User Guide</a></li>
- <li class="breadcrumb-item"><a href="../../index.html"
class="nav-link">Comet 0.16.0-SNAPSHOT User Guide</a></li>
+ <li class="breadcrumb-item"><a href="../../index.html"
class="nav-link">Comet 0.17.0-SNAPSHOT User Guide</a></li>
<li class="breadcrumb-item"><a href="../index.html"
class="nav-link">Compatibility Guide</a></li>
diff --git a/user-guide/latest/compatibility/expressions/datetime.html
b/user-guide/latest/compatibility/expressions/datetime.html
index 8302bf83d..de6ce7395 100644
--- a/user-guide/latest/compatibility/expressions/datetime.html
+++ b/user-guide/latest/compatibility/expressions/datetime.html
@@ -430,7 +430,7 @@ under the License.
<li class="breadcrumb-item"><a href="../../../index.html"
class="nav-link">Comet User Guide</a></li>
- <li class="breadcrumb-item"><a href="../../index.html"
class="nav-link">Comet 0.16.0-SNAPSHOT User Guide</a></li>
+ <li class="breadcrumb-item"><a href="../../index.html"
class="nav-link">Comet 0.17.0-SNAPSHOT User Guide</a></li>
<li class="breadcrumb-item"><a href="../index.html"
class="nav-link">Compatibility Guide</a></li>
diff --git a/user-guide/latest/compatibility/expressions/index.html
b/user-guide/latest/compatibility/expressions/index.html
index fa27724f5..163f3ff2c 100644
--- a/user-guide/latest/compatibility/expressions/index.html
+++ b/user-guide/latest/compatibility/expressions/index.html
@@ -430,7 +430,7 @@ under the License.
<li class="breadcrumb-item"><a href="../../../index.html"
class="nav-link">Comet User Guide</a></li>
- <li class="breadcrumb-item"><a href="../../index.html"
class="nav-link">Comet 0.16.0-SNAPSHOT User Guide</a></li>
+ <li class="breadcrumb-item"><a href="../../index.html"
class="nav-link">Comet 0.17.0-SNAPSHOT User Guide</a></li>
<li class="breadcrumb-item"><a href="../index.html"
class="nav-link">Compatibility Guide</a></li>
diff --git a/user-guide/latest/compatibility/expressions/map.html
b/user-guide/latest/compatibility/expressions/map.html
index f66e62fd1..6903c948a 100644
--- a/user-guide/latest/compatibility/expressions/map.html
+++ b/user-guide/latest/compatibility/expressions/map.html
@@ -430,7 +430,7 @@ under the License.
<li class="breadcrumb-item"><a href="../../../index.html"
class="nav-link">Comet User Guide</a></li>
- <li class="breadcrumb-item"><a href="../../index.html"
class="nav-link">Comet 0.16.0-SNAPSHOT User Guide</a></li>
+ <li class="breadcrumb-item"><a href="../../index.html"
class="nav-link">Comet 0.17.0-SNAPSHOT User Guide</a></li>
<li class="breadcrumb-item"><a href="../index.html"
class="nav-link">Compatibility Guide</a></li>
diff --git a/user-guide/latest/compatibility/expressions/math.html
b/user-guide/latest/compatibility/expressions/math.html
index fd25c5a7b..059088b20 100644
--- a/user-guide/latest/compatibility/expressions/math.html
+++ b/user-guide/latest/compatibility/expressions/math.html
@@ -430,7 +430,7 @@ under the License.
<li class="breadcrumb-item"><a href="../../../index.html"
class="nav-link">Comet User Guide</a></li>
- <li class="breadcrumb-item"><a href="../../index.html"
class="nav-link">Comet 0.16.0-SNAPSHOT User Guide</a></li>
+ <li class="breadcrumb-item"><a href="../../index.html"
class="nav-link">Comet 0.17.0-SNAPSHOT User Guide</a></li>
<li class="breadcrumb-item"><a href="../index.html"
class="nav-link">Compatibility Guide</a></li>
diff --git a/user-guide/latest/compatibility/expressions/misc.html
b/user-guide/latest/compatibility/expressions/misc.html
index 80bba1941..86fab8ed4 100644
--- a/user-guide/latest/compatibility/expressions/misc.html
+++ b/user-guide/latest/compatibility/expressions/misc.html
@@ -430,7 +430,7 @@ under the License.
<li class="breadcrumb-item"><a href="../../../index.html"
class="nav-link">Comet User Guide</a></li>
- <li class="breadcrumb-item"><a href="../../index.html"
class="nav-link">Comet 0.16.0-SNAPSHOT User Guide</a></li>
+ <li class="breadcrumb-item"><a href="../../index.html"
class="nav-link">Comet 0.17.0-SNAPSHOT User Guide</a></li>
<li class="breadcrumb-item"><a href="../index.html"
class="nav-link">Compatibility Guide</a></li>
diff --git a/user-guide/latest/compatibility/expressions/string.html
b/user-guide/latest/compatibility/expressions/string.html
index 7fc73b716..308d67c45 100644
--- a/user-guide/latest/compatibility/expressions/string.html
+++ b/user-guide/latest/compatibility/expressions/string.html
@@ -430,7 +430,7 @@ under the License.
<li class="breadcrumb-item"><a href="../../../index.html"
class="nav-link">Comet User Guide</a></li>
- <li class="breadcrumb-item"><a href="../../index.html"
class="nav-link">Comet 0.16.0-SNAPSHOT User Guide</a></li>
+ <li class="breadcrumb-item"><a href="../../index.html"
class="nav-link">Comet 0.17.0-SNAPSHOT User Guide</a></li>
<li class="breadcrumb-item"><a href="../index.html"
class="nav-link">Compatibility Guide</a></li>
diff --git a/user-guide/latest/compatibility/expressions/struct.html
b/user-guide/latest/compatibility/expressions/struct.html
index 114f81a1a..c413c21bc 100644
--- a/user-guide/latest/compatibility/expressions/struct.html
+++ b/user-guide/latest/compatibility/expressions/struct.html
@@ -430,7 +430,7 @@ under the License.
<li class="breadcrumb-item"><a href="../../../index.html"
class="nav-link">Comet User Guide</a></li>
- <li class="breadcrumb-item"><a href="../../index.html"
class="nav-link">Comet 0.16.0-SNAPSHOT User Guide</a></li>
+ <li class="breadcrumb-item"><a href="../../index.html"
class="nav-link">Comet 0.17.0-SNAPSHOT User Guide</a></li>
<li class="breadcrumb-item"><a href="../index.html"
class="nav-link">Compatibility Guide</a></li>
diff --git a/user-guide/latest/compatibility/floating-point.html
b/user-guide/latest/compatibility/floating-point.html
index 44c6c4ee9..e8ac15821 100644
--- a/user-guide/latest/compatibility/floating-point.html
+++ b/user-guide/latest/compatibility/floating-point.html
@@ -430,7 +430,7 @@ under the License.
<li class="breadcrumb-item"><a href="../../index.html"
class="nav-link">Comet User Guide</a></li>
- <li class="breadcrumb-item"><a href="../index.html" class="nav-link">Comet
0.16.0-SNAPSHOT User Guide</a></li>
+ <li class="breadcrumb-item"><a href="../index.html" class="nav-link">Comet
0.17.0-SNAPSHOT User Guide</a></li>
<li class="breadcrumb-item"><a href="index.html"
class="nav-link">Compatibility Guide</a></li>
diff --git a/user-guide/latest/compatibility/index.html
b/user-guide/latest/compatibility/index.html
index 5ca9a5e34..c68fbedaa 100644
--- a/user-guide/latest/compatibility/index.html
+++ b/user-guide/latest/compatibility/index.html
@@ -430,7 +430,7 @@ under the License.
<li class="breadcrumb-item"><a href="../../index.html"
class="nav-link">Comet User Guide</a></li>
- <li class="breadcrumb-item"><a href="../index.html" class="nav-link">Comet
0.16.0-SNAPSHOT User Guide</a></li>
+ <li class="breadcrumb-item"><a href="../index.html" class="nav-link">Comet
0.17.0-SNAPSHOT User Guide</a></li>
<li class="breadcrumb-item active" aria-current="page"><span
class="ellipsis">Compatibility Guide</span></li>
</ul>
diff --git a/user-guide/latest/compatibility/operators.html
b/user-guide/latest/compatibility/operators.html
index 46868c987..b6f4c5d4a 100644
--- a/user-guide/latest/compatibility/operators.html
+++ b/user-guide/latest/compatibility/operators.html
@@ -430,7 +430,7 @@ under the License.
<li class="breadcrumb-item"><a href="../../index.html"
class="nav-link">Comet User Guide</a></li>
- <li class="breadcrumb-item"><a href="../index.html" class="nav-link">Comet
0.16.0-SNAPSHOT User Guide</a></li>
+ <li class="breadcrumb-item"><a href="../index.html" class="nav-link">Comet
0.17.0-SNAPSHOT User Guide</a></li>
<li class="breadcrumb-item"><a href="index.html"
class="nav-link">Compatibility Guide</a></li>
diff --git a/user-guide/latest/compatibility/regex.html
b/user-guide/latest/compatibility/regex.html
index 8ffd39697..12ce28baf 100644
--- a/user-guide/latest/compatibility/regex.html
+++ b/user-guide/latest/compatibility/regex.html
@@ -430,7 +430,7 @@ under the License.
<li class="breadcrumb-item"><a href="../../index.html"
class="nav-link">Comet User Guide</a></li>
- <li class="breadcrumb-item"><a href="../index.html" class="nav-link">Comet
0.16.0-SNAPSHOT User Guide</a></li>
+ <li class="breadcrumb-item"><a href="../index.html" class="nav-link">Comet
0.17.0-SNAPSHOT User Guide</a></li>
<li class="breadcrumb-item"><a href="index.html"
class="nav-link">Compatibility Guide</a></li>
diff --git a/user-guide/latest/compatibility/scans.html
b/user-guide/latest/compatibility/scans.html
index 9027924d7..e9b815245 100644
--- a/user-guide/latest/compatibility/scans.html
+++ b/user-guide/latest/compatibility/scans.html
@@ -430,7 +430,7 @@ under the License.
<li class="breadcrumb-item"><a href="../../index.html"
class="nav-link">Comet User Guide</a></li>
- <li class="breadcrumb-item"><a href="../index.html" class="nav-link">Comet
0.16.0-SNAPSHOT User Guide</a></li>
+ <li class="breadcrumb-item"><a href="../index.html" class="nav-link">Comet
0.17.0-SNAPSHOT User Guide</a></li>
<li class="breadcrumb-item"><a href="index.html"
class="nav-link">Compatibility Guide</a></li>
diff --git a/user-guide/latest/compatibility/spark-versions.html
b/user-guide/latest/compatibility/spark-versions.html
index 31cd064f9..3442aa7bc 100644
--- a/user-guide/latest/compatibility/spark-versions.html
+++ b/user-guide/latest/compatibility/spark-versions.html
@@ -430,7 +430,7 @@ under the License.
<li class="breadcrumb-item"><a href="../../index.html"
class="nav-link">Comet User Guide</a></li>
- <li class="breadcrumb-item"><a href="../index.html" class="nav-link">Comet
0.16.0-SNAPSHOT User Guide</a></li>
+ <li class="breadcrumb-item"><a href="../index.html" class="nav-link">Comet
0.17.0-SNAPSHOT User Guide</a></li>
<li class="breadcrumb-item"><a href="index.html"
class="nav-link">Compatibility Guide</a></li>
diff --git a/user-guide/latest/configs.html b/user-guide/latest/configs.html
index c907b54eb..95f5b8ced 100644
--- a/user-guide/latest/configs.html
+++ b/user-guide/latest/configs.html
@@ -422,7 +422,7 @@ under the License.
<li class="breadcrumb-item"><a href="../index.html" class="nav-link">Comet
User Guide</a></li>
- <li class="breadcrumb-item"><a href="index.html" class="nav-link">Comet
0.16.0-SNAPSHOT User Guide</a></li>
+ <li class="breadcrumb-item"><a href="index.html" class="nav-link">Comet
0.17.0-SNAPSHOT User Guide</a></li>
<li class="breadcrumb-item active" aria-current="page"><span
class="ellipsis">Comet Configuration Settings</span></li>
</ul>
diff --git a/user-guide/latest/datasources.html
b/user-guide/latest/datasources.html
index 50d49b17d..5349de822 100644
--- a/user-guide/latest/datasources.html
+++ b/user-guide/latest/datasources.html
@@ -422,7 +422,7 @@ under the License.
<li class="breadcrumb-item"><a href="../index.html" class="nav-link">Comet
User Guide</a></li>
- <li class="breadcrumb-item"><a href="index.html" class="nav-link">Comet
0.16.0-SNAPSHOT User Guide</a></li>
+ <li class="breadcrumb-item"><a href="index.html" class="nav-link">Comet
0.17.0-SNAPSHOT User Guide</a></li>
<li class="breadcrumb-item active" aria-current="page"><span
class="ellipsis">Supported Spark Data Sources</span></li>
</ul>
@@ -531,7 +531,7 @@ root
<span class="w"> </span><span class="p">|</span><span class="w">
</span><span class="p">|</span>--<span class="w"> </span>lastName:<span
class="w"> </span>string<span class="w"> </span><span class="o">(</span><span
class="nv">nullable</span><span class="w"> </span><span class="o">=</span><span
class="w"> </span><span class="nb">true</span><span class="o">)</span>
<span class="w"> </span><span class="p">|</span><span class="w">
</span><span class="p">|</span>--<span class="w"> </span>ageInYears:<span
class="w"> </span>integer<span class="w"> </span><span class="o">(</span><span
class="nv">nullable</span><span class="w"> </span><span class="o">=</span><span
class="w"> </span><span class="nb">true</span><span class="o">)</span>
-<span class="m">25</span>/01/30<span class="w"> </span><span
class="m">16</span>:50:43<span class="w"> </span>INFO<span class="w">
</span>core/src/lib.rs:<span class="w"> </span>Comet<span class="w">
</span>native<span class="w"> </span>library<span class="w">
</span>version<span class="w"> </span><span
class="m">0</span>.16.0-SNAPSHOT<span class="w"> </span><span
class="nv">initialized</span>
+<span class="m">25</span>/01/30<span class="w"> </span><span
class="m">16</span>:50:43<span class="w"> </span>INFO<span class="w">
</span>core/src/lib.rs:<span class="w"> </span>Comet<span class="w">
</span>native<span class="w"> </span>library<span class="w">
</span>version<span class="w"> </span><span
class="m">0</span>.17.0-SNAPSHOT<span class="w"> </span><span
class="nv">initialized</span>
<span class="o">==</span><span class="w"> </span>Physical<span class="w">
</span><span class="nv">Plan</span><span class="w"> </span><span
class="o">==</span>
*<span class="w"> </span>CometColumnarToRow<span class="w"> </span><span
class="o">(</span><span class="m">2</span><span class="o">)</span>
+-<span class="w"> </span>CometNativeScan:<span class="w"> </span><span
class="o">(</span><span class="m">1</span><span class="o">)</span>
diff --git a/user-guide/latest/datatypes.html b/user-guide/latest/datatypes.html
index a75660ee4..7dde75951 100644
--- a/user-guide/latest/datatypes.html
+++ b/user-guide/latest/datatypes.html
@@ -422,7 +422,7 @@ under the License.
<li class="breadcrumb-item"><a href="../index.html" class="nav-link">Comet
User Guide</a></li>
- <li class="breadcrumb-item"><a href="index.html" class="nav-link">Comet
0.16.0-SNAPSHOT User Guide</a></li>
+ <li class="breadcrumb-item"><a href="index.html" class="nav-link">Comet
0.17.0-SNAPSHOT User Guide</a></li>
<li class="breadcrumb-item active" aria-current="page"><span
class="ellipsis">Supported Spark Data Types</span></li>
</ul>
diff --git a/user-guide/latest/expressions.html
b/user-guide/latest/expressions.html
index ba0cd3625..fda9ed0c4 100644
--- a/user-guide/latest/expressions.html
+++ b/user-guide/latest/expressions.html
@@ -422,7 +422,7 @@ under the License.
<li class="breadcrumb-item"><a href="../index.html" class="nav-link">Comet
User Guide</a></li>
- <li class="breadcrumb-item"><a href="index.html" class="nav-link">Comet
0.16.0-SNAPSHOT User Guide</a></li>
+ <li class="breadcrumb-item"><a href="index.html" class="nav-link">Comet
0.17.0-SNAPSHOT User Guide</a></li>
<li class="breadcrumb-item active" aria-current="page"><span
class="ellipsis">Supported Spark Expressions</span></li>
</ul>
diff --git a/user-guide/latest/iceberg.html b/user-guide/latest/iceberg.html
index 2d228d5e6..21ceeda27 100644
--- a/user-guide/latest/iceberg.html
+++ b/user-guide/latest/iceberg.html
@@ -422,7 +422,7 @@ under the License.
<li class="breadcrumb-item"><a href="../index.html" class="nav-link">Comet
User Guide</a></li>
- <li class="breadcrumb-item"><a href="index.html" class="nav-link">Comet
0.16.0-SNAPSHOT User Guide</a></li>
+ <li class="breadcrumb-item"><a href="index.html" class="nav-link">Comet
0.17.0-SNAPSHOT User Guide</a></li>
<li class="breadcrumb-item active" aria-current="page"><span
class="ellipsis">Accelerating Apache Iceberg Parquet Scans using
Comet</span></li>
</ul>
@@ -466,14 +466,14 @@ under the License.
<p>Comet’s native Iceberg reader relies on reflection to extract <code
class="docutils literal notranslate"><span
class="pre">FileScanTask</span></code>s from Iceberg, which are
then serialized to Comet’s native execution engine (see
<a class="reference external"
href="https://github.com/apache/datafusion-comet/pull/2528";>PR #2528</a>).</p>
-<p>The example below uses Spark’s package downloader to retrieve Comet
0.16.0-SNAPSHOT and Iceberg
+<p>The example below uses Spark’s package downloader to retrieve Comet
0.17.0-SNAPSHOT and Iceberg
1.8.1, but Comet has been tested with Iceberg 1.5, 1.7, 1.8, 1.9, and 1.10.
The native Iceberg
reader is enabled by default. To disable it, set <code class="docutils literal
notranslate"><span
class="pre">spark.comet.scan.icebergNative.enabled=false</span></code>.</p>
<p>The example uses the Spark 3.5 / Scala 2.12 build of Comet; substitute the
Comet artifact
matching your Spark and Scala versions (Comet also ships Spark 3.5 / Scala
2.13 and Spark
4.0/4.1 / Scala 2.13 jars; see the <a class="reference internal"
href="installation.html"><span class="std std-doc">installation
guide</span></a> for the full list).</p>
<div class="highlight-shell notranslate"><div
class="highlight"><pre><span></span><span
class="nv">$SPARK_HOME</span>/bin/spark-shell<span class="w"> </span><span
class="se">\</span>
-<span class="w"> </span>--packages<span class="w">
</span>org.apache.datafusion:comet-spark-spark3.5_2.12:0.16.0-SNAPSHOT,org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.8.1,org.apache.iceberg:iceberg-core:1.8.1<span
class="w"> </span><span class="se">\</span>
+<span class="w"> </span>--packages<span class="w">
</span>org.apache.datafusion:comet-spark-spark3.5_2.12:0.17.0-SNAPSHOT,org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.8.1,org.apache.iceberg:iceberg-core:1.8.1<span
class="w"> </span><span class="se">\</span>
<span class="w"> </span>--repositories<span class="w">
</span>https://repo1.maven.org/maven2/<span class="w"> </span><span
class="se">\</span>
<span class="w"> </span>--conf<span class="w">
</span>spark.sql.extensions<span
class="o">=</span>org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions<span
class="w"> </span><span class="se">\</span>
<span class="w"> </span>--conf<span class="w">
</span>spark.sql.catalog.spark_catalog<span
class="o">=</span>org.apache.iceberg.spark.SparkCatalog<span class="w">
</span><span class="se">\</span>
@@ -545,7 +545,7 @@ values between 2 and 8 based on your workload.</p>
<p>Comet’s native Iceberg reader also supports REST catalogs. The following
example shows how to
configure Spark to use a REST catalog with Comet’s native Iceberg scan:</p>
<div class="highlight-shell notranslate"><div
class="highlight"><pre><span></span><span
class="nv">$SPARK_HOME</span>/bin/spark-shell<span class="w"> </span><span
class="se">\</span>
-<span class="w"> </span>--packages<span class="w">
</span>org.apache.datafusion:comet-spark-spark3.5_2.12:0.16.0-SNAPSHOT,org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.8.1,org.apache.iceberg:iceberg-core:1.8.1<span
class="w"> </span><span class="se">\</span>
+<span class="w"> </span>--packages<span class="w">
</span>org.apache.datafusion:comet-spark-spark3.5_2.12:0.17.0-SNAPSHOT,org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.8.1,org.apache.iceberg:iceberg-core:1.8.1<span
class="w"> </span><span class="se">\</span>
<span class="w"> </span>--repositories<span class="w">
</span>https://repo1.maven.org/maven2/<span class="w"> </span><span
class="se">\</span>
<span class="w"> </span>--conf<span class="w">
</span>spark.sql.extensions<span
class="o">=</span>org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions<span
class="w"> </span><span class="se">\</span>
<span class="w"> </span>--conf<span class="w">
</span>spark.sql.catalog.rest_cat<span
class="o">=</span>org.apache.iceberg.spark.SparkCatalog<span class="w">
</span><span class="se">\</span>
diff --git a/user-guide/latest/index.html b/user-guide/latest/index.html
index 2547bad9a..9f6b03405 100644
--- a/user-guide/latest/index.html
+++ b/user-guide/latest/index.html
@@ -27,7 +27,7 @@ under the License.
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0"
/><meta name="viewport" content="width=device-width, initial-scale=1" />
- <title>Comet 0.16.0-SNAPSHOT User Guide — Apache DataFusion Comet
documentation</title>
+ <title>Comet 0.17.0-SNAPSHOT User Guide — Apache DataFusion Comet
documentation</title>
@@ -421,7 +421,7 @@ under the License.
<li class="breadcrumb-item"><a href="../index.html" class="nav-link">Comet
User Guide</a></li>
- <li class="breadcrumb-item active" aria-current="page"><span
class="ellipsis">Comet 0.16.0-SNAPSHOT User Guide</span></li>
+ <li class="breadcrumb-item active" aria-current="page"><span
class="ellipsis">Comet 0.17.0-SNAPSHOT User Guide</span></li>
</ul>
</nav>
</div>
@@ -439,16 +439,16 @@ under the License.
<article class="bd-article">
<img alt="DataFusion Comet Logo"
src="../../_images/DataFusionComet-Logo-Light.png" />
-<section id="comet-0-16-0-snapshot-user-guide">
-<h1>Comet 0.16.0-SNAPSHOT User Guide<a class="headerlink"
href="#comet-0-16-0-snapshot-user-guide" title="Link to this heading">#</a></h1>
-<p>This guide covers Comet 0.16.0-SNAPSHOT: how to install it, build it from
source, configure it for
+<section id="comet-0-17-0-snapshot-user-guide">
+<h1>Comet 0.17.0-SNAPSHOT User Guide<a class="headerlink"
href="#comet-0-17-0-snapshot-user-guide" title="Link to this heading">#</a></h1>
+<p>This guide covers Comet 0.17.0-SNAPSHOT: how to install it, build it from
source, configure it for
your Spark deployment, and get the best results from it. It also documents the
data sources, data
types, operators, and expressions that Comet supports, along with a
compatibility guide describing
known differences from Apache Spark.</p>
<p>Operational topics include reading and understanding Comet query plans,
tuning, available metrics,
and integration guides for Apache Iceberg and Kubernetes. Select a topic from
the navigation menu
to read more.</p>
-<div class="toctree-wrapper compound"
id="toc-user-guide-links-0-16-0-snapshot">
+<div class="toctree-wrapper compound"
id="toc-user-guide-links-0-17-0-snapshot">
</div>
</section>
diff --git a/user-guide/latest/installation.html
b/user-guide/latest/installation.html
index adb445b27..f4c36d0df 100644
--- a/user-guide/latest/installation.html
+++ b/user-guide/latest/installation.html
@@ -66,7 +66,7 @@ under the License.
<link rel="index" title="Index" href="../../genindex.html" />
<link rel="search" title="Search" href="../../search.html" />
<link rel="next" title="Building Comet From Source" href="source.html" />
- <link rel="prev" title="Comet 0.16.0-SNAPSHOT User Guide"
href="index.html" />
+ <link rel="prev" title="Comet 0.17.0-SNAPSHOT User Guide"
href="index.html" />
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<meta name="docsearch:language" content="en"/>
<meta name="docsearch:version" content="" />
@@ -422,7 +422,7 @@ under the License.
<li class="breadcrumb-item"><a href="../index.html" class="nav-link">Comet
User Guide</a></li>
- <li class="breadcrumb-item"><a href="index.html" class="nav-link">Comet
0.16.0-SNAPSHOT User Guide</a></li>
+ <li class="breadcrumb-item"><a href="index.html" class="nav-link">Comet
0.17.0-SNAPSHOT User Guide</a></li>
<li class="breadcrumb-item active" aria-current="page"><span
class="ellipsis">Installing DataFusion Comet</span></li>
</ul>
@@ -494,7 +494,7 @@ users must <a class="reference internal"
href="source.html"><span class="std std
</section>
<section id="supported-spark-versions">
<h3>Supported Spark Versions<a class="headerlink"
href="#supported-spark-versions" title="Link to this heading">#</a></h3>
-<p>Comet 0.16.0-SNAPSHOT supports the following versions of Apache Spark.
Refer to the <a class="reference internal"
href="compatibility/spark-versions.html"><span class="std std-doc">Spark
Version Compatibility</span></a> page
+<p>Comet 0.17.0-SNAPSHOT supports the following versions of Apache Spark.
Refer to the <a class="reference internal"
href="compatibility/spark-versions.html"><span class="std std-doc">Spark
Version Compatibility</span></a> page
in the <a class="reference internal" href="compatibility/index.html"><span
class="std std-doc">Compatibility Guide</span></a> for more information, such
as known limitations per Spark version.</p>
<p>We recommend only using Comet with Spark versions where we currently have
both Comet and Spark tests enabled in CI.
Other versions may work well enough for development and evaluation
purposes.</p>
@@ -578,7 +578,7 @@ source releases, or from the latest code in the GitHub
repository.</p>
<section id="run-spark-shell-with-comet-enabled">
<h2>Run Spark Shell with Comet enabled<a class="headerlink"
href="#run-spark-shell-with-comet-enabled" title="Link to this
heading">#</a></h2>
<p>Make sure <code class="docutils literal notranslate"><span
class="pre">SPARK_HOME</span></code> points to the same Spark version as Comet
was built for.</p>
-<div class="highlight-shell notranslate"><div
class="highlight"><pre><span></span><span class="nb">export</span><span
class="w"> </span><span class="nv">COMET_JAR</span><span
class="o">=</span>spark/target/comet-spark-spark4.1_2.13-0.16.0-SNAPSHOT.jar
+<div class="highlight-shell notranslate"><div
class="highlight"><pre><span></span><span class="nb">export</span><span
class="w"> </span><span class="nv">COMET_JAR</span><span
class="o">=</span>spark/target/comet-spark-spark4.1_2.13-0.17.0-SNAPSHOT.jar
<span class="nv">$SPARK_HOME</span>/bin/spark-shell<span class="w">
</span><span class="se">\</span>
<span class="w"> </span>--jars<span class="w"> </span><span
class="nv">$COMET_JAR</span><span class="w"> </span><span class="se">\</span>
@@ -598,7 +598,7 @@ source releases, or from the latest code in the GitHub
repository.</p>
</pre></div>
</div>
<p>Comet will log output similar to:</p>
-<div class="highlight-shell notranslate"><div
class="highlight"><pre><span></span>INFO<span class="w">
</span>core/src/lib.rs:<span class="w"> </span>Comet<span class="w">
</span>native<span class="w"> </span>library<span class="w">
</span>version<span class="w"> </span><span
class="m">0</span>.16.0-SNAPSHOT<span class="w"> </span>initialized
+<div class="highlight-shell notranslate"><div
class="highlight"><pre><span></span>INFO<span class="w">
</span>core/src/lib.rs:<span class="w"> </span>Comet<span class="w">
</span>native<span class="w"> </span>library<span class="w">
</span>version<span class="w"> </span><span
class="m">0</span>.17.0-SNAPSHOT<span class="w"> </span>initialized
WARN<span class="w"> </span>CometExecRule:<span class="w"> </span>Comet<span
class="w"> </span>cannot<span class="w"> </span>execute<span class="w">
</span>some<span class="w"> </span>parts<span class="w"> </span>of<span
class="w"> </span>this<span class="w"> </span>plan<span class="w">
</span>natively<span class="w"> </span><span class="o">(</span><span
class="nb">set</span><span class="w">
</span>spark.comet.explainFallback.enabled<span class="o">=</span><span
class="nb">false</span><s [...]
<span class="w"> </span>Execute<span class="w">
</span>InsertIntoHadoopFsRelationCommand<span class="w"> </span><span
class="o">[</span>COMET:<span class="w"> </span>Native<span class="w">
</span>support<span class="w"> </span><span class="k">for</span><span
class="w"> </span>operator<span class="w"> </span>DataWritingCommandExec<span
class="w"> </span>is<span class="w"> </span>disabled.<span class="w">
</span>Set<span class="w"> </span>spark.comet.parquet.write.enabled<span
class="o">= [...]
+-<span class="w"> </span>WriteFiles
@@ -628,7 +628,7 @@ CometNativeColumnarToRow
<p>Depending on your deployment mode you may also need to set the driver &
executor class path(s) to
explicitly contain Comet otherwise Spark may use a different class-loader for
the Comet components than its internal
components which will then fail at runtime. For example:</p>
-<div class="highlight-shell notranslate"><div
class="highlight"><pre><span></span>--driver-class-path<span class="w">
</span>spark/target/comet-spark-spark4.1_2.13-0.16.0-SNAPSHOT.jar
+<div class="highlight-shell notranslate"><div
class="highlight"><pre><span></span>--driver-class-path<span class="w">
</span>spark/target/comet-spark-spark4.1_2.13-0.17.0-SNAPSHOT.jar
</pre></div>
</div>
<p>Some cluster managers may require additional configuration, see <a
class="reference external"
href="https://spark.apache.org/docs/latest/cluster-overview.html";>https://spark.apache.org/docs/latest/cluster-overview.html</a></p>
@@ -656,7 +656,7 @@ allocation for native execution. See <a class="reference
internal" href="tuning.
<i class="fa-solid fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
- <p class="prev-next-title">Comet 0.16.0-SNAPSHOT User Guide</p>
+ <p class="prev-next-title">Comet 0.17.0-SNAPSHOT User Guide</p>
</div>
</a>
<a class="right-next"
diff --git a/user-guide/latest/kubernetes.html
b/user-guide/latest/kubernetes.html
index 6c7cf1b0a..d5b3fe523 100644
--- a/user-guide/latest/kubernetes.html
+++ b/user-guide/latest/kubernetes.html
@@ -422,7 +422,7 @@ under the License.
<li class="breadcrumb-item"><a href="../index.html" class="nav-link">Comet
User Guide</a></li>
- <li class="breadcrumb-item"><a href="index.html" class="nav-link">Comet
0.16.0-SNAPSHOT User Guide</a></li>
+ <li class="breadcrumb-item"><a href="index.html" class="nav-link">Comet
0.17.0-SNAPSHOT User Guide</a></li>
<li class="breadcrumb-item active" aria-current="page"><span
class="ellipsis">Comet Kubernetes Support</span></li>
</ul>
@@ -504,13 +504,13 @@ metadata:
spec:
<span class="w"> </span>type:<span class="w"> </span>Scala
<span class="w"> </span>mode:<span class="w"> </span>cluster
-<span class="w"> </span>image:<span class="w">
</span>apache/datafusion-comet:0.16.0-SNAPSHOT-spark3.5.5-scala2.12-java11
+<span class="w"> </span>image:<span class="w">
</span>apache/datafusion-comet:0.17.0-SNAPSHOT-spark3.5.5-scala2.12-java11
<span class="w"> </span>imagePullPolicy:<span class="w"> </span>IfNotPresent
<span class="w"> </span>mainClass:<span class="w">
</span>org.apache.spark.examples.SparkPi
<span class="w"> </span>mainApplicationFile:<span class="w">
</span>local:///opt/spark/examples/jars/spark-examples_2.13-4.1.1.jar
<span class="w"> </span>sparkConf:
-<span class="w"> </span><span
class="s2">"spark.executor.extraClassPath"</span>:<span class="w">
</span><span
class="s2">"/opt/spark/jars/comet-spark-spark3.5_2.12-0.16.0-SNAPSHOT.jar"</span>
-<span class="w"> </span><span
class="s2">"spark.driver.extraClassPath"</span>:<span class="w">
</span><span
class="s2">"/opt/spark/jars/comet-spark-spark3.5_2.12-0.16.0-SNAPSHOT.jar"</span>
+<span class="w"> </span><span
class="s2">"spark.executor.extraClassPath"</span>:<span class="w">
</span><span
class="s2">"/opt/spark/jars/comet-spark-spark3.5_2.12-0.17.0-SNAPSHOT.jar"</span>
+<span class="w"> </span><span
class="s2">"spark.driver.extraClassPath"</span>:<span class="w">
</span><span
class="s2">"/opt/spark/jars/comet-spark-spark3.5_2.12-0.17.0-SNAPSHOT.jar"</span>
<span class="w"> </span><span
class="s2">"spark.plugins"</span>:<span class="w"> </span><span
class="s2">"org.apache.spark.CometPlugin"</span>
<span class="w"> </span><span
class="s2">"spark.comet.enabled"</span>:<span class="w"> </span><span
class="s2">"true"</span>
<span class="w"> </span><span
class="s2">"spark.comet.exec.enabled"</span>:<span class="w">
</span><span class="s2">"true"</span>
diff --git a/user-guide/latest/metrics.html b/user-guide/latest/metrics.html
index 488a64767..14bb9fe88 100644
--- a/user-guide/latest/metrics.html
+++ b/user-guide/latest/metrics.html
@@ -422,7 +422,7 @@ under the License.
<li class="breadcrumb-item"><a href="../index.html" class="nav-link">Comet
User Guide</a></li>
- <li class="breadcrumb-item"><a href="index.html" class="nav-link">Comet
0.16.0-SNAPSHOT User Guide</a></li>
+ <li class="breadcrumb-item"><a href="index.html" class="nav-link">Comet
0.17.0-SNAPSHOT User Guide</a></li>
<li class="breadcrumb-item active" aria-current="page"><span
class="ellipsis">Comet Metrics</span></li>
</ul>
diff --git a/user-guide/latest/operators.html b/user-guide/latest/operators.html
index cb873b2b4..ccd30ad67 100644
--- a/user-guide/latest/operators.html
+++ b/user-guide/latest/operators.html
@@ -422,7 +422,7 @@ under the License.
<li class="breadcrumb-item"><a href="../index.html" class="nav-link">Comet
User Guide</a></li>
- <li class="breadcrumb-item"><a href="index.html" class="nav-link">Comet
0.16.0-SNAPSHOT User Guide</a></li>
+ <li class="breadcrumb-item"><a href="index.html" class="nav-link">Comet
0.17.0-SNAPSHOT User Guide</a></li>
<li class="breadcrumb-item active" aria-current="page"><span
class="ellipsis">Supported Spark Operators</span></li>
</ul>
diff --git a/user-guide/latest/source.html b/user-guide/latest/source.html
index 23b48452a..2370b2bc3 100644
--- a/user-guide/latest/source.html
+++ b/user-guide/latest/source.html
@@ -422,7 +422,7 @@ under the License.
<li class="breadcrumb-item"><a href="../index.html" class="nav-link">Comet
User Guide</a></li>
- <li class="breadcrumb-item"><a href="index.html" class="nav-link">Comet
0.16.0-SNAPSHOT User Guide</a></li>
+ <li class="breadcrumb-item"><a href="index.html" class="nav-link">Comet
0.17.0-SNAPSHOT User Guide</a></li>
<li class="breadcrumb-item active" aria-current="page"><span
class="ellipsis">Building Comet From Source</span></li>
</ul>
diff --git a/user-guide/latest/tuning.html b/user-guide/latest/tuning.html
index 09276d48d..ec808f4cd 100644
--- a/user-guide/latest/tuning.html
+++ b/user-guide/latest/tuning.html
@@ -422,7 +422,7 @@ under the License.
<li class="breadcrumb-item"><a href="../index.html" class="nav-link">Comet
User Guide</a></li>
- <li class="breadcrumb-item"><a href="index.html" class="nav-link">Comet
0.16.0-SNAPSHOT User Guide</a></li>
+ <li class="breadcrumb-item"><a href="index.html" class="nav-link">Comet
0.17.0-SNAPSHOT User Guide</a></li>
<li class="breadcrumb-item active" aria-current="page"><span
class="ellipsis">Comet Tuning Guide</span></li>
</ul>
diff --git a/user-guide/latest/understanding-comet-plans.html
b/user-guide/latest/understanding-comet-plans.html
index 5af4b4a9a..55a633dda 100644
--- a/user-guide/latest/understanding-comet-plans.html
+++ b/user-guide/latest/understanding-comet-plans.html
@@ -422,7 +422,7 @@ under the License.
<li class="breadcrumb-item"><a href="../index.html" class="nav-link">Comet
User Guide</a></li>
- <li class="breadcrumb-item"><a href="index.html" class="nav-link">Comet
0.16.0-SNAPSHOT User Guide</a></li>
+ <li class="breadcrumb-item"><a href="index.html" class="nav-link">Comet
0.17.0-SNAPSHOT User Guide</a></li>
<li class="breadcrumb-item active" aria-current="page"><span
class="ellipsis">Understanding Comet Plans</span></li>
</ul>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
