This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-staging
in repository https://gitbox.apache.org/repos/asf/datafusion-site.git
The following commit(s) were added to refs/heads/asf-staging by this push:
new afd76e4 Commit build products
afd76e4 is described below
commit afd76e415883c6b8141bd72edf399d80dd9c53a4
Author: Build Pelican (action) <[email protected]>
AuthorDate: Thu Aug 6 19:46:42 2026 +0000
Commit build products
---
blog/2026/08/07/datafusion-comet-1.0.0/index.html | 11 ++++++++++-
blog/feeds/all-en.atom.xml | 7 ++++++-
blog/feeds/blog.atom.xml | 7 ++++++-
blog/feeds/pmc.atom.xml | 7 ++++++-
4 files changed, 28 insertions(+), 4 deletions(-)
diff --git a/blog/2026/08/07/datafusion-comet-1.0.0/index.html
b/blog/2026/08/07/datafusion-comet-1.0.0/index.html
index 10b6e0d..2a2559b 100644
--- a/blog/2026/08/07/datafusion-comet-1.0.0/index.html
+++ b/blog/2026/08/07/datafusion-comet-1.0.0/index.html
@@ -55,11 +55,13 @@
<aside class="toc-container d-md-none mb-2">
<div class="toc"><span class="toctitle">Contents</span><ul>
<li><a href="#the-road-to-10">The Road to 1.0</a><ul>
+<li><a href="#support-for-spark-40-with-ansi-mode">Support for Spark 4.0+ with
ANSI mode</a></li>
<li><a href="#correctness-testing">Correctness Testing</a></li>
<li><a href="#performance">Performance</a></li>
<li><a href="#codegen-dispatch">Codegen Dispatch</a></li>
</ul>
</li>
+<li><a href="#scala-and-java-udf-support">Scala and Java UDF Support</a></li>
<li><a href="#improvements-since-0171">Improvements since 0.17.1</a><ul>
<li><a href="#experimental-pyarrow-udf-support">Experimental PyArrow UDF
Support</a></li>
<li><a href="#new-expression-and-aggregate-support">New Expression and
Aggregate Support</a></li>
@@ -104,6 +106,8 @@ contributors. See the <a
href="https://github.com/apache/datafusion-comet/blob/m
months later with support for 13 operators, and 106 expressions.</p>
<p>Comet 1.0.0 now supports more than 400 expressions, but that isn't the only
way the project has grown over this time.</p>
<p>Here's a recap of some of the main advances over the past two years</p>
+<h3 id="support-for-spark-40-with-ansi-mode">Support for Spark 4.0+ with ANSI
mode<a class="headerlink" href="#support-for-spark-40-with-ansi-mode"
title="Permanent link">¶</a></h3>
+<p>Comet 1.0.0 supports Spark versions 3.4 though 4.1, with experimental
support for 4.2. Comet fully supports Spark's ANSI mode, which is enabled by
default starting with Spark 4.0</p>
<h3 id="correctness-testing">Correctness Testing<a class="headerlink"
href="#correctness-testing" title="Permanent link">¶</a></h3>
<p>It is important that queries accelerated by Comet produce the same results
as Spark. Correctness checking has always been a large effort in Comet
development, but the approach has evolved over time.</p>
<ul>
@@ -115,9 +119,12 @@ months later with support for 13 operators, and 106
expressions.</p>
</ul>
<h3 id="performance">Performance<a class="headerlink" href="#performance"
title="Permanent link">¶</a></h3>
<p>The early Comet releases provided a very modest speedup and the published
benchmark results were based on running TPC workloads at small scale factors on
a single node. There are now independent benchmark results published by AWS
Labs that show significant speedups for <a
href="https://awslabs.github.io/data-on-eks/docs/benchmarks/spark-datafusion-comet-benchmark";>TPC-DS
@ 3TB running in EKS</a>.</p>
-<p>AWS Labs also published performance results <a
href="https://awslabs.github.io/data-on-eks/docs/benchmarks/spark-gluten-velox-comet-benchmark";>comparing
Comet with Apache Gluten</a>, concluding that "Gluten + Velox v1.6.0 and
DataFusion Comet v0.16.0 deliver similar overall runtime".</p>
+<p>AWS Labs also published performance results <a
href="https://awslabs.github.io/data-on-eks/docs/benchmarks/spark-gluten-velox-comet-benchmark";>comparing
Comet with Apache Gluten</a>, concluding that "Gluten + Velox v1.6.0 and
DataFusion Comet v0.16.0 deliver similar overall runtime". Comet's
documentation also has a guide explaining how <a
href="https://datafusion.apache.org/comet/about/gluten_comparison.html";>Comet
compares to Gluten</a>.</p>
<h3 id="codegen-dispatch">Codegen Dispatch<a class="headerlink"
href="#codegen-dispatch" title="Permanent link">¶</a></h3>
<p>A significant innovation that landed in 0.17.0 was to change the approach
to "native" acceleration. Rather than falling back to Spark row-based execution
whenever Comet lacked a native Rust implementation of an expression, Comet is
now able to execute Spark's own expression evaluation logic directly against
Arrow data in the Comet pipeline. This immediately expanded the number of
expressions that Comet could support without an expensive transition from
columnar to row-based data.</p>
+<p>Another advantage of this approch is that Comet can support certain
categories of expression, such as regular expressions, with 100% compatibility
with Spark, which is not practical when delegating to native code due to the
many differences between Java's regular expression engine and those available
in Rust or C++.</p>
+<h2 id="scala-and-java-udf-support">Scala and Java UDF Support<a
class="headerlink" href="#scala-and-java-udf-support" title="Permanent
link">¶</a></h2>
+<p>The codegen dispatch approach described in the previous chapter</p>
<h2 id="improvements-since-0171">Improvements since 0.17.1<a
class="headerlink" href="#improvements-since-0171" title="Permanent
link">¶</a></h2>
<p>The rest of this post covers what is new since the 0.17.1 release.</p>
<h3 id="experimental-pyarrow-udf-support">Experimental PyArrow UDF Support<a
class="headerlink" href="#experimental-pyarrow-udf-support" title="Permanent
link">¶</a></h3>
@@ -223,11 +230,13 @@ to get up and running, then point Comet at your existing
Spark workloads and see
<aside class="toc-container d-none d-md-block col-md-4 col-xl-3 ms-xl-2">
<div class="toc"><span class="toctitle">Contents</span><ul>
<li><a href="#the-road-to-10">The Road to 1.0</a><ul>
+<li><a href="#support-for-spark-40-with-ansi-mode">Support for Spark 4.0+ with
ANSI mode</a></li>
<li><a href="#correctness-testing">Correctness Testing</a></li>
<li><a href="#performance">Performance</a></li>
<li><a href="#codegen-dispatch">Codegen Dispatch</a></li>
</ul>
</li>
+<li><a href="#scala-and-java-udf-support">Scala and Java UDF Support</a></li>
<li><a href="#improvements-since-0171">Improvements since 0.17.1</a><ul>
<li><a href="#experimental-pyarrow-udf-support">Experimental PyArrow UDF
Support</a></li>
<li><a href="#new-expression-and-aggregate-support">New Expression and
Aggregate Support</a></li>
diff --git a/blog/feeds/all-en.atom.xml b/blog/feeds/all-en.atom.xml
index 02869f7..e06f0ff 100644
--- a/blog/feeds/all-en.atom.xml
+++ b/blog/feeds/all-en.atom.xml
@@ -50,6 +50,8 @@ contributors. See the <a
href="https://github.com/apache/datafusion-comet/blo
months later with support for 13 operators, and 106 expressions.</p>
<p>Comet 1.0.0 now supports more than 400 expressions, but that isn't
the only way the project has grown over this time.</p>
<p>Here's a recap of some of the main advances over the past two
years</p>
+<h3 id="support-for-spark-40-with-ansi-mode">Support for Spark 4.0+ with
ANSI mode<a class="headerlink" href="#support-for-spark-40-with-ansi-mode"
title="Permanent link">¶</a></h3>
+<p>Comet 1.0.0 supports Spark versions 3.4 though 4.1, with experimental
support for 4.2. Comet fully supports Spark's ANSI mode, which is enabled by
default starting with Spark 4.0</p>
<h3 id="correctness-testing">Correctness Testing<a class="headerlink"
href="#correctness-testing" title="Permanent link">¶</a></h3>
<p>It is important that queries accelerated by Comet produce the same
results as Spark. Correctness checking has always been a large effort in Comet
development, but the approach has evolved over time.</p>
<ul>
@@ -61,9 +63,12 @@ months later with support for 13 operators, and 106
expressions.</p>
</ul>
<h3 id="performance">Performance<a class="headerlink"
href="#performance" title="Permanent link">¶</a></h3>
<p>The early Comet releases provided a very modest speedup and the
published benchmark results were based on running TPC workloads at small scale
factors on a single node. There are now independent benchmark results published
by AWS Labs that show significant speedups for <a
href="https://awslabs.github.io/data-on-eks/docs/benchmarks/spark-datafusion-comet-benchmark">TPC-DS
@ 3TB running in EKS</a>.</p>
-<p>AWS Labs also published performance results <a
href="https://awslabs.github.io/data-on-eks/docs/benchmarks/spark-gluten-velox-comet-benchmark">comparing
Comet with Apache Gluten</a>, concluding that "Gluten + Velox v1.6.0 and
DataFusion Comet v0.16.0 deliver similar overall runtime".</p>
+<p>AWS Labs also published performance results <a
href="https://awslabs.github.io/data-on-eks/docs/benchmarks/spark-gluten-velox-comet-benchmark">comparing
Comet with Apache Gluten</a>, concluding that "Gluten + Velox v1.6.0 and
DataFusion Comet v0.16.0 deliver similar overall runtime". Comet's
documentation also has a guide explaining how <a
href="https://datafusion.apache.org/comet/about/gluten_comparison.html">Comet
compares to Gluten</a>.</p>
<h3 id="codegen-dispatch">Codegen Dispatch<a class="headerlink"
href="#codegen-dispatch" title="Permanent link">¶</a></h3>
<p>A significant innovation that landed in 0.17.0 was to change the
approach to "native" acceleration. Rather than falling back to Spark row-based
execution whenever Comet lacked a native Rust implementation of an expression,
Comet is now able to execute Spark's own expression evaluation logic directly
against Arrow data in the Comet pipeline. This immediately expanded the number
of expressions that Comet could support without an expensive transition from
columnar to row-based data [...]
+<p>Another advantage of this approch is that Comet can support certain
categories of expression, such as regular expressions, with 100% compatibility
with Spark, which is not practical when delegating to native code due to the
many differences between Java's regular expression engine and those available
in Rust or C++.</p>
+<h2 id="scala-and-java-udf-support">Scala and Java UDF Support<a
class="headerlink" href="#scala-and-java-udf-support" title="Permanent
link">¶</a></h2>
+<p>The codegen dispatch approach described in the previous
chapter</p>
<h2 id="improvements-since-0171">Improvements since 0.17.1<a
class="headerlink" href="#improvements-since-0171" title="Permanent
link">¶</a></h2>
<p>The rest of this post covers what is new since the 0.17.1
release.</p>
<h3 id="experimental-pyarrow-udf-support">Experimental PyArrow UDF
Support<a class="headerlink" href="#experimental-pyarrow-udf-support"
title="Permanent link">¶</a></h3>
diff --git a/blog/feeds/blog.atom.xml b/blog/feeds/blog.atom.xml
index 141b1e2..eda66db 100644
--- a/blog/feeds/blog.atom.xml
+++ b/blog/feeds/blog.atom.xml
@@ -50,6 +50,8 @@ contributors. See the <a
href="https://github.com/apache/datafusion-comet/blo
months later with support for 13 operators, and 106 expressions.</p>
<p>Comet 1.0.0 now supports more than 400 expressions, but that isn't
the only way the project has grown over this time.</p>
<p>Here's a recap of some of the main advances over the past two
years</p>
+<h3 id="support-for-spark-40-with-ansi-mode">Support for Spark 4.0+ with
ANSI mode<a class="headerlink" href="#support-for-spark-40-with-ansi-mode"
title="Permanent link">¶</a></h3>
+<p>Comet 1.0.0 supports Spark versions 3.4 though 4.1, with experimental
support for 4.2. Comet fully supports Spark's ANSI mode, which is enabled by
default starting with Spark 4.0</p>
<h3 id="correctness-testing">Correctness Testing<a class="headerlink"
href="#correctness-testing" title="Permanent link">¶</a></h3>
<p>It is important that queries accelerated by Comet produce the same
results as Spark. Correctness checking has always been a large effort in Comet
development, but the approach has evolved over time.</p>
<ul>
@@ -61,9 +63,12 @@ months later with support for 13 operators, and 106
expressions.</p>
</ul>
<h3 id="performance">Performance<a class="headerlink"
href="#performance" title="Permanent link">¶</a></h3>
<p>The early Comet releases provided a very modest speedup and the
published benchmark results were based on running TPC workloads at small scale
factors on a single node. There are now independent benchmark results published
by AWS Labs that show significant speedups for <a
href="https://awslabs.github.io/data-on-eks/docs/benchmarks/spark-datafusion-comet-benchmark">TPC-DS
@ 3TB running in EKS</a>.</p>
-<p>AWS Labs also published performance results <a
href="https://awslabs.github.io/data-on-eks/docs/benchmarks/spark-gluten-velox-comet-benchmark">comparing
Comet with Apache Gluten</a>, concluding that "Gluten + Velox v1.6.0 and
DataFusion Comet v0.16.0 deliver similar overall runtime".</p>
+<p>AWS Labs also published performance results <a
href="https://awslabs.github.io/data-on-eks/docs/benchmarks/spark-gluten-velox-comet-benchmark">comparing
Comet with Apache Gluten</a>, concluding that "Gluten + Velox v1.6.0 and
DataFusion Comet v0.16.0 deliver similar overall runtime". Comet's
documentation also has a guide explaining how <a
href="https://datafusion.apache.org/comet/about/gluten_comparison.html">Comet
compares to Gluten</a>.</p>
<h3 id="codegen-dispatch">Codegen Dispatch<a class="headerlink"
href="#codegen-dispatch" title="Permanent link">¶</a></h3>
<p>A significant innovation that landed in 0.17.0 was to change the
approach to "native" acceleration. Rather than falling back to Spark row-based
execution whenever Comet lacked a native Rust implementation of an expression,
Comet is now able to execute Spark's own expression evaluation logic directly
against Arrow data in the Comet pipeline. This immediately expanded the number
of expressions that Comet could support without an expensive transition from
columnar to row-based data [...]
+<p>Another advantage of this approch is that Comet can support certain
categories of expression, such as regular expressions, with 100% compatibility
with Spark, which is not practical when delegating to native code due to the
many differences between Java's regular expression engine and those available
in Rust or C++.</p>
+<h2 id="scala-and-java-udf-support">Scala and Java UDF Support<a
class="headerlink" href="#scala-and-java-udf-support" title="Permanent
link">¶</a></h2>
+<p>The codegen dispatch approach described in the previous
chapter</p>
<h2 id="improvements-since-0171">Improvements since 0.17.1<a
class="headerlink" href="#improvements-since-0171" title="Permanent
link">¶</a></h2>
<p>The rest of this post covers what is new since the 0.17.1
release.</p>
<h3 id="experimental-pyarrow-udf-support">Experimental PyArrow UDF
Support<a class="headerlink" href="#experimental-pyarrow-udf-support"
title="Permanent link">¶</a></h3>
diff --git a/blog/feeds/pmc.atom.xml b/blog/feeds/pmc.atom.xml
index b09f4e6..adb6bd6 100644
--- a/blog/feeds/pmc.atom.xml
+++ b/blog/feeds/pmc.atom.xml
@@ -50,6 +50,8 @@ contributors. See the <a
href="https://github.com/apache/datafusion-comet/blo
months later with support for 13 operators, and 106 expressions.</p>
<p>Comet 1.0.0 now supports more than 400 expressions, but that isn't
the only way the project has grown over this time.</p>
<p>Here's a recap of some of the main advances over the past two
years</p>
+<h3 id="support-for-spark-40-with-ansi-mode">Support for Spark 4.0+ with
ANSI mode<a class="headerlink" href="#support-for-spark-40-with-ansi-mode"
title="Permanent link">¶</a></h3>
+<p>Comet 1.0.0 supports Spark versions 3.4 though 4.1, with experimental
support for 4.2. Comet fully supports Spark's ANSI mode, which is enabled by
default starting with Spark 4.0</p>
<h3 id="correctness-testing">Correctness Testing<a class="headerlink"
href="#correctness-testing" title="Permanent link">¶</a></h3>
<p>It is important that queries accelerated by Comet produce the same
results as Spark. Correctness checking has always been a large effort in Comet
development, but the approach has evolved over time.</p>
<ul>
@@ -61,9 +63,12 @@ months later with support for 13 operators, and 106
expressions.</p>
</ul>
<h3 id="performance">Performance<a class="headerlink"
href="#performance" title="Permanent link">¶</a></h3>
<p>The early Comet releases provided a very modest speedup and the
published benchmark results were based on running TPC workloads at small scale
factors on a single node. There are now independent benchmark results published
by AWS Labs that show significant speedups for <a
href="https://awslabs.github.io/data-on-eks/docs/benchmarks/spark-datafusion-comet-benchmark">TPC-DS
@ 3TB running in EKS</a>.</p>
-<p>AWS Labs also published performance results <a
href="https://awslabs.github.io/data-on-eks/docs/benchmarks/spark-gluten-velox-comet-benchmark">comparing
Comet with Apache Gluten</a>, concluding that "Gluten + Velox v1.6.0 and
DataFusion Comet v0.16.0 deliver similar overall runtime".</p>
+<p>AWS Labs also published performance results <a
href="https://awslabs.github.io/data-on-eks/docs/benchmarks/spark-gluten-velox-comet-benchmark">comparing
Comet with Apache Gluten</a>, concluding that "Gluten + Velox v1.6.0 and
DataFusion Comet v0.16.0 deliver similar overall runtime". Comet's
documentation also has a guide explaining how <a
href="https://datafusion.apache.org/comet/about/gluten_comparison.html">Comet
compares to Gluten</a>.</p>
<h3 id="codegen-dispatch">Codegen Dispatch<a class="headerlink"
href="#codegen-dispatch" title="Permanent link">¶</a></h3>
<p>A significant innovation that landed in 0.17.0 was to change the
approach to "native" acceleration. Rather than falling back to Spark row-based
execution whenever Comet lacked a native Rust implementation of an expression,
Comet is now able to execute Spark's own expression evaluation logic directly
against Arrow data in the Comet pipeline. This immediately expanded the number
of expressions that Comet could support without an expensive transition from
columnar to row-based data [...]
+<p>Another advantage of this approch is that Comet can support certain
categories of expression, such as regular expressions, with 100% compatibility
with Spark, which is not practical when delegating to native code due to the
many differences between Java's regular expression engine and those available
in Rust or C++.</p>
+<h2 id="scala-and-java-udf-support">Scala and Java UDF Support<a
class="headerlink" href="#scala-and-java-udf-support" title="Permanent
link">¶</a></h2>
+<p>The codegen dispatch approach described in the previous
chapter</p>
<h2 id="improvements-since-0171">Improvements since 0.17.1<a
class="headerlink" href="#improvements-since-0171" title="Permanent
link">¶</a></h2>
<p>The rest of this post covers what is new since the 0.17.1
release.</p>
<h3 id="experimental-pyarrow-udf-support">Experimental PyArrow UDF
Support<a class="headerlink" href="#experimental-pyarrow-udf-support"
title="Permanent link">¶</a></h3>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
