This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-staging
in repository https://gitbox.apache.org/repos/asf/datafusion-site.git
The following commit(s) were added to refs/heads/asf-staging by this push:
new e526151 Commit build products
e526151 is described below
commit e52615130134051f4569f6b5eb76bf3dc9331c40
Author: Build Pelican (action) <[email protected]>
AuthorDate: Thu Aug 6 19:55:49 2026 +0000
Commit build products
---
blog/2026/08/07/datafusion-comet-1.0.0/index.html | 15 +++++++--------
blog/feeds/all-en.atom.xml | 15 +++++++--------
blog/feeds/blog.atom.xml | 15 +++++++--------
blog/feeds/pmc.atom.xml | 15 +++++++--------
4 files changed, 28 insertions(+), 32 deletions(-)
diff --git a/blog/2026/08/07/datafusion-comet-1.0.0/index.html
b/blog/2026/08/07/datafusion-comet-1.0.0/index.html
index 2a2559b..2fda272 100644
--- a/blog/2026/08/07/datafusion-comet-1.0.0/index.html
+++ b/blog/2026/08/07/datafusion-comet-1.0.0/index.html
@@ -103,11 +103,11 @@ improved performance and efficiency without requiring any
code changes.</p>
contributors. See the <a
href="https://github.com/apache/datafusion-comet/blob/main/docs/source/changelog/1.0.0.md";>change
log</a> for the full list of changes.</p>
<h2 id="the-road-to-10">The Road to 1.0<a class="headerlink"
href="#the-road-to-10" title="Permanent link">¶</a></h2>
<p>Comet was <a
href="https://datafusion.apache.org/blog/2024/03/06/comet-donation/";>donated</a>
to the Apache DataFusion project in March 2024 and cut its first release,
0.1.0, five
-months later with support for 13 operators, and 106 expressions.</p>
+months later with support for 13 operators and 106 expressions.</p>
<p>Comet 1.0.0 now supports more than 400 expressions, but that isn't the only
way the project has grown over this time.</p>
-<p>Here's a recap of some of the main advances over the past two years</p>
+<p>Here's a recap of some of the main advances over the past two years.</p>
<h3 id="support-for-spark-40-with-ansi-mode">Support for Spark 4.0+ with ANSI
mode<a class="headerlink" href="#support-for-spark-40-with-ansi-mode"
title="Permanent link">¶</a></h3>
-<p>Comet 1.0.0 supports Spark versions 3.4 though 4.1, with experimental
support for 4.2. Comet fully supports Spark's ANSI mode, which is enabled by
default starting with Spark 4.0</p>
+<p>Comet 1.0.0 supports Spark versions 3.4 though 4.1, with experimental
support for 4.2. Comet fully supports Spark's ANSI mode, which is enabled by
default starting with Spark 4.0.</p>
<h3 id="correctness-testing">Correctness Testing<a class="headerlink"
href="#correctness-testing" title="Permanent link">¶</a></h3>
<p>It is important that queries accelerated by Comet produce the same results
as Spark. Correctness checking has always been a large effort in Comet
development, but the approach has evolved over time.</p>
<ul>
@@ -115,14 +115,13 @@ months later with support for 13 operators, and 106
expressions.</p>
<li>Scala tests: Comet has Scala tests that run queries end to end with Comet
enabled vs disabled and ensure that the results match</li>
<li>Fuzz testing: Many of the scala tests use a fuzz testing approach to
generate randomized data that queries run against, helping to catch regressions
around edge cases such as nulls, NaN, Infinity, and timezone issues</li>
<li>Comet SQL Tests: In an effort to make it easier to write tests, Comet now
provides a SQL-based testing approach that is inspired by sqllogictest</li>
-<li>Generative AI: More recently, Comet has taken advantage of agentic skills
to perform audit sweeps of ll expressions, comapring the implementation to
Spark's souce code and ensuring that Comet has tests covering all important
edge cases</li>
+<li>Generative AI: More recently, Comet has taken advantage of agentic skills
to perform audit sweeps of all expressions, comparing the implementation to
Spark's source code and ensuring that Comet has tests covering all important
edge cases</li>
</ul>
<h3 id="performance">Performance<a class="headerlink" href="#performance"
title="Permanent link">¶</a></h3>
<p>The early Comet releases provided a very modest speedup and the published
benchmark results were based on running TPC workloads at small scale factors on
a single node. There are now independent benchmark results published by AWS
Labs that show significant speedups for <a
href="https://awslabs.github.io/data-on-eks/docs/benchmarks/spark-datafusion-comet-benchmark";>TPC-DS
@ 3TB running in EKS</a>.</p>
-<p>AWS Labs also published performance results <a
href="https://awslabs.github.io/data-on-eks/docs/benchmarks/spark-gluten-velox-comet-benchmark";>comparing
Comet with Apache Gluten</a>, concluding that "Gluten + Velox v1.6.0 and
DataFusion Comet v0.16.0 deliver similar overall runtime". Comet's
documentation also has a guide explaining how <a
href="https://datafusion.apache.org/comet/about/gluten_comparison.html";>Comet
compares to Gluten</a>.</p>
<h3 id="codegen-dispatch">Codegen Dispatch<a class="headerlink"
href="#codegen-dispatch" title="Permanent link">¶</a></h3>
<p>A significant innovation that landed in 0.17.0 was to change the approach
to "native" acceleration. Rather than falling back to Spark row-based execution
whenever Comet lacked a native Rust implementation of an expression, Comet is
now able to execute Spark's own expression evaluation logic directly against
Arrow data in the Comet pipeline. This immediately expanded the number of
expressions that Comet could support without an expensive transition from
columnar to row-based data.</p>
-<p>Another advantage of this approch is that Comet can support certain
categories of expression, such as regular expressions, with 100% compatibility
with Spark, which is not practical when delegating to native code due to the
many differences between Java's regular expression engine and those available
in Rust or C++.</p>
+<p>Another advantage of this approach is that Comet can support certain
categories of expression, such as regular expressions, with 100% compatibility
with Spark, which is not practical when delegating to native code due to the
many differences between Java's regular expression engine and those available
in Rust or C++.</p>
<h2 id="scala-and-java-udf-support">Scala and Java UDF Support<a
class="headerlink" href="#scala-and-java-udf-support" title="Permanent
link">¶</a></h2>
<p>The codegen dispatch approach described in the previous chapter</p>
<h2 id="improvements-since-0171">Improvements since 0.17.1<a
class="headerlink" href="#improvements-since-0171" title="Permanent
link">¶</a></h2>
@@ -136,8 +135,8 @@ exists. This is an early-stage feature and we welcome
feedback from users experi
<p>This release expands the set of Spark expressions and aggregates that are
accelerated by Comet:</p>
<ul>
<li><strong>Aggregates</strong>: <code>approx_percentile</code> /
<code>percentile_approx</code>, exact <code>percentile</code> /
<code>median</code>,
- <code>approx_count_distinct</code>, and native <code>collect_list</code> /
<code>array_agg</code>.
- <strong>Cast</strong>: Cast expressions where the native implementation is
marked as incompatible or unsupported
+ <code>approx_count_distinct</code>, and native <code>collect_list</code> /
<code>array_agg</code>.</li>
+<li><strong>Cast</strong>: Cast expressions where the native implementation is
marked as incompatible or unsupported
are now routed through codegen dispatch.</li>
<li><strong>Grouping</strong>: <code>grouping()</code> and
<code>grouping_id()</code>.</li>
<li><strong>Intervals</strong>: interval types via
<code>make_ym_interval</code> and <code>make_dt_interval</code>,
<code>CalendarIntervalType</code>,
diff --git a/blog/feeds/all-en.atom.xml b/blog/feeds/all-en.atom.xml
index e06f0ff..14b8511 100644
--- a/blog/feeds/all-en.atom.xml
+++ b/blog/feeds/all-en.atom.xml
@@ -47,11 +47,11 @@ improved performance and efficiency without requiring any
code changes.</p>
contributors. See the <a
href="https://github.com/apache/datafusion-comet/blob/main/docs/source/changelog/1.0.0.md">change
log</a> for the full list of changes.</p>
<h2 id="the-road-to-10">The Road to 1.0<a class="headerlink"
href="#the-road-to-10" title="Permanent link">¶</a></h2>
<p>Comet was <a
href="https://datafusion.apache.org/blog/2024/03/06/comet-donation/">donated</a>
to the Apache DataFusion project in March 2024 and cut its first release,
0.1.0, five
-months later with support for 13 operators, and 106 expressions.</p>
+months later with support for 13 operators and 106 expressions.</p>
<p>Comet 1.0.0 now supports more than 400 expressions, but that isn't
the only way the project has grown over this time.</p>
-<p>Here's a recap of some of the main advances over the past two
years</p>
+<p>Here's a recap of some of the main advances over the past two
years.</p>
<h3 id="support-for-spark-40-with-ansi-mode">Support for Spark 4.0+ with
ANSI mode<a class="headerlink" href="#support-for-spark-40-with-ansi-mode"
title="Permanent link">¶</a></h3>
-<p>Comet 1.0.0 supports Spark versions 3.4 though 4.1, with experimental
support for 4.2. Comet fully supports Spark's ANSI mode, which is enabled by
default starting with Spark 4.0</p>
+<p>Comet 1.0.0 supports Spark versions 3.4 though 4.1, with experimental
support for 4.2. Comet fully supports Spark's ANSI mode, which is enabled by
default starting with Spark 4.0.</p>
<h3 id="correctness-testing">Correctness Testing<a class="headerlink"
href="#correctness-testing" title="Permanent link">¶</a></h3>
<p>It is important that queries accelerated by Comet produce the same
results as Spark. Correctness checking has always been a large effort in Comet
development, but the approach has evolved over time.</p>
<ul>
@@ -59,14 +59,13 @@ months later with support for 13 operators, and 106
expressions.</p>
<li>Scala tests: Comet has Scala tests that run queries end to end with
Comet enabled vs disabled and ensure that the results match</li>
<li>Fuzz testing: Many of the scala tests use a fuzz testing approach to
generate randomized data that queries run against, helping to catch regressions
around edge cases such as nulls, NaN, Infinity, and timezone issues</li>
<li>Comet SQL Tests: In an effort to make it easier to write tests,
Comet now provides a SQL-based testing approach that is inspired by
sqllogictest</li>
-<li>Generative AI: More recently, Comet has taken advantage of agentic
skills to perform audit sweeps of ll expressions, comapring the implementation
to Spark's souce code and ensuring that Comet has tests covering all important
edge cases</li>
+<li>Generative AI: More recently, Comet has taken advantage of agentic
skills to perform audit sweeps of all expressions, comparing the implementation
to Spark's source code and ensuring that Comet has tests covering all important
edge cases</li>
</ul>
<h3 id="performance">Performance<a class="headerlink"
href="#performance" title="Permanent link">¶</a></h3>
<p>The early Comet releases provided a very modest speedup and the
published benchmark results were based on running TPC workloads at small scale
factors on a single node. There are now independent benchmark results published
by AWS Labs that show significant speedups for <a
href="https://awslabs.github.io/data-on-eks/docs/benchmarks/spark-datafusion-comet-benchmark">TPC-DS
@ 3TB running in EKS</a>.</p>
-<p>AWS Labs also published performance results <a
href="https://awslabs.github.io/data-on-eks/docs/benchmarks/spark-gluten-velox-comet-benchmark">comparing
Comet with Apache Gluten</a>, concluding that "Gluten + Velox v1.6.0 and
DataFusion Comet v0.16.0 deliver similar overall runtime". Comet's
documentation also has a guide explaining how <a
href="https://datafusion.apache.org/comet/about/gluten_comparison.html">Comet
compares to Gluten</a>.</p>
<h3 id="codegen-dispatch">Codegen Dispatch<a class="headerlink"
href="#codegen-dispatch" title="Permanent link">¶</a></h3>
<p>A significant innovation that landed in 0.17.0 was to change the
approach to "native" acceleration. Rather than falling back to Spark row-based
execution whenever Comet lacked a native Rust implementation of an expression,
Comet is now able to execute Spark's own expression evaluation logic directly
against Arrow data in the Comet pipeline. This immediately expanded the number
of expressions that Comet could support without an expensive transition from
columnar to row-based data [...]
-<p>Another advantage of this approch is that Comet can support certain
categories of expression, such as regular expressions, with 100% compatibility
with Spark, which is not practical when delegating to native code due to the
many differences between Java's regular expression engine and those available
in Rust or C++.</p>
+<p>Another advantage of this approach is that Comet can support certain
categories of expression, such as regular expressions, with 100% compatibility
with Spark, which is not practical when delegating to native code due to the
many differences between Java's regular expression engine and those available
in Rust or C++.</p>
<h2 id="scala-and-java-udf-support">Scala and Java UDF Support<a
class="headerlink" href="#scala-and-java-udf-support" title="Permanent
link">¶</a></h2>
<p>The codegen dispatch approach described in the previous
chapter</p>
<h2 id="improvements-since-0171">Improvements since 0.17.1<a
class="headerlink" href="#improvements-since-0171" title="Permanent
link">¶</a></h2>
@@ -80,8 +79,8 @@ exists. This is an early-stage feature and we welcome
feedback from users experi
<p>This release expands the set of Spark expressions and aggregates that
are accelerated by Comet:</p>
<ul>
<li><strong>Aggregates</strong>:
<code>approx_percentile</code> /
<code>percentile_approx</code>, exact
<code>percentile</code> / <code>median</code>,
- <code>approx_count_distinct</code>, and native
<code>collect_list</code> / <code>array_agg</code>.
- <strong>Cast</strong>: Cast expressions where the native
implementation is marked as incompatible or unsupported
+ <code>approx_count_distinct</code>, and native
<code>collect_list</code> /
<code>array_agg</code>.</li>
+<li><strong>Cast</strong>: Cast expressions where the native
implementation is marked as incompatible or unsupported
are now routed through codegen dispatch.</li>
<li><strong>Grouping</strong>:
<code>grouping()</code> and
<code>grouping_id()</code>.</li>
<li><strong>Intervals</strong>: interval types via
<code>make_ym_interval</code> and
<code>make_dt_interval</code>,
<code>CalendarIntervalType</code>,
diff --git a/blog/feeds/blog.atom.xml b/blog/feeds/blog.atom.xml
index eda66db..03885ec 100644
--- a/blog/feeds/blog.atom.xml
+++ b/blog/feeds/blog.atom.xml
@@ -47,11 +47,11 @@ improved performance and efficiency without requiring any
code changes.</p>
contributors. See the <a
href="https://github.com/apache/datafusion-comet/blob/main/docs/source/changelog/1.0.0.md">change
log</a> for the full list of changes.</p>
<h2 id="the-road-to-10">The Road to 1.0<a class="headerlink"
href="#the-road-to-10" title="Permanent link">¶</a></h2>
<p>Comet was <a
href="https://datafusion.apache.org/blog/2024/03/06/comet-donation/">donated</a>
to the Apache DataFusion project in March 2024 and cut its first release,
0.1.0, five
-months later with support for 13 operators, and 106 expressions.</p>
+months later with support for 13 operators and 106 expressions.</p>
<p>Comet 1.0.0 now supports more than 400 expressions, but that isn't
the only way the project has grown over this time.</p>
-<p>Here's a recap of some of the main advances over the past two
years</p>
+<p>Here's a recap of some of the main advances over the past two
years.</p>
<h3 id="support-for-spark-40-with-ansi-mode">Support for Spark 4.0+ with
ANSI mode<a class="headerlink" href="#support-for-spark-40-with-ansi-mode"
title="Permanent link">¶</a></h3>
-<p>Comet 1.0.0 supports Spark versions 3.4 though 4.1, with experimental
support for 4.2. Comet fully supports Spark's ANSI mode, which is enabled by
default starting with Spark 4.0</p>
+<p>Comet 1.0.0 supports Spark versions 3.4 though 4.1, with experimental
support for 4.2. Comet fully supports Spark's ANSI mode, which is enabled by
default starting with Spark 4.0.</p>
<h3 id="correctness-testing">Correctness Testing<a class="headerlink"
href="#correctness-testing" title="Permanent link">¶</a></h3>
<p>It is important that queries accelerated by Comet produce the same
results as Spark. Correctness checking has always been a large effort in Comet
development, but the approach has evolved over time.</p>
<ul>
@@ -59,14 +59,13 @@ months later with support for 13 operators, and 106
expressions.</p>
<li>Scala tests: Comet has Scala tests that run queries end to end with
Comet enabled vs disabled and ensure that the results match</li>
<li>Fuzz testing: Many of the scala tests use a fuzz testing approach to
generate randomized data that queries run against, helping to catch regressions
around edge cases such as nulls, NaN, Infinity, and timezone issues</li>
<li>Comet SQL Tests: In an effort to make it easier to write tests,
Comet now provides a SQL-based testing approach that is inspired by
sqllogictest</li>
-<li>Generative AI: More recently, Comet has taken advantage of agentic
skills to perform audit sweeps of ll expressions, comapring the implementation
to Spark's souce code and ensuring that Comet has tests covering all important
edge cases</li>
+<li>Generative AI: More recently, Comet has taken advantage of agentic
skills to perform audit sweeps of all expressions, comparing the implementation
to Spark's source code and ensuring that Comet has tests covering all important
edge cases</li>
</ul>
<h3 id="performance">Performance<a class="headerlink"
href="#performance" title="Permanent link">¶</a></h3>
<p>The early Comet releases provided a very modest speedup and the
published benchmark results were based on running TPC workloads at small scale
factors on a single node. There are now independent benchmark results published
by AWS Labs that show significant speedups for <a
href="https://awslabs.github.io/data-on-eks/docs/benchmarks/spark-datafusion-comet-benchmark">TPC-DS
@ 3TB running in EKS</a>.</p>
-<p>AWS Labs also published performance results <a
href="https://awslabs.github.io/data-on-eks/docs/benchmarks/spark-gluten-velox-comet-benchmark">comparing
Comet with Apache Gluten</a>, concluding that "Gluten + Velox v1.6.0 and
DataFusion Comet v0.16.0 deliver similar overall runtime". Comet's
documentation also has a guide explaining how <a
href="https://datafusion.apache.org/comet/about/gluten_comparison.html">Comet
compares to Gluten</a>.</p>
<h3 id="codegen-dispatch">Codegen Dispatch<a class="headerlink"
href="#codegen-dispatch" title="Permanent link">¶</a></h3>
<p>A significant innovation that landed in 0.17.0 was to change the
approach to "native" acceleration. Rather than falling back to Spark row-based
execution whenever Comet lacked a native Rust implementation of an expression,
Comet is now able to execute Spark's own expression evaluation logic directly
against Arrow data in the Comet pipeline. This immediately expanded the number
of expressions that Comet could support without an expensive transition from
columnar to row-based data [...]
-<p>Another advantage of this approch is that Comet can support certain
categories of expression, such as regular expressions, with 100% compatibility
with Spark, which is not practical when delegating to native code due to the
many differences between Java's regular expression engine and those available
in Rust or C++.</p>
+<p>Another advantage of this approach is that Comet can support certain
categories of expression, such as regular expressions, with 100% compatibility
with Spark, which is not practical when delegating to native code due to the
many differences between Java's regular expression engine and those available
in Rust or C++.</p>
<h2 id="scala-and-java-udf-support">Scala and Java UDF Support<a
class="headerlink" href="#scala-and-java-udf-support" title="Permanent
link">¶</a></h2>
<p>The codegen dispatch approach described in the previous
chapter</p>
<h2 id="improvements-since-0171">Improvements since 0.17.1<a
class="headerlink" href="#improvements-since-0171" title="Permanent
link">¶</a></h2>
@@ -80,8 +79,8 @@ exists. This is an early-stage feature and we welcome
feedback from users experi
<p>This release expands the set of Spark expressions and aggregates that
are accelerated by Comet:</p>
<ul>
<li><strong>Aggregates</strong>:
<code>approx_percentile</code> /
<code>percentile_approx</code>, exact
<code>percentile</code> / <code>median</code>,
- <code>approx_count_distinct</code>, and native
<code>collect_list</code> / <code>array_agg</code>.
- <strong>Cast</strong>: Cast expressions where the native
implementation is marked as incompatible or unsupported
+ <code>approx_count_distinct</code>, and native
<code>collect_list</code> /
<code>array_agg</code>.</li>
+<li><strong>Cast</strong>: Cast expressions where the native
implementation is marked as incompatible or unsupported
are now routed through codegen dispatch.</li>
<li><strong>Grouping</strong>:
<code>grouping()</code> and
<code>grouping_id()</code>.</li>
<li><strong>Intervals</strong>: interval types via
<code>make_ym_interval</code> and
<code>make_dt_interval</code>,
<code>CalendarIntervalType</code>,
diff --git a/blog/feeds/pmc.atom.xml b/blog/feeds/pmc.atom.xml
index adb6bd6..0a8c910 100644
--- a/blog/feeds/pmc.atom.xml
+++ b/blog/feeds/pmc.atom.xml
@@ -47,11 +47,11 @@ improved performance and efficiency without requiring any
code changes.</p>
contributors. See the <a
href="https://github.com/apache/datafusion-comet/blob/main/docs/source/changelog/1.0.0.md">change
log</a> for the full list of changes.</p>
<h2 id="the-road-to-10">The Road to 1.0<a class="headerlink"
href="#the-road-to-10" title="Permanent link">¶</a></h2>
<p>Comet was <a
href="https://datafusion.apache.org/blog/2024/03/06/comet-donation/">donated</a>
to the Apache DataFusion project in March 2024 and cut its first release,
0.1.0, five
-months later with support for 13 operators, and 106 expressions.</p>
+months later with support for 13 operators and 106 expressions.</p>
<p>Comet 1.0.0 now supports more than 400 expressions, but that isn't
the only way the project has grown over this time.</p>
-<p>Here's a recap of some of the main advances over the past two
years</p>
+<p>Here's a recap of some of the main advances over the past two
years.</p>
<h3 id="support-for-spark-40-with-ansi-mode">Support for Spark 4.0+ with
ANSI mode<a class="headerlink" href="#support-for-spark-40-with-ansi-mode"
title="Permanent link">¶</a></h3>
-<p>Comet 1.0.0 supports Spark versions 3.4 though 4.1, with experimental
support for 4.2. Comet fully supports Spark's ANSI mode, which is enabled by
default starting with Spark 4.0</p>
+<p>Comet 1.0.0 supports Spark versions 3.4 though 4.1, with experimental
support for 4.2. Comet fully supports Spark's ANSI mode, which is enabled by
default starting with Spark 4.0.</p>
<h3 id="correctness-testing">Correctness Testing<a class="headerlink"
href="#correctness-testing" title="Permanent link">¶</a></h3>
<p>It is important that queries accelerated by Comet produce the same
results as Spark. Correctness checking has always been a large effort in Comet
development, but the approach has evolved over time.</p>
<ul>
@@ -59,14 +59,13 @@ months later with support for 13 operators, and 106
expressions.</p>
<li>Scala tests: Comet has Scala tests that run queries end to end with
Comet enabled vs disabled and ensure that the results match</li>
<li>Fuzz testing: Many of the scala tests use a fuzz testing approach to
generate randomized data that queries run against, helping to catch regressions
around edge cases such as nulls, NaN, Infinity, and timezone issues</li>
<li>Comet SQL Tests: In an effort to make it easier to write tests,
Comet now provides a SQL-based testing approach that is inspired by
sqllogictest</li>
-<li>Generative AI: More recently, Comet has taken advantage of agentic
skills to perform audit sweeps of ll expressions, comapring the implementation
to Spark's souce code and ensuring that Comet has tests covering all important
edge cases</li>
+<li>Generative AI: More recently, Comet has taken advantage of agentic
skills to perform audit sweeps of all expressions, comparing the implementation
to Spark's source code and ensuring that Comet has tests covering all important
edge cases</li>
</ul>
<h3 id="performance">Performance<a class="headerlink"
href="#performance" title="Permanent link">¶</a></h3>
<p>The early Comet releases provided a very modest speedup and the
published benchmark results were based on running TPC workloads at small scale
factors on a single node. There are now independent benchmark results published
by AWS Labs that show significant speedups for <a
href="https://awslabs.github.io/data-on-eks/docs/benchmarks/spark-datafusion-comet-benchmark">TPC-DS
@ 3TB running in EKS</a>.</p>
-<p>AWS Labs also published performance results <a
href="https://awslabs.github.io/data-on-eks/docs/benchmarks/spark-gluten-velox-comet-benchmark">comparing
Comet with Apache Gluten</a>, concluding that "Gluten + Velox v1.6.0 and
DataFusion Comet v0.16.0 deliver similar overall runtime". Comet's
documentation also has a guide explaining how <a
href="https://datafusion.apache.org/comet/about/gluten_comparison.html">Comet
compares to Gluten</a>.</p>
<h3 id="codegen-dispatch">Codegen Dispatch<a class="headerlink"
href="#codegen-dispatch" title="Permanent link">¶</a></h3>
<p>A significant innovation that landed in 0.17.0 was to change the
approach to "native" acceleration. Rather than falling back to Spark row-based
execution whenever Comet lacked a native Rust implementation of an expression,
Comet is now able to execute Spark's own expression evaluation logic directly
against Arrow data in the Comet pipeline. This immediately expanded the number
of expressions that Comet could support without an expensive transition from
columnar to row-based data [...]
-<p>Another advantage of this approch is that Comet can support certain
categories of expression, such as regular expressions, with 100% compatibility
with Spark, which is not practical when delegating to native code due to the
many differences between Java's regular expression engine and those available
in Rust or C++.</p>
+<p>Another advantage of this approach is that Comet can support certain
categories of expression, such as regular expressions, with 100% compatibility
with Spark, which is not practical when delegating to native code due to the
many differences between Java's regular expression engine and those available
in Rust or C++.</p>
<h2 id="scala-and-java-udf-support">Scala and Java UDF Support<a
class="headerlink" href="#scala-and-java-udf-support" title="Permanent
link">¶</a></h2>
<p>The codegen dispatch approach described in the previous
chapter</p>
<h2 id="improvements-since-0171">Improvements since 0.17.1<a
class="headerlink" href="#improvements-since-0171" title="Permanent
link">¶</a></h2>
@@ -80,8 +79,8 @@ exists. This is an early-stage feature and we welcome
feedback from users experi
<p>This release expands the set of Spark expressions and aggregates that
are accelerated by Comet:</p>
<ul>
<li><strong>Aggregates</strong>:
<code>approx_percentile</code> /
<code>percentile_approx</code>, exact
<code>percentile</code> / <code>median</code>,
- <code>approx_count_distinct</code>, and native
<code>collect_list</code> / <code>array_agg</code>.
- <strong>Cast</strong>: Cast expressions where the native
implementation is marked as incompatible or unsupported
+ <code>approx_count_distinct</code>, and native
<code>collect_list</code> /
<code>array_agg</code>.</li>
+<li><strong>Cast</strong>: Cast expressions where the native
implementation is marked as incompatible or unsupported
are now routed through codegen dispatch.</li>
<li><strong>Grouping</strong>:
<code>grouping()</code> and
<code>grouping_id()</code>.</li>
<li><strong>Intervals</strong>: interval types via
<code>make_ym_interval</code> and
<code>make_dt_interval</code>,
<code>CalendarIntervalType</code>,
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
