This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-staging
in repository https://gitbox.apache.org/repos/asf/datafusion-site.git
The following commit(s) were added to refs/heads/asf-staging by this push:
new 949990d Commit build products
949990d is described below
commit 949990d25050d4ce8628fc8c68b1209fee4624b9
Author: Build Pelican (action) <[email protected]>
AuthorDate: Fri Aug 7 19:47:49 2026 +0000
Commit build products
---
blog/2026/08/07/datafusion-comet-1.0.0/index.html | 18 ++++++++++++++++--
blog/feeds/all-en.atom.xml | 18 ++++++++++++++++--
blog/feeds/blog.atom.xml | 18 ++++++++++++++++--
blog/feeds/pmc.atom.xml | 18 ++++++++++++++++--
4 files changed, 64 insertions(+), 8 deletions(-)
diff --git a/blog/2026/08/07/datafusion-comet-1.0.0/index.html
b/blog/2026/08/07/datafusion-comet-1.0.0/index.html
index b784fee..2a0325d 100644
--- a/blog/2026/08/07/datafusion-comet-1.0.0/index.html
+++ b/blog/2026/08/07/datafusion-comet-1.0.0/index.html
@@ -135,8 +135,7 @@ and Comet invokes Spark's own generated code for just the
missing expression, le
running natively. Four consequences are worth calling out.</p>
<ul>
<li><strong>Coverage.</strong> Expressions that would previously have blocked
native execution of a whole subtree are
- now supported immediately, without a Rust port. In 1.0, this pathway also
handles cast fallbacks,
- several interval expressions, and additional string and timestamp
functions.</li>
+ now supported immediately, without a Rust port.</li>
<li><strong>Compatibility.</strong> For categories where a native
reimplementation would inevitably diverge from Spark's
semantics — regular expressions being the canonical case, given the gap
between Java's regex engine and
any Rust or C++ equivalent — codegen dispatch delivers bit-for-bit Spark
parity because it <em>is</em> Spark's
@@ -150,6 +149,21 @@ running natively. Four consequences are worth calling
out.</p>
were previously disqualified from acceleration only because they contained a
UDF can now benefit as
long as the surrounding operators are supported. See the <a
href="https://datafusion.apache.org/comet/user-guide/latest/scala_java_udfs.html";>Scala
and Java UDF guide</a> for details.</li>
</ul>
+<p>Comet 1.0 widens the mechanism in three ways.</p>
+<p>The first is the biggest. An expression that opts into codegen dispatch
previously reached the dispatcher
+only when Comet reported it as <em>incompatible</em> for the given input; an
<em>unsupported</em> report still sent the
+whole subtree back to Spark. In 1.0 both support levels route through the
dispatcher, so an input that
+Spark handles and Comet's native code does not now stays inside the Comet
pipeline. This covers <code>concat</code>
+over non-string children, <code>sort_array</code> over nested arrays with
struct or null children, <code>array_intersect</code>
+over collated strings, and <code>trunc</code> / <code>date_trunc</code>
formats outside the native set.</p>
+<p>Second, casts join the same path. Cast expressions that Comet declines to
run natively — including legacy
+configuration variants such as
<code>spark.sql.legacy.castComplexTypesToString.enabled</code> — are now
dispatched
+rather than falling back. More expressions were opted in as well:
<code>concat</code> under non-<code>UTF8_BINARY</code>
+collations, <code>sort_array</code> under strict floating-point mode,
<code>multiply_dt_interval</code>, and interval dispatch
+for nested values and native shuffle.</p>
+<p>Third, the path is now visible. Comet's extended explain output reports
native versus codegen-dispatch
+coverage for a plan, so you can see which path each expression actually took
rather than inferring it from
+the absence of a fallback reason.</p>
<h2 id="improvements-since-0170">Improvements since 0.17.0<a
class="headerlink" href="#improvements-since-0170" title="Permanent
link">¶</a></h2>
<p>The rest of this post covers what is new since the 0.17.0 release.</p>
<h3 id="experimental-pyarrow-udf-support">Experimental PyArrow UDF Support<a
class="headerlink" href="#experimental-pyarrow-udf-support" title="Permanent
link">¶</a></h3>
diff --git a/blog/feeds/all-en.atom.xml b/blog/feeds/all-en.atom.xml
index 9ec3bd3..5480347 100644
--- a/blog/feeds/all-en.atom.xml
+++ b/blog/feeds/all-en.atom.xml
@@ -80,8 +80,7 @@ and Comet invokes Spark's own generated code for just the
missing expression, le
running natively. Four consequences are worth calling out.</p>
<ul>
<li><strong>Coverage.</strong> Expressions that would
previously have blocked native execution of a whole subtree are
- now supported immediately, without a Rust port. In 1.0, this pathway also
handles cast fallbacks,
- several interval expressions, and additional string and timestamp
functions.</li>
+ now supported immediately, without a Rust port.</li>
<li><strong>Compatibility.</strong> For categories where a
native reimplementation would inevitably diverge from Spark's
semantics — regular expressions being the canonical case, given the gap
between Java's regex engine and
any Rust or C++ equivalent — codegen dispatch delivers bit-for-bit Spark
parity because it <em>is</em> Spark's
@@ -95,6 +94,21 @@ running natively. Four consequences are worth calling
out.</p>
were previously disqualified from acceleration only because they contained a
UDF can now benefit as
long as the surrounding operators are supported. See the <a
href="https://datafusion.apache.org/comet/user-guide/latest/scala_java_udfs.html">Scala
and Java UDF guide</a> for details.</li>
</ul>
+<p>Comet 1.0 widens the mechanism in three ways.</p>
+<p>The first is the biggest. An expression that opts into codegen
dispatch previously reached the dispatcher
+only when Comet reported it as <em>incompatible</em> for the given
input; an <em>unsupported</em> report still sent the
+whole subtree back to Spark. In 1.0 both support levels route through the
dispatcher, so an input that
+Spark handles and Comet's native code does not now stays inside the Comet
pipeline. This covers <code>concat</code>
+over non-string children, <code>sort_array</code> over nested
arrays with struct or null children, <code>array_intersect</code>
+over collated strings, and <code>trunc</code> /
<code>date_trunc</code> formats outside the native set.</p>
+<p>Second, casts join the same path. Cast expressions that Comet
declines to run natively — including legacy
+configuration variants such as
<code>spark.sql.legacy.castComplexTypesToString.enabled</code> —
are now dispatched
+rather than falling back. More expressions were opted in as well:
<code>concat</code> under non-<code>UTF8_BINARY</code>
+collations, <code>sort_array</code> under strict floating-point
mode, <code>multiply_dt_interval</code>, and interval dispatch
+for nested values and native shuffle.</p>
+<p>Third, the path is now visible. Comet's extended explain output
reports native versus codegen-dispatch
+coverage for a plan, so you can see which path each expression actually took
rather than inferring it from
+the absence of a fallback reason.</p>
<h2 id="improvements-since-0170">Improvements since 0.17.0<a
class="headerlink" href="#improvements-since-0170" title="Permanent
link">¶</a></h2>
<p>The rest of this post covers what is new since the 0.17.0
release.</p>
<h3 id="experimental-pyarrow-udf-support">Experimental PyArrow UDF
Support<a class="headerlink" href="#experimental-pyarrow-udf-support"
title="Permanent link">¶</a></h3>
diff --git a/blog/feeds/blog.atom.xml b/blog/feeds/blog.atom.xml
index 4040a15..da253ea 100644
--- a/blog/feeds/blog.atom.xml
+++ b/blog/feeds/blog.atom.xml
@@ -80,8 +80,7 @@ and Comet invokes Spark's own generated code for just the
missing expression, le
running natively. Four consequences are worth calling out.</p>
<ul>
<li><strong>Coverage.</strong> Expressions that would
previously have blocked native execution of a whole subtree are
- now supported immediately, without a Rust port. In 1.0, this pathway also
handles cast fallbacks,
- several interval expressions, and additional string and timestamp
functions.</li>
+ now supported immediately, without a Rust port.</li>
<li><strong>Compatibility.</strong> For categories where a
native reimplementation would inevitably diverge from Spark's
semantics — regular expressions being the canonical case, given the gap
between Java's regex engine and
any Rust or C++ equivalent — codegen dispatch delivers bit-for-bit Spark
parity because it <em>is</em> Spark's
@@ -95,6 +94,21 @@ running natively. Four consequences are worth calling
out.</p>
were previously disqualified from acceleration only because they contained a
UDF can now benefit as
long as the surrounding operators are supported. See the <a
href="https://datafusion.apache.org/comet/user-guide/latest/scala_java_udfs.html">Scala
and Java UDF guide</a> for details.</li>
</ul>
+<p>Comet 1.0 widens the mechanism in three ways.</p>
+<p>The first is the biggest. An expression that opts into codegen
dispatch previously reached the dispatcher
+only when Comet reported it as <em>incompatible</em> for the given
input; an <em>unsupported</em> report still sent the
+whole subtree back to Spark. In 1.0 both support levels route through the
dispatcher, so an input that
+Spark handles and Comet's native code does not now stays inside the Comet
pipeline. This covers <code>concat</code>
+over non-string children, <code>sort_array</code> over nested
arrays with struct or null children, <code>array_intersect</code>
+over collated strings, and <code>trunc</code> /
<code>date_trunc</code> formats outside the native set.</p>
+<p>Second, casts join the same path. Cast expressions that Comet
declines to run natively — including legacy
+configuration variants such as
<code>spark.sql.legacy.castComplexTypesToString.enabled</code> —
are now dispatched
+rather than falling back. More expressions were opted in as well:
<code>concat</code> under non-<code>UTF8_BINARY</code>
+collations, <code>sort_array</code> under strict floating-point
mode, <code>multiply_dt_interval</code>, and interval dispatch
+for nested values and native shuffle.</p>
+<p>Third, the path is now visible. Comet's extended explain output
reports native versus codegen-dispatch
+coverage for a plan, so you can see which path each expression actually took
rather than inferring it from
+the absence of a fallback reason.</p>
<h2 id="improvements-since-0170">Improvements since 0.17.0<a
class="headerlink" href="#improvements-since-0170" title="Permanent
link">¶</a></h2>
<p>The rest of this post covers what is new since the 0.17.0
release.</p>
<h3 id="experimental-pyarrow-udf-support">Experimental PyArrow UDF
Support<a class="headerlink" href="#experimental-pyarrow-udf-support"
title="Permanent link">¶</a></h3>
diff --git a/blog/feeds/pmc.atom.xml b/blog/feeds/pmc.atom.xml
index 985c371..908d6f5 100644
--- a/blog/feeds/pmc.atom.xml
+++ b/blog/feeds/pmc.atom.xml
@@ -80,8 +80,7 @@ and Comet invokes Spark's own generated code for just the
missing expression, le
running natively. Four consequences are worth calling out.</p>
<ul>
<li><strong>Coverage.</strong> Expressions that would
previously have blocked native execution of a whole subtree are
- now supported immediately, without a Rust port. In 1.0, this pathway also
handles cast fallbacks,
- several interval expressions, and additional string and timestamp
functions.</li>
+ now supported immediately, without a Rust port.</li>
<li><strong>Compatibility.</strong> For categories where a
native reimplementation would inevitably diverge from Spark's
semantics — regular expressions being the canonical case, given the gap
between Java's regex engine and
any Rust or C++ equivalent — codegen dispatch delivers bit-for-bit Spark
parity because it <em>is</em> Spark's
@@ -95,6 +94,21 @@ running natively. Four consequences are worth calling
out.</p>
were previously disqualified from acceleration only because they contained a
UDF can now benefit as
long as the surrounding operators are supported. See the <a
href="https://datafusion.apache.org/comet/user-guide/latest/scala_java_udfs.html">Scala
and Java UDF guide</a> for details.</li>
</ul>
+<p>Comet 1.0 widens the mechanism in three ways.</p>
+<p>The first is the biggest. An expression that opts into codegen
dispatch previously reached the dispatcher
+only when Comet reported it as <em>incompatible</em> for the given
input; an <em>unsupported</em> report still sent the
+whole subtree back to Spark. In 1.0 both support levels route through the
dispatcher, so an input that
+Spark handles and Comet's native code does not now stays inside the Comet
pipeline. This covers <code>concat</code>
+over non-string children, <code>sort_array</code> over nested
arrays with struct or null children, <code>array_intersect</code>
+over collated strings, and <code>trunc</code> /
<code>date_trunc</code> formats outside the native set.</p>
+<p>Second, casts join the same path. Cast expressions that Comet
declines to run natively — including legacy
+configuration variants such as
<code>spark.sql.legacy.castComplexTypesToString.enabled</code> —
are now dispatched
+rather than falling back. More expressions were opted in as well:
<code>concat</code> under non-<code>UTF8_BINARY</code>
+collations, <code>sort_array</code> under strict floating-point
mode, <code>multiply_dt_interval</code>, and interval dispatch
+for nested values and native shuffle.</p>
+<p>Third, the path is now visible. Comet's extended explain output
reports native versus codegen-dispatch
+coverage for a plan, so you can see which path each expression actually took
rather than inferring it from
+the absence of a fallback reason.</p>
<h2 id="improvements-since-0170">Improvements since 0.17.0<a
class="headerlink" href="#improvements-since-0170" title="Permanent
link">¶</a></h2>
<p>The rest of this post covers what is new since the 0.17.0
release.</p>
<h3 id="experimental-pyarrow-udf-support">Experimental PyArrow UDF
Support<a class="headerlink" href="#experimental-pyarrow-udf-support"
title="Permanent link">¶</a></h3>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
