This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-staging
in repository https://gitbox.apache.org/repos/asf/datafusion-site.git
The following commit(s) were added to refs/heads/asf-staging by this push:
new ddc6b82 Commit build products
ddc6b82 is described below
commit ddc6b822f1348483d680f4469caab2b54578d0a7
Author: Build Pelican (action) <[email protected]>
AuthorDate: Sun Aug 9 14:43:43 2026 +0000
Commit build products
---
blog/2026/08/07/datafusion-comet-1.0.0/index.html | 318 ---------------------
.../08/09/datafusion-ballista-54.1.0/index.html | 201 +++++++++++++
blog/author/pmc.html | 12 +-
blog/category/blog.html | 12 +-
blog/feed.xml | 10 +-
blog/feeds/all-en.atom.xml | 193 ++++---------
blog/feeds/blog.atom.xml | 193 ++++---------
blog/feeds/pmc.atom.xml | 203 ++++---------
blog/feeds/pmc.rss.xml | 10 +-
blog/index.html | 14 +-
10 files changed, 388 insertions(+), 778 deletions(-)
diff --git a/blog/2026/08/07/datafusion-comet-1.0.0/index.html
b/blog/2026/08/07/datafusion-comet-1.0.0/index.html
deleted file mode 100644
index a909d02..0000000
--- a/blog/2026/08/07/datafusion-comet-1.0.0/index.html
+++ /dev/null
@@ -1,318 +0,0 @@
-<!doctype html>
-<html class="no-js" lang="en" dir="ltr">
- <head>
- <meta charset="utf-8">
- <meta http-equiv="x-ua-compatible" content="ie=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>Apache DataFusion Comet 1.0.0 Release - Apache DataFusion
Blog</title>
-<link href="/blog/favicon.ico" rel="icon" type="image/x-icon">
-<link href="/blog/favicon.ico" rel="shortcut icon" type="image/x-icon">
-<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
-<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
-<link href="/blog/css/headerlink.css" rel="stylesheet">
-<link href="/blog/highlight/default.min.css" rel="stylesheet">
-<link href="/blog/css/app.css" rel="stylesheet">
-<link href="/blog/css/dark-mode.css" rel="stylesheet">
-<script src="/blog/js/dark-mode.js"></script>
-<script src="/blog/highlight/highlight.js"></script>
-<script>hljs.highlightAll();</script> </head>
- <body class="d-flex flex-column h-100">
- <main class="flex-shrink-0">
-<!-- nav bar -->
-<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth
navbar example">
- <div class="container-fluid">
- <a class="navbar-brand" href="/blog/"><img
src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache
DataFusion Blog</a>
- <button class="navbar-toggler" type="button" data-bs-toggle="collapse"
data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false"
aria-label="Toggle navigation">
- <span class="navbar-toggler-icon"></span>
- </button>
-
- <div class="collapse navbar-collapse" id="navbarADP">
- <ul class="navbar-nav me-auto mb-2 mb-lg-0">
- <li class="nav-item">
- <a class="nav-link" href="/blog/about.html">About</a>
- </li>
- <li class="nav-item">
- <a class="nav-link" href="/blog/feed.xml">RSS</a>
- </li>
- </ul>
- <button id="dark-mode-toggle" type="button"
class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false"
title="Toggle dark mode">
- <span class="sun-icon" aria-hidden="true">☀</span>
- <span class="moon-icon" aria-hidden="true">☾</span>
- </button>
- </div>
- </div>
-</nav>
-<!-- article contents -->
-<div id="contents">
- <div class="bg-white p-4 p-md-5 rounded">
- <div class="row justify-content-center">
- <div class="col-12 col-md-8 main-content">
- <h1>
- Apache DataFusion Comet 1.0.0 Release
- </h1>
- <p>Posted on: Fri 07 August 2026 by pmc</p>
-
- <aside class="toc-container d-md-none mb-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#the-road-to-10">The Road to 1.0</a><ul>
-<li><a href="#support-for-spark-40-with-ansi-mode">Support for Spark 4.0+ with
ANSI mode</a></li>
-<li><a href="#correctness-testing">Correctness Testing</a></li>
-<li><a href="#performance">Performance</a></li>
-<li><a href="#codegen-dispatch">Codegen Dispatch</a></li>
-</ul>
-</li>
-<li><a href="#improvements-since-0170">Improvements since 0.17.0</a><ul>
-<li><a href="#experimental-pyarrow-udf-support">Experimental PyArrow UDF
Support</a></li>
-<li><a href="#new-expression-and-aggregate-support">New Expression and
Aggregate Support</a></li>
-<li><a href="#faster-parquet-scans">Faster Parquet Scans</a></li>
-<li><a href="#iceberg-table-format-v3">Iceberg Table Format V3</a></li>
-<li><a href="#native-expression-performance">Native Expression
Performance</a></li>
-</ul>
-</li>
-<li><a href="#deprecation-notice">Deprecation Notice</a></li>
-<li><a href="#compatibility">Compatibility</a></li>
-<li><a href="#get-started-with-comet-100">Get Started with Comet 1.0.0</a></li>
-</ul>
-</div>
- </aside>
-
- <!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-
-<p>The Apache DataFusion PMC is pleased to announce version 1.0.0 of the <a
href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
-<p>Comet is an accelerator for Apache Spark that translates Spark physical
plans to DataFusion physical plans for
-improved performance and efficiency without requiring any code changes.</p>
-<p>This release covers roughly six weeks of development since 0.17.0 and
consists of 244 commits from 23
-contributors. See the <a
href="https://github.com/apache/datafusion-comet/blob/main/docs/source/changelog/1.0.0.md">change
log</a> for the full list of changes.</p>
-<h2 id="the-road-to-10">The Road to 1.0<a class="headerlink"
href="#the-road-to-10" title="Permanent link">¶</a></h2>
-<p>Comet was <a
href="https://datafusion.apache.org/blog/2024/03/06/comet-donation/">donated</a>
to the Apache DataFusion project in March 2024 and cut its first release,
0.1.0, five
-months later with support for 13 operators and 106 expressions. Since then,
the project has shipped 20
-releases and drawn contributions from more than 120 developers, and the
codebase now recognizes over 400
-Spark expressions. Operator coverage has grown alongside it: 1.0 accelerates
each of Spark's four join
-operators, window functions, generators (<code>explode</code>,
<code>explode_outer</code>, <code>posexplode</code>, and
<code>posexplode_outer</code>
-over arrays), sampling, in-memory table scans, and a fully native shuffle.</p>
-<p>The 1.0 release marks the point at which Comet begins following <a
href="https://datafusion.apache.org/comet/about/versioning_policy.html">semantic
versioning</a>. Users upgrading
-within the 1.x line can expect backward-compatible changes only; features
slated for removal will be
-deprecated in a minor release before being dropped in the next major version.
This is why the deprecations
-of JDK 11 and Spark 3.4 announced below are scheduled for 1.1 rather than
landing in 1.0 itself.</p>
-<h3 id="support-for-spark-40-with-ansi-mode">Support for Spark 4.0+ with ANSI
mode<a class="headerlink" href="#support-for-spark-40-with-ansi-mode"
title="Permanent link">¶</a></h3>
-<p>Comet 1.0.0 supports Spark versions 3.4 through 4.1, with experimental
support for 4.2. Comet fully supports Spark's ANSI mode, which is enabled by
default starting with Spark 4.0.</p>
-<h3 id="correctness-testing">Correctness Testing<a class="headerlink"
href="#correctness-testing" title="Permanent link">¶</a></h3>
-<p>It is important that queries accelerated by Comet produce the same results
as Spark. Correctness checking has always been a large effort in Comet
development, but the approach has evolved over time.</p>
-<ul>
-<li><strong>Upstream Spark tests</strong>: Comet runs Spark's own test suite
with Comet enabled, providing more than 24,000 unit tests effectively for free.
These tests run in Comet's CI for all supported Spark versions.</li>
-<li><strong>Scala tests</strong>: end-to-end queries that run with Comet
enabled versus disabled, checking that results match.</li>
-<li><strong>Fuzz testing</strong>: many of the Scala tests generate randomized
data to catch regressions around edge cases such as nulls, NaN, Infinity, and
timezone issues.</li>
-<li><strong>Comet SQL tests</strong>: a sqllogictest-inspired approach that
makes end-to-end tests easier to write.</li>
-<li><strong>Generative AI audits</strong>: agentic skills sweep every
expression, comparing Comet's implementation to Spark's source and ensuring
tests cover important edge cases.</li>
-</ul>
-<h3 id="performance">Performance<a class="headerlink" href="#performance"
title="Permanent link">¶</a></h3>
-<p>The early Comet releases provided a very modest speedup and the published
benchmark results were based on running TPC workloads at small scale factors on
a single node. There are now independent benchmark results published by AWS
Labs that show significant speedups for <a
href="https://awslabs.github.io/data-on-eks/docs/benchmarks/spark-datafusion-comet-benchmark">TPC-DS
@ 3TB running in EKS</a>.</p>
-<h3 id="codegen-dispatch">Codegen Dispatch<a class="headerlink"
href="#codegen-dispatch" title="Permanent link">¶</a></h3>
-<p>Comet 0.17.0 introduced a new approach to filling gaps in expression
coverage. In earlier releases,
-whenever Comet's planner encountered an expression that lacked a native Rust
implementation, it fell back
-to executing an entire subtree of the plan in Spark. That required converting
Arrow columns back to Spark
-rows before the expression ran and back to Arrow after, and the cost was often
enough to erase the speedup
-Comet had bought elsewhere in the plan.</p>
-<p>Codegen dispatch narrows that fallback to the expression itself: the batch
stays in the Comet pipeline
-and Comet invokes Spark's own generated code for just the missing expression,
leaving the rest of the query
-running natively. Four consequences are worth calling out.</p>
-<ul>
-<li><strong>Coverage.</strong> Expressions that would previously have blocked
native execution of a whole subtree are
- now supported immediately, without a Rust port.</li>
-<li><strong>Compatibility.</strong> For categories where a native
reimplementation would inevitably diverge from Spark's
- semantics — regular expressions being the canonical case, given the gap
between Java's regex engine and
- any Rust or C++ equivalent — codegen dispatch delivers bit-for-bit Spark
parity because it <em>is</em> Spark's
- implementation.</li>
-<li><strong>Expression fusion.</strong> A dispatched expression tree
(<em>i.e.</em>, nested expressions) is compiled into a single method, so the
Arrow
- input reads, the expression evaluation, and the Arrow output writes are
fused together. The compiler
- is free to optimize across the whole tree, and no intermediate Arrow
<code>RecordBatch</code> is materialized
- between one expression and the next.</li>
-<li><strong>Scala and Java UDFs.</strong> User-defined functions are compiled
to the same codegen surface as built-in
- expressions, so they can flow through codegen dispatch without any change
from the user. Queries that
- were previously disqualified from acceleration only because they contained a
UDF can now benefit as
- long as the surrounding operators are supported. See the <a
href="https://datafusion.apache.org/comet/user-guide/latest/scala_java_udfs.html">Scala
and Java UDF guide</a> for details.</li>
-</ul>
-<p>Comet 1.0 widens the mechanism in three ways.</p>
-<p>The first is the biggest. An expression that opts into codegen dispatch
previously reached the dispatcher
-only when Comet reported it as <em>incompatible</em> for the given input; an
<em>unsupported</em> report still sent the
-whole subtree back to Spark. In 1.0 both support levels route through the
dispatcher, so an input that
-Spark handles and Comet's native code does not now stays inside the Comet
pipeline.</p>
-<p>Second, casts join the same path. Cast expressions that Comet declines to
run natively — including legacy
-configuration variants such as
<code>spark.sql.legacy.castComplexTypesToString.enabled</code> — are now
dispatched
-rather than falling back, and more string, array, and interval expressions
were opted in as well.</p>
-<p>Third, the path is now visible. Comet's extended explain output reports
native versus codegen-dispatch
-coverage for a plan, so you can see which path each expression actually took
rather than inferring it from
-the absence of a fallback reason.</p>
-<h2 id="improvements-since-0170">Improvements since 0.17.0<a
class="headerlink" href="#improvements-since-0170" title="Permanent
link">¶</a></h2>
-<p>The rest of this post covers what is new since the 0.17.0 release.</p>
-<h3 id="experimental-pyarrow-udf-support">Experimental PyArrow UDF Support<a
class="headerlink" href="#experimental-pyarrow-udf-support" title="Permanent
link">¶</a></h3>
-<p>This release adds experimental support for accelerated PyArrow UDFs,
allowing PyArrow-based user-defined
-functions to participate in native execution instead of forcing a fallback to
Spark. When the feature is
-disabled, Comet now hints at the native PyArrow UDF path in its fallback
reasons so users know the option
-exists. This is an early-stage feature and we welcome feedback from users
experimenting with it.</p>
-<h3 id="new-expression-and-aggregate-support">New Expression and Aggregate
Support<a class="headerlink" href="#new-expression-and-aggregate-support"
title="Permanent link">¶</a></h3>
-<p>This release expands the set of Spark expressions and aggregates that are
accelerated by Comet:</p>
-<ul>
-<li><strong>Aggregates</strong>: <code>approx_percentile</code> /
<code>percentile_approx</code>, exact <code>percentile</code> /
<code>median</code>,
- <code>approx_count_distinct</code>, and native <code>collect_list</code> /
<code>array_agg</code>.</li>
-<li><strong>Cast</strong>: Cast expressions where the native implementation is
marked as incompatible or unsupported
- are now routed through codegen dispatch.</li>
-<li><strong>Grouping</strong>: <code>grouping()</code> and
<code>grouping_id()</code>.</li>
-<li><strong>Intervals</strong>: interval types via
<code>make_ym_interval</code> and <code>make_dt_interval</code>,
<code>CalendarIntervalType</code>,
- <code>multiply_dt_interval</code>, and interval codegen dispatch for nested
values and native shuffle.</li>
-<li><strong>String</strong>: <code>base64</code>, <code>split_part</code> via
<code>StringSplitSQL</code>, native <code>levenshtein</code>, and native
<code>randstr</code> and
- <code>uuid</code> — both bit-for-bit compatible with Spark for a given
seed.</li>
-<li><strong>Array / map</strong>: <code>array_prepend</code>, the
<code>shuffle()</code> array function, <code>size()</code> for
<code>MapType</code>, and <code>ElementAt</code>
- over <code>MapType</code>.</li>
-<li><strong>Date/time</strong>: native <code>TimestampNTZ</code> inputs for
<code>hour</code> / <code>minute</code> / <code>second</code> and
- <code>PreciseTimestampConversion</code> for native time-window grouping.</li>
-<li><strong>Windows</strong>: extended native window function support and
Spark 4 decimal window average.</li>
-</ul>
-<h3 id="faster-parquet-scans">Faster Parquet Scans<a class="headerlink"
href="#faster-parquet-scans" title="Permanent link">¶</a></h3>
-<p>Parquet reads pick up several improvements as well. Full Parquet metadata,
including the page index, is now
-cached via DataFusion's <code>CachedParquetFileReaderFactory</code>; identity
casts are unwrapped in the schema adapter
-so Parquet statistics pruning can engage; filter pushdown configuration has
been revised; the native scan
-passes a metadata size hint so a single read usually captures the footer; and
the native Parquet scan seeds
-its reader options from the session config so Parquet settings you already set
take effect.</p>
-<h3 id="iceberg-table-format-v3">Iceberg Table Format V3<a class="headerlink"
href="#iceberg-table-format-v3" title="Permanent link">¶</a></h3>
-<p>Comet now supports Iceberg 1.11 and its first Iceberg table format V3
feature: full table encryption. Other V3 features like deletion vectors and new
data types (<em>e.g.</em>, <code>VARIANT</code>) fall back gracefully. The
native Iceberg scan supports the <code>_pos</code>, <code>_spec</code>,
<code>_file</code>, and
-<code>_partition</code> metadata columns, sizes delete files correctly to
avoid dropped deletes, disambiguates scans that
-share a <code>metadata_location</code>, and dedupes residuals and delete files
in the native scan serde. A prior case
-where Iceberg native scan exchange reuse with different pushed filters could
produce wrong results is also
-fixed.</p>
-<h3 id="native-expression-performance">Native Expression Performance<a
class="headerlink" href="#native-expression-performance" title="Permanent
link">¶</a></h3>
-<p>Many native expression implementations have been optimized to more
efficiently leverage Arrow kernels or to avoid per-row builders.</p>
-<ul>
-<li><strong>Casts</strong> between numeric, string, decimal, and date types,
including a faster float-to-decimal cast, an
- optimized integer-to-integer cast, shared no-overflow fast paths in
<code>CheckOverflow</code> and
- <code>DecimalRescaleCheckOverflow</code>, and a
<code>cast_binary_to_string</code> that is up to 27x faster on binary-format
- styles.</li>
-<li><strong>JSON, regex, and URL parsing</strong>:
<code>get_json_object</code>, <code>regexp_extract</code>, and
<code>parse_url</code>.</li>
-<li><strong>Date/time and decimal kernels</strong>: <code>date_trunc</code>,
<code>spark_ceil</code>, and a vectorized
<code>spark_unscaled_value</code>.</li>
-<li><strong>String and array kernels</strong>: <code>lpad</code>,
<code>unhex</code>, <code>size</code>, <code>arrays_overlap</code>,
<code>escape_string</code>, and the <code>try_*</code>
- arithmetic kernel.</li>
-</ul>
-<p>To make this kind of work repeatable, the release also adds a <a
href="https://datafusion.apache.org/comet/contributor-guide/optimizing_expressions.html">scalar
expression optimization guide</a>
-documenting how to benchmark a kernel, keep its output bit-identical to Spark,
and gate changes on a
-no-regression check.</p>
-<h2 id="deprecation-notice">Deprecation Notice<a class="headerlink"
href="#deprecation-notice" title="Permanent link">¶</a></h2>
-<p>With the move to a stable 1.0 release line, Comet begins deprecating older
platforms under its
-<a
href="https://datafusion.apache.org/comet/about/versioning_policy.html">versioning
policy</a>:</p>
-<ul>
-<li><strong>JDK 11</strong> is deprecated and scheduled for removal in Comet
1.1.0.</li>
-<li><strong>Apache Spark 3.4</strong> is deprecated and scheduled for removal
in Comet 1.1.0.</li>
-</ul>
-<p>Comet aligns its Spark support window with upstream Apache Spark
maintenance. Spark 3.4 is no longer
-maintained upstream, so under the versioning policy it is deprecated in the
first Comet minor release
-after that point and removed in the following one. Comet 1.0.0 still builds
and publishes Spark 3.4
-binaries.</p>
-<p>Users on these platforms should plan to move to JDK 17+ and Spark 3.5 or
later before upgrading to 1.1.0.</p>
-<h2 id="compatibility">Compatibility<a class="headerlink"
href="#compatibility" title="Permanent link">¶</a></h2>
-<p>Supported platforms include:</p>
-<ul>
-<li><strong>Spark 3.4.3</strong> with Java 11/17 and Scala 2.12/2.13
(deprecated, removal in 1.1.0)</li>
-<li><strong>Spark 3.5.9</strong> with Java 11/17 and Scala 2.12/2.13</li>
-<li><strong>Spark 4.0.4</strong> with Java 17 and Scala 2.13</li>
-<li><strong>Spark 4.1.3</strong> with Java 17/21 and Scala 2.13</li>
-<li><strong>Spark 4.2</strong> with Java 17 and Scala 2.13 (experimental, for
early evaluation only)</li>
-</ul>
-<p>See the <a
href="https://datafusion.apache.org/comet/user-guide/latest/compatibility/spark-versions.html">Spark
Version Compatibility</a> page for known limitations specific to each
version.</p>
-<p>This release upgrades to <strong>DataFusion 54.1</strong> and <strong>Arrow
58.4</strong>.</p>
-<h2 id="get-started-with-comet-100">Get Started with Comet 1.0.0<a
class="headerlink" href="#get-started-with-comet-100" title="Permanent
link">¶</a></h2>
-<p>Ready to try it out? Follow the <a
href="https://datafusion.apache.org/comet/user-guide/1.0/installation.html">Comet
1.0.0 Installation Guide</a>
-to get up and running, then point Comet at your existing Spark workloads and
see the speedup for yourself.</p>
-
-<!--
- Comments Section
- Loaded only after explicit visitor consent to comply with ASF policy.
--->
-
-<div>
- <hr>
- <h3 id="comments">Comments<a class="headerlink" href="#comments"
title="Permanent link">¶</a></h3>
-
- <!-- Local loader script -->
- <script src="/blog/js/giscus-consent.js" defer></script>
-
- <!-- Consent UI -->
- <div id="giscus-consent">
- <p>
- We use <a href="https://giscus.app/">Giscus</a> for comments, powered
by GitHub Discussions.
- To respect your privacy, Giscus and comments will load only if you
click "Show Comments"
- </p>
-
- <div class="consent-actions">
- <button id="giscus-load" type="button">Show Comments</button>
- <button id="giscus-revoke" type="button" hidden>Hide Comments</button>
- </div>
-
- <noscript>JavaScript is required to load comments from Giscus.</noscript>
- </div>
-
- <!-- Container where Giscus will render -->
- <div id="comment-thread"></div>
-</div> </div>
- <aside class="toc-container d-none d-md-block col-md-4 col-xl-3 ms-xl-2">
- <div class="toc"><span class="toctitle">Contents</span><ul>
-<li><a href="#the-road-to-10">The Road to 1.0</a><ul>
-<li><a href="#support-for-spark-40-with-ansi-mode">Support for Spark 4.0+ with
ANSI mode</a></li>
-<li><a href="#correctness-testing">Correctness Testing</a></li>
-<li><a href="#performance">Performance</a></li>
-<li><a href="#codegen-dispatch">Codegen Dispatch</a></li>
-</ul>
-</li>
-<li><a href="#improvements-since-0170">Improvements since 0.17.0</a><ul>
-<li><a href="#experimental-pyarrow-udf-support">Experimental PyArrow UDF
Support</a></li>
-<li><a href="#new-expression-and-aggregate-support">New Expression and
Aggregate Support</a></li>
-<li><a href="#faster-parquet-scans">Faster Parquet Scans</a></li>
-<li><a href="#iceberg-table-format-v3">Iceberg Table Format V3</a></li>
-<li><a href="#native-expression-performance">Native Expression
Performance</a></li>
-</ul>
-</li>
-<li><a href="#deprecation-notice">Deprecation Notice</a></li>
-<li><a href="#compatibility">Compatibility</a></li>
-<li><a href="#get-started-with-comet-100">Get Started with Comet 1.0.0</a></li>
-</ul>
-</div>
- </aside>
- </div>
- </div>
-</div>
- <!-- footer -->
- <div class="row g-0">
- <div class="col-12">
- <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
- Copyright 2026, <a href="https://www.apache.org/">The Apache
Software Foundation</a>, Licensed under the <a
href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version
2.0</a>.<br/>
- Apache® and the Apache logo are trademarks of The Apache
Software Foundation.
- </p>
- <p style="font-size: 0.8rem; text-align: center;">
- <a href="https://www.apache.org/">Foundation</a> |
- <a
href="https://www.apache.org/events/current-event.html">Events</a> |
- <a href="https://www.apache.org/licenses/">License</a> |
- <a href="https://www.apache.org/security/">Security</a> |
- <a
href="https://www.apache.org/foundation/sponsorship.html">Sponsorship</a> |
- <a href="https://www.apache.org/foundation/thanks.html">Thanks</a> |
- <a
href="https://privacy.apache.org/policies/privacy-policy-public.html">Privacy</a>
- </p>
- </div>
- </div>
- <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
- </body>
-</html>
diff --git a/blog/2026/08/09/datafusion-ballista-54.1.0/index.html
b/blog/2026/08/09/datafusion-ballista-54.1.0/index.html
new file mode 100644
index 0000000..ce0fe81
--- /dev/null
+++ b/blog/2026/08/09/datafusion-ballista-54.1.0/index.html
@@ -0,0 +1,201 @@
+<!doctype html>
+<html class="no-js" lang="en" dir="ltr">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="x-ua-compatible" content="ie=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+ <title>Apache DataFusion Ballista 54.1.0 Released - Apache DataFusion
Blog</title>
+<link href="/blog/favicon.ico" rel="icon" type="image/x-icon">
+<link href="/blog/favicon.ico" rel="shortcut icon" type="image/x-icon">
+<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
+<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
+<link href="/blog/css/headerlink.css" rel="stylesheet">
+<link href="/blog/highlight/default.min.css" rel="stylesheet">
+<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
+<script src="/blog/highlight/highlight.js"></script>
+<script>hljs.highlightAll();</script> </head>
+ <body class="d-flex flex-column h-100">
+ <main class="flex-shrink-0">
+<!-- nav bar -->
+<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth
navbar example">
+ <div class="container-fluid">
+ <a class="navbar-brand" href="/blog/"><img
src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache
DataFusion Blog</a>
+ <button class="navbar-toggler" type="button" data-bs-toggle="collapse"
data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false"
aria-label="Toggle navigation">
+ <span class="navbar-toggler-icon"></span>
+ </button>
+
+ <div class="collapse navbar-collapse" id="navbarADP">
+ <ul class="navbar-nav me-auto mb-2 mb-lg-0">
+ <li class="nav-item">
+ <a class="nav-link" href="/blog/about.html">About</a>
+ </li>
+ <li class="nav-item">
+ <a class="nav-link" href="/blog/feed.xml">RSS</a>
+ </li>
+ </ul>
+ <button id="dark-mode-toggle" type="button"
class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false"
title="Toggle dark mode">
+ <span class="sun-icon" aria-hidden="true">☀</span>
+ <span class="moon-icon" aria-hidden="true">☾</span>
+ </button>
+ </div>
+ </div>
+</nav>
+<!-- article contents -->
+<div id="contents">
+ <div class="bg-white p-4 p-md-5 rounded">
+ <div class="row justify-content-center">
+ <div class="col-12 col-md-8 main-content">
+ <h1>
+ Apache DataFusion Ballista 54.1.0 Released
+ </h1>
+ <p>Posted on: Sun 09 August 2026 by pmc</p>
+
+ <aside class="toc-container d-md-none mb-2">
+ <div class="toc"><span class="toctitle">Contents</span><ul>
+<li><a href="#upgrade-to-datafusion-5410">Upgrade to DataFusion 54.1.0</a></li>
+<li><a href="#cluster-stability-fixes">Cluster stability fixes</a></li>
+<li><a href="#task-retry-and-shuffle-memory">Task retry and shuffle
memory</a></li>
+<li><a href="#planning-correctness">Planning correctness</a></li>
+<li><a href="#thank-you">Thank You</a></li>
+</ul>
+</div>
+ </aside>
+
+ <!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>We are pleased to announce version <a
href="https://github.com/apache/datafusion-ballista/blob/54.1.0/docs/source/changelog/54.1.0.md">54.1.0</a>
of <a href="https://datafusion.apache.org/ballista/">Apache DataFusion
Ballista</a>. Ballista is a distributed query
+execution engine that enhances <a href="https://datafusion.apache.org">Apache
DataFusion</a> by enabling parallel execution of workloads across multiple
+nodes.</p>
+<p>This is a patch release, consisting of 10 commits from 3 contributors. It
follows <a href="/blog/2026/07/12/datafusion-ballista-54.0.0/">54.0.0</a> and
contains no
+new features and no breaking changes — only an upgrade to DataFusion 54.1.0
and a set of bug fixes
+backported from <code>main</code>. Users running 54.0.0 are encouraged to
upgrade.</p>
+<h2 id="upgrade-to-datafusion-5410">Upgrade to DataFusion 54.1.0<a
class="headerlink" href="#upgrade-to-datafusion-5410" title="Permanent
link">¶</a></h2>
+<p>Ballista 54.1.0 is built on DataFusion 54.1.0, picking up the fixes in that
patch release. No API changes
+were required, so the <a
href="https://datafusion.apache.org/ballista/upgrading/54.0.0.html">54.0.0
upgrade guide</a> still applies for anyone moving from an earlier release.</p>
+<h2 id="cluster-stability-fixes">Cluster stability fixes<a class="headerlink"
href="#cluster-stability-fixes" title="Permanent link">¶</a></h2>
+<p>Three fixes address problems that only appear in a real multi-node
deployment:</p>
+<ul>
+<li><strong>Executor client pool enabled by default.</strong> The executor's
<code>client_ttl</code> defaulted to <code>0</code>, which disabled
+ the gRPC client pool entirely. Without a pool, the shuffle-read path opened
and discarded a TCP connection
+ per fetch, so a single shuffle-heavy query could exhaust the host's
ephemeral ports and fail with
+ <code>AddrNotAvailable</code> — as would every query after it, until those
sockets left <code>TIME_WAIT</code>. A single-executor
+ cluster never showed this, because shuffle reads are served from local
files. <code>client_ttl</code> now defaults to
+ 30 seconds, so the pool is on unless explicitly disabled.</li>
+<li><strong>Executor startup race.</strong> The executor spawned its gRPC
server and registered with the scheduler
+ immediately afterwards, but the socket was not bound yet when the scheduler
dialed back. Losing that race
+ produced <code>ECONNREFUSED</code>, a failed registration, and an executor
that died at startup. The listener is now
+ bound before registration can run.</li>
+<li><strong>Executor shutdown.</strong> The shutdown notification is now sent
before the notifier is dropped, so a graceful
+ shutdown is actually observed by the components waiting on it.</li>
+</ul>
+<h2 id="task-retry-and-shuffle-memory">Task retry and shuffle memory<a
class="headerlink" href="#task-retry-and-shuffle-memory" title="Permanent
link">¶</a></h2>
+<ul>
+<li><strong>Retryable IO errors on a join's build side.</strong> A retryable
IO error raised on a join's shared build side
+ arrives at the failed-task classifier wrapped in
<code>DataFusionError::Shared</code>, so a match on the outermost
+ variant never saw the underlying <code>IoError</code> and the task was
marked non-retryable. Classification now
+ operates on the root error, regardless of wrapping.</li>
+<li><strong>Sort shuffle writer respects memory pool pressure.</strong> The
sort shuffle writer discarded the result of
+ <code>try_grow</code> on its memory reservation, so a rejected grow produced
no spill: the batch stayed buffered while
+ the shared <code>MemoryPool</code> had not accounted for it, and other
consumers saw memory as free that was not. A
+ rejected grow is now treated as a spill trigger alongside the existing
per-task budget.</li>
+</ul>
+<h2 id="planning-correctness">Planning correctness<a class="headerlink"
href="#planning-correctness" title="Permanent link">¶</a></h2>
+<ul>
+<li><strong><code>GROUP BY</code>-less aggregates are preserved</strong> when
the scheduler propagates empty stages, so a query whose
+ input produces no rows still returns the correct single-row aggregate result
rather than nothing.</li>
+<li><strong><code>EmptyExec</code> partition counts survive stage
serialization</strong>, fixing plans whose partitioning changed as a
+ stage was sent to an executor.</li>
+<li><strong>Task partitions are mapped through <code>UnionExec</code></strong>
when restricting file scans, so each task scans the
+ files it was assigned.</li>
+<li><strong><code>SortMergeJoinExec</code> broadcast conversion is now
disabled by default.</strong> The static-planner optimization
+ that broadcasts the small build side of a sort-merge join, added in 54.0.0,
is opt-in in this release while
+ outstanding issues are resolved.</li>
+</ul>
+<h2 id="thank-you">Thank You<a class="headerlink" href="#thank-you"
title="Permanent link">¶</a></h2>
+<p>Thanks to Andy Grove, Alexander Domenti, and goingforstudying-ctrl for the
commits in this release, and to
+everyone who contributed by filing issues, reviewing PRs, and providing
feedback. Full details are in the
+<a
href="https://github.com/apache/datafusion-ballista/blob/54.1.0/docs/source/changelog/54.1.0.md">changelog</a>.</p>
+
+<!--
+ Comments Section
+ Loaded only after explicit visitor consent to comply with ASF policy.
+-->
+
+<div>
+ <hr>
+ <h3 id="comments">Comments<a class="headerlink" href="#comments"
title="Permanent link">¶</a></h3>
+
+ <!-- Local loader script -->
+ <script src="/blog/js/giscus-consent.js" defer></script>
+
+ <!-- Consent UI -->
+ <div id="giscus-consent">
+ <p>
+ We use <a href="https://giscus.app/">Giscus</a> for comments, powered
by GitHub Discussions.
+ To respect your privacy, Giscus and comments will load only if you
click "Show Comments"
+ </p>
+
+ <div class="consent-actions">
+ <button id="giscus-load" type="button">Show Comments</button>
+ <button id="giscus-revoke" type="button" hidden>Hide Comments</button>
+ </div>
+
+ <noscript>JavaScript is required to load comments from Giscus.</noscript>
+ </div>
+
+ <!-- Container where Giscus will render -->
+ <div id="comment-thread"></div>
+</div> </div>
+ <aside class="toc-container d-none d-md-block col-md-4 col-xl-3 ms-xl-2">
+ <div class="toc"><span class="toctitle">Contents</span><ul>
+<li><a href="#upgrade-to-datafusion-5410">Upgrade to DataFusion 54.1.0</a></li>
+<li><a href="#cluster-stability-fixes">Cluster stability fixes</a></li>
+<li><a href="#task-retry-and-shuffle-memory">Task retry and shuffle
memory</a></li>
+<li><a href="#planning-correctness">Planning correctness</a></li>
+<li><a href="#thank-you">Thank You</a></li>
+</ul>
+</div>
+ </aside>
+ </div>
+ </div>
+</div>
+ <!-- footer -->
+ <div class="row g-0">
+ <div class="col-12">
+ <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
+ Copyright 2026, <a href="https://www.apache.org/">The Apache
Software Foundation</a>, Licensed under the <a
href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version
2.0</a>.<br/>
+ Apache® and the Apache logo are trademarks of The Apache
Software Foundation.
+ </p>
+ <p style="font-size: 0.8rem; text-align: center;">
+ <a href="https://www.apache.org/">Foundation</a> |
+ <a
href="https://www.apache.org/events/current-event.html">Events</a> |
+ <a href="https://www.apache.org/licenses/">License</a> |
+ <a href="https://www.apache.org/security/">Security</a> |
+ <a
href="https://www.apache.org/foundation/sponsorship.html">Sponsorship</a> |
+ <a href="https://www.apache.org/foundation/thanks.html">Thanks</a> |
+ <a
href="https://privacy.apache.org/policies/privacy-policy-public.html">Privacy</a>
+ </p>
+ </div>
+ </div>
+ <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
+ </body>
+</html>
diff --git a/blog/author/pmc.html b/blog/author/pmc.html
index fdd6050..9c4c717 100644
--- a/blog/author/pmc.html
+++ b/blog/author/pmc.html
@@ -22,7 +22,7 @@
<article>
- <header> <h2><a
href="https://datafusion.apache.org/blog/2026/08/07/datafusion-comet-1.0.0"
rel="bookmark" title="Permalink to Apache DataFusion Comet 1.0.0
Release">Apache DataFusion Comet 1.0.0 Release</a></h2> </header>
+ <header> <h2><a
href="https://datafusion.apache.org/blog/2026/08/09/datafusion-ballista-54.1.0"
rel="bookmark" title="Permalink to Apache DataFusion Ballista 54.1.0
Released">Apache DataFusion Ballista 54.1.0 Released</a></h2> </header>
<section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
@@ -42,12 +42,12 @@ limitations under the License.
{% endcomment %}
-->
-<p>The Apache DataFusion PMC is pleased to announce version 1.0.0 of the <a
href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
-<p>Comet is an accelerator for Apache Spark that translates Spark physical
plans to DataFusion physical plans for
-improved performance and efficiency without requiring any code changes.</p>
-<p>This release covers roughly six weeks of development …</p></section>
+<p>We are pleased to announce version <a
href="https://github.com/apache/datafusion-ballista/blob/54.1.0/docs/source/changelog/54.1.0.md">54.1.0</a>
of <a href="https://datafusion.apache.org/ballista/">Apache DataFusion
Ballista</a>. Ballista is a distributed query
+execution engine that enhances <a href="https://datafusion.apache.org">Apache
DataFusion</a> by enabling parallel execution of workloads across multiple
+nodes.</p>
+<p>This is a patch release, consisting of 10 commits from 3 contributors. It
follows <a href="/blog/2026/07/12/datafusion-ballista-54.0.0/">54.0.0
…</a></p></section>
<footer>
- <p>Published: <time datetime="2026-08-07T00:00:00+00:00"> Fri
07 August 2026 </time></p>
+ <p>Published: <time datetime="2026-08-09T00:00:00+00:00"> Sun
09 August 2026 </time></p>
<address>By
<a
href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
</address>
diff --git a/blog/category/blog.html b/blog/category/blog.html
index f86bab5..dd438c3 100644
--- a/blog/category/blog.html
+++ b/blog/category/blog.html
@@ -23,7 +23,7 @@
<article>
- <header> <h2><a
href="https://datafusion.apache.org/blog/2026/08/07/datafusion-comet-1.0.0"
rel="bookmark" title="Permalink to Apache DataFusion Comet 1.0.0
Release">Apache DataFusion Comet 1.0.0 Release</a></h2> </header>
+ <header> <h2><a
href="https://datafusion.apache.org/blog/2026/08/09/datafusion-ballista-54.1.0"
rel="bookmark" title="Permalink to Apache DataFusion Ballista 54.1.0
Released">Apache DataFusion Ballista 54.1.0 Released</a></h2> </header>
<section><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
@@ -43,12 +43,12 @@ limitations under the License.
{% endcomment %}
-->
-<p>The Apache DataFusion PMC is pleased to announce version 1.0.0 of the <a
href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
-<p>Comet is an accelerator for Apache Spark that translates Spark physical
plans to DataFusion physical plans for
-improved performance and efficiency without requiring any code changes.</p>
-<p>This release covers roughly six weeks of development …</p></section>
+<p>We are pleased to announce version <a
href="https://github.com/apache/datafusion-ballista/blob/54.1.0/docs/source/changelog/54.1.0.md">54.1.0</a>
of <a href="https://datafusion.apache.org/ballista/">Apache DataFusion
Ballista</a>. Ballista is a distributed query
+execution engine that enhances <a href="https://datafusion.apache.org">Apache
DataFusion</a> by enabling parallel execution of workloads across multiple
+nodes.</p>
+<p>This is a patch release, consisting of 10 commits from 3 contributors. It
follows <a href="/blog/2026/07/12/datafusion-ballista-54.0.0/">54.0.0
…</a></p></section>
<footer>
- <p>Published: <time datetime="2026-08-07T00:00:00+00:00"> Fri
07 August 2026 </time></p>
+ <p>Published: <time datetime="2026-08-09T00:00:00+00:00"> Sun
09 August 2026 </time></p>
<address>By
<a
href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
</address>
diff --git a/blog/feed.xml b/blog/feed.xml
index 9d4ef18..e9bb105 100644
--- a/blog/feed.xml
+++ b/blog/feed.xml
@@ -1,5 +1,5 @@
<?xml version="1.0" encoding="utf-8"?>
-<rss version="2.0"
xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Apache DataFusion
Blog</title><link>https://datafusion.apache.org/blog/</link><description/><atom:link
href="https://datafusion.apache.org/blog/feed.xml"
rel="self"/><lastBuildDate>Fri, 07 Aug 2026 00:00:00
+0000</lastBuildDate><item><title>Apache DataFusion Comet 1.0.0
Release</title><link>https://datafusion.apache.org/blog/2026/08/07/datafusion-comet-1.0.0</link><description><!--
+<rss version="2.0"
xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Apache DataFusion
Blog</title><link>https://datafusion.apache.org/blog/</link><description/><atom:link
href="https://datafusion.apache.org/blog/feed.xml"
rel="self"/><lastBuildDate>Sun, 09 Aug 2026 00:00:00
+0000</lastBuildDate><item><title>Apache DataFusion Ballista 54.1.0
Released</title><link>https://datafusion.apache.org/blog/2026/08/09/datafusion-ballista-54.1.0</link><description><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -18,10 +18,10 @@ limitations under the License.
{% endcomment %}
-->
-<p>The Apache DataFusion PMC is pleased to announce version 1.0.0 of the
<a href="https://datafusion.apache.org/comet/">Comet</a>
subproject.</p>
-<p>Comet is an accelerator for Apache Spark that translates Spark
physical plans to DataFusion physical plans for
-improved performance and efficiency without requiring any code
changes.</p>
-<p>This release covers roughly six weeks of development
…</p></description><dc:creator
xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Fri, 07
Aug 2026 00:00:00
+0000</pubDate><guid>tag:datafusion.apache.org,2026-08-07:/blog/2026/08/07/datafusion-comet-1.0.0</guid><category>blog</category></item><item><title>Optimizing
for Almost Sorted Data: Sort Pushdown in Apache
DataFusion</title><link>https://datafusion.apache.org/blog/2026/07/20/sort-pushdown</link>
[...]
+<p>We are pleased to announce version <a
href="https://github.com/apache/datafusion-ballista/blob/54.1.0/docs/source/changelog/54.1.0.md">54.1.0</a>
of <a href="https://datafusion.apache.org/ballista/">Apache DataFusion
Ballista</a>. Ballista is a distributed query
+execution engine that enhances <a
href="https://datafusion.apache.org">Apache DataFusion</a> by enabling
parallel execution of workloads across multiple
+nodes.</p>
+<p>This is a patch release, consisting of 10 commits from 3
contributors. It follows <a
href="/blog/2026/07/12/datafusion-ballista-54.0.0/">54.0.0
…</a></p></description><dc:creator
xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Sun, 09
Aug 2026 00:00:00
+0000</pubDate><guid>tag:datafusion.apache.org,2026-08-09:/blog/2026/08/09/datafusion-ballista-54.1.0</guid><category>blog</category></item><item><title>Optimizing
for Almost Sorted Data: Sor [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
diff --git a/blog/feeds/all-en.atom.xml b/blog/feeds/all-en.atom.xml
index 0a7a7dc..4c3e65e 100644
--- a/blog/feeds/all-en.atom.xml
+++ b/blog/feeds/all-en.atom.xml
@@ -1,5 +1,5 @@
<?xml version="1.0" encoding="utf-8"?>
-<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion
Blog</title><link href="https://datafusion.apache.org/blog/"
rel="alternate"/><link
href="https://datafusion.apache.org/blog/feeds/all-en.atom.xml"
rel="self"/><id>https://datafusion.apache.org/blog/</id><updated>2026-08-07T00:00:00+00:00</updated><entry><title>Apache
DataFusion Comet 1.0.0 Release</title><link
href="https://datafusion.apache.org/blog/2026/08/07/datafusion-comet-1.0.0"
rel="alternate"/><published>2026-08- [...]
+<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion
Blog</title><link href="https://datafusion.apache.org/blog/"
rel="alternate"/><link
href="https://datafusion.apache.org/blog/feeds/all-en.atom.xml"
rel="self"/><id>https://datafusion.apache.org/blog/</id><updated>2026-08-09T00:00:00+00:00</updated><entry><title>Apache
DataFusion Ballista 54.1.0 Released</title><link
href="https://datafusion.apache.org/blog/2026/08/09/datafusion-ballista-54.1.0"
rel="alternate"/><published [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -18,10 +18,10 @@ limitations under the License.
{% endcomment %}
-->
-<p>The Apache DataFusion PMC is pleased to announce version 1.0.0 of the
<a href="https://datafusion.apache.org/comet/">Comet</a>
subproject.</p>
-<p>Comet is an accelerator for Apache Spark that translates Spark
physical plans to DataFusion physical plans for
-improved performance and efficiency without requiring any code
changes.</p>
-<p>This release covers roughly six weeks of development
…</p></summary><content type="html"><!--
+<p>We are pleased to announce version <a
href="https://github.com/apache/datafusion-ballista/blob/54.1.0/docs/source/changelog/54.1.0.md">54.1.0</a>
of <a href="https://datafusion.apache.org/ballista/">Apache DataFusion
Ballista</a>. Ballista is a distributed query
+execution engine that enhances <a
href="https://datafusion.apache.org">Apache DataFusion</a> by enabling
parallel execution of workloads across multiple
+nodes.</p>
+<p>This is a patch release, consisting of 10 commits from 3
contributors. It follows <a
href="/blog/2026/07/12/datafusion-ballista-54.0.0/">54.0.0
…</a></p></summary><content type="html"><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -40,149 +40,58 @@ limitations under the License.
{% endcomment %}
-->
-<p>The Apache DataFusion PMC is pleased to announce version 1.0.0 of the
<a href="https://datafusion.apache.org/comet/">Comet</a>
subproject.</p>
-<p>Comet is an accelerator for Apache Spark that translates Spark
physical plans to DataFusion physical plans for
-improved performance and efficiency without requiring any code
changes.</p>
-<p>This release covers roughly six weeks of development since 0.17.0 and
consists of 244 commits from 23
-contributors. See the <a
href="https://github.com/apache/datafusion-comet/blob/main/docs/source/changelog/1.0.0.md">change
log</a> for the full list of changes.</p>
-<h2 id="the-road-to-10">The Road to 1.0<a class="headerlink"
href="#the-road-to-10" title="Permanent link">¶</a></h2>
-<p>Comet was <a
href="https://datafusion.apache.org/blog/2024/03/06/comet-donation/">donated</a>
to the Apache DataFusion project in March 2024 and cut its first release,
0.1.0, five
-months later with support for 13 operators and 106 expressions. Since then,
the project has shipped 20
-releases and drawn contributions from more than 120 developers, and the
codebase now recognizes over 400
-Spark expressions. Operator coverage has grown alongside it: 1.0 accelerates
each of Spark's four join
-operators, window functions, generators (<code>explode</code>,
<code>explode_outer</code>, <code>posexplode</code>,
and <code>posexplode_outer</code>
-over arrays), sampling, in-memory table scans, and a fully native
shuffle.</p>
-<p>The 1.0 release marks the point at which Comet begins following <a
href="https://datafusion.apache.org/comet/about/versioning_policy.html">semantic
versioning</a>. Users upgrading
-within the 1.x line can expect backward-compatible changes only; features
slated for removal will be
-deprecated in a minor release before being dropped in the next major version.
This is why the deprecations
-of JDK 11 and Spark 3.4 announced below are scheduled for 1.1 rather than
landing in 1.0 itself.</p>
-<h3 id="support-for-spark-40-with-ansi-mode">Support for Spark 4.0+ with
ANSI mode<a class="headerlink" href="#support-for-spark-40-with-ansi-mode"
title="Permanent link">¶</a></h3>
-<p>Comet 1.0.0 supports Spark versions 3.4 through 4.1, with
experimental support for 4.2. Comet fully supports Spark's ANSI mode, which is
enabled by default starting with Spark 4.0.</p>
-<h3 id="correctness-testing">Correctness Testing<a class="headerlink"
href="#correctness-testing" title="Permanent link">¶</a></h3>
-<p>It is important that queries accelerated by Comet produce the same
results as Spark. Correctness checking has always been a large effort in Comet
development, but the approach has evolved over time.</p>
-<ul>
-<li><strong>Upstream Spark tests</strong>: Comet runs
Spark's own test suite with Comet enabled, providing more than 24,000 unit
tests effectively for free. These tests run in Comet's CI for all supported
Spark versions.</li>
-<li><strong>Scala tests</strong>: end-to-end queries that
run with Comet enabled versus disabled, checking that results match.</li>
-<li><strong>Fuzz testing</strong>: many of the Scala tests
generate randomized data to catch regressions around edge cases such as nulls,
NaN, Infinity, and timezone issues.</li>
-<li><strong>Comet SQL tests</strong>: a
sqllogictest-inspired approach that makes end-to-end tests easier to
write.</li>
-<li><strong>Generative AI audits</strong>: agentic skills
sweep every expression, comparing Comet's implementation to Spark's source and
ensuring tests cover important edge cases.</li>
-</ul>
-<h3 id="performance">Performance<a class="headerlink"
href="#performance" title="Permanent link">¶</a></h3>
-<p>The early Comet releases provided a very modest speedup and the
published benchmark results were based on running TPC workloads at small scale
factors on a single node. There are now independent benchmark results published
by AWS Labs that show significant speedups for <a
href="https://awslabs.github.io/data-on-eks/docs/benchmarks/spark-datafusion-comet-benchmark">TPC-DS
@ 3TB running in EKS</a>.</p>
-<h3 id="codegen-dispatch">Codegen Dispatch<a class="headerlink"
href="#codegen-dispatch" title="Permanent link">¶</a></h3>
-<p>Comet 0.17.0 introduced a new approach to filling gaps in expression
coverage. In earlier releases,
-whenever Comet's planner encountered an expression that lacked a native Rust
implementation, it fell back
-to executing an entire subtree of the plan in Spark. That required converting
Arrow columns back to Spark
-rows before the expression ran and back to Arrow after, and the cost was often
enough to erase the speedup
-Comet had bought elsewhere in the plan.</p>
-<p>Codegen dispatch narrows that fallback to the expression itself: the
batch stays in the Comet pipeline
-and Comet invokes Spark's own generated code for just the missing expression,
leaving the rest of the query
-running natively. Four consequences are worth calling out.</p>
-<ul>
-<li><strong>Coverage.</strong> Expressions that would
previously have blocked native execution of a whole subtree are
- now supported immediately, without a Rust port.</li>
-<li><strong>Compatibility.</strong> For categories where a
native reimplementation would inevitably diverge from Spark's
- semantics — regular expressions being the canonical case, given the gap
between Java's regex engine and
- any Rust or C++ equivalent — codegen dispatch delivers bit-for-bit Spark
parity because it <em>is</em> Spark's
- implementation.</li>
-<li><strong>Expression fusion.</strong> A dispatched
expression tree (<em>i.e.</em>, nested expressions) is compiled
into a single method, so the Arrow
- input reads, the expression evaluation, and the Arrow output writes are
fused together. The compiler
- is free to optimize across the whole tree, and no intermediate Arrow
<code>RecordBatch</code> is materialized
- between one expression and the next.</li>
-<li><strong>Scala and Java UDFs.</strong> User-defined
functions are compiled to the same codegen surface as built-in
- expressions, so they can flow through codegen dispatch without any change
from the user. Queries that
- were previously disqualified from acceleration only because they contained a
UDF can now benefit as
- long as the surrounding operators are supported. See the <a
href="https://datafusion.apache.org/comet/user-guide/latest/scala_java_udfs.html">Scala
and Java UDF guide</a> for details.</li>
-</ul>
-<p>Comet 1.0 widens the mechanism in three ways.</p>
-<p>The first is the biggest. An expression that opts into codegen
dispatch previously reached the dispatcher
-only when Comet reported it as <em>incompatible</em> for the given
input; an <em>unsupported</em> report still sent the
-whole subtree back to Spark. In 1.0 both support levels route through the
dispatcher, so an input that
-Spark handles and Comet's native code does not now stays inside the Comet
pipeline.</p>
-<p>Second, casts join the same path. Cast expressions that Comet
declines to run natively — including legacy
-configuration variants such as
<code>spark.sql.legacy.castComplexTypesToString.enabled</code> —
are now dispatched
-rather than falling back, and more string, array, and interval expressions
were opted in as well.</p>
-<p>Third, the path is now visible. Comet's extended explain output
reports native versus codegen-dispatch
-coverage for a plan, so you can see which path each expression actually took
rather than inferring it from
-the absence of a fallback reason.</p>
-<h2 id="improvements-since-0170">Improvements since 0.17.0<a
class="headerlink" href="#improvements-since-0170" title="Permanent
link">¶</a></h2>
-<p>The rest of this post covers what is new since the 0.17.0
release.</p>
-<h3 id="experimental-pyarrow-udf-support">Experimental PyArrow UDF
Support<a class="headerlink" href="#experimental-pyarrow-udf-support"
title="Permanent link">¶</a></h3>
-<p>This release adds experimental support for accelerated PyArrow UDFs,
allowing PyArrow-based user-defined
-functions to participate in native execution instead of forcing a fallback to
Spark. When the feature is
-disabled, Comet now hints at the native PyArrow UDF path in its fallback
reasons so users know the option
-exists. This is an early-stage feature and we welcome feedback from users
experimenting with it.</p>
-<h3 id="new-expression-and-aggregate-support">New Expression and
Aggregate Support<a class="headerlink"
href="#new-expression-and-aggregate-support" title="Permanent
link">¶</a></h3>
-<p>This release expands the set of Spark expressions and aggregates that
are accelerated by Comet:</p>
+<p>We are pleased to announce version <a
href="https://github.com/apache/datafusion-ballista/blob/54.1.0/docs/source/changelog/54.1.0.md">54.1.0</a>
of <a href="https://datafusion.apache.org/ballista/">Apache DataFusion
Ballista</a>. Ballista is a distributed query
+execution engine that enhances <a
href="https://datafusion.apache.org">Apache DataFusion</a> by enabling
parallel execution of workloads across multiple
+nodes.</p>
+<p>This is a patch release, consisting of 10 commits from 3
contributors. It follows <a
href="/blog/2026/07/12/datafusion-ballista-54.0.0/">54.0.0</a> and
contains no
+new features and no breaking changes — only an upgrade to DataFusion 54.1.0
and a set of bug fixes
+backported from <code>main</code>. Users running 54.0.0 are
encouraged to upgrade.</p>
+<h2 id="upgrade-to-datafusion-5410">Upgrade to DataFusion 54.1.0<a
class="headerlink" href="#upgrade-to-datafusion-5410" title="Permanent
link">¶</a></h2>
+<p>Ballista 54.1.0 is built on DataFusion 54.1.0, picking up the fixes
in that patch release. No API changes
+were required, so the <a
href="https://datafusion.apache.org/ballista/upgrading/54.0.0.html">54.0.0
upgrade guide</a> still applies for anyone moving from an earlier
release.</p>
+<h2 id="cluster-stability-fixes">Cluster stability fixes<a
class="headerlink" href="#cluster-stability-fixes" title="Permanent
link">¶</a></h2>
+<p>Three fixes address problems that only appear in a real multi-node
deployment:</p>
<ul>
-<li><strong>Aggregates</strong>:
<code>approx_percentile</code> /
<code>percentile_approx</code>, exact
<code>percentile</code> / <code>median</code>,
- <code>approx_count_distinct</code>, and native
<code>collect_list</code> /
<code>array_agg</code>.</li>
-<li><strong>Cast</strong>: Cast expressions where the native
implementation is marked as incompatible or unsupported
- are now routed through codegen dispatch.</li>
-<li><strong>Grouping</strong>:
<code>grouping()</code> and
<code>grouping_id()</code>.</li>
-<li><strong>Intervals</strong>: interval types via
<code>make_ym_interval</code> and
<code>make_dt_interval</code>,
<code>CalendarIntervalType</code>,
- <code>multiply_dt_interval</code>, and interval codegen dispatch
for nested values and native shuffle.</li>
-<li><strong>String</strong>:
<code>base64</code>, <code>split_part</code> via
<code>StringSplitSQL</code>, native
<code>levenshtein</code>, and native
<code>randstr</code> and
- <code>uuid</code> — both bit-for-bit compatible with Spark for a
given seed.</li>
-<li><strong>Array / map</strong>:
<code>array_prepend</code>, the <code>shuffle()</code>
array function, <code>size()</code> for
<code>MapType</code>, and <code>ElementAt</code>
- over <code>MapType</code>.</li>
-<li><strong>Date/time</strong>: native
<code>TimestampNTZ</code> inputs for <code>hour</code>
/ <code>minute</code> / <code>second</code> and
- <code>PreciseTimestampConversion</code> for native time-window
grouping.</li>
-<li><strong>Windows</strong>: extended native window
function support and Spark 4 decimal window average.</li>
+<li><strong>Executor client pool enabled by
default.</strong> The executor's <code>client_ttl</code>
defaulted to <code>0</code>, which disabled
+ the gRPC client pool entirely. Without a pool, the shuffle-read path opened
and discarded a TCP connection
+ per fetch, so a single shuffle-heavy query could exhaust the host's
ephemeral ports and fail with
+ <code>AddrNotAvailable</code> — as would every query after it,
until those sockets left <code>TIME_WAIT</code>. A single-executor
+ cluster never showed this, because shuffle reads are served from local
files. <code>client_ttl</code> now defaults to
+ 30 seconds, so the pool is on unless explicitly disabled.</li>
+<li><strong>Executor startup race.</strong> The executor
spawned its gRPC server and registered with the scheduler
+ immediately afterwards, but the socket was not bound yet when the scheduler
dialed back. Losing that race
+ produced <code>ECONNREFUSED</code>, a failed registration, and
an executor that died at startup. The listener is now
+ bound before registration can run.</li>
+<li><strong>Executor shutdown.</strong> The shutdown
notification is now sent before the notifier is dropped, so a graceful
+ shutdown is actually observed by the components waiting on it.</li>
</ul>
-<h3 id="faster-parquet-scans">Faster Parquet Scans<a
class="headerlink" href="#faster-parquet-scans" title="Permanent
link">¶</a></h3>
-<p>Parquet reads pick up several improvements as well. Full Parquet
metadata, including the page index, is now
-cached via DataFusion's
<code>CachedParquetFileReaderFactory</code>; identity casts are
unwrapped in the schema adapter
-so Parquet statistics pruning can engage; filter pushdown configuration has
been revised; the native scan
-passes a metadata size hint so a single read usually captures the footer; and
the native Parquet scan seeds
-its reader options from the session config so Parquet settings you already set
take effect.</p>
-<h3 id="iceberg-table-format-v3">Iceberg Table Format V3<a
class="headerlink" href="#iceberg-table-format-v3" title="Permanent
link">¶</a></h3>
-<p>Comet now supports Iceberg 1.11 and its first Iceberg table format V3
feature: full table encryption. Other V3 features like deletion vectors and new
data types (<em>e.g.</em>, <code>VARIANT</code>) fall
back gracefully. The native Iceberg scan supports the
<code>_pos</code>, <code>_spec</code>,
<code>_file</code>, and
-<code>_partition</code> metadata columns, sizes delete files
correctly to avoid dropped deletes, disambiguates scans that
-share a <code>metadata_location</code>, and dedupes residuals and
delete files in the native scan serde. A prior case
-where Iceberg native scan exchange reuse with different pushed filters could
produce wrong results is also
-fixed.</p>
-<h3 id="native-expression-performance">Native Expression
Performance<a class="headerlink" href="#native-expression-performance"
title="Permanent link">¶</a></h3>
-<p>Many native expression implementations have been optimized to more
efficiently leverage Arrow kernels or to avoid per-row builders.</p>
+<h2 id="task-retry-and-shuffle-memory">Task retry and shuffle
memory<a class="headerlink" href="#task-retry-and-shuffle-memory"
title="Permanent link">¶</a></h2>
<ul>
-<li><strong>Casts</strong> between numeric, string, decimal,
and date types, including a faster float-to-decimal cast, an
- optimized integer-to-integer cast, shared no-overflow fast paths in
<code>CheckOverflow</code> and
- <code>DecimalRescaleCheckOverflow</code>, and a
<code>cast_binary_to_string</code> that is up to 27x faster on
binary-format
- styles.</li>
-<li><strong>JSON, regex, and URL parsing</strong>:
<code>get_json_object</code>,
<code>regexp_extract</code>, and
<code>parse_url</code>.</li>
-<li><strong>Date/time and decimal kernels</strong>:
<code>date_trunc</code>, <code>spark_ceil</code>, and a
vectorized <code>spark_unscaled_value</code>.</li>
-<li><strong>String and array kernels</strong>:
<code>lpad</code>, <code>unhex</code>,
<code>size</code>, <code>arrays_overlap</code>,
<code>escape_string</code>, and the <code>try_*</code>
- arithmetic kernel.</li>
+<li><strong>Retryable IO errors on a join's build
side.</strong> A retryable IO error raised on a join's shared build side
+ arrives at the failed-task classifier wrapped in
<code>DataFusionError::Shared</code>, so a match on the outermost
+ variant never saw the underlying <code>IoError</code> and the
task was marked non-retryable. Classification now
+ operates on the root error, regardless of wrapping.</li>
+<li><strong>Sort shuffle writer respects memory pool
pressure.</strong> The sort shuffle writer discarded the result of
+ <code>try_grow</code> on its memory reservation, so a rejected
grow produced no spill: the batch stayed buffered while
+ the shared <code>MemoryPool</code> had not accounted for it, and
other consumers saw memory as free that was not. A
+ rejected grow is now treated as a spill trigger alongside the existing
per-task budget.</li>
</ul>
-<p>To make this kind of work repeatable, the release also adds a <a
href="https://datafusion.apache.org/comet/contributor-guide/optimizing_expressions.html">scalar
expression optimization guide</a>
-documenting how to benchmark a kernel, keep its output bit-identical to Spark,
and gate changes on a
-no-regression check.</p>
-<h2 id="deprecation-notice">Deprecation Notice<a class="headerlink"
href="#deprecation-notice" title="Permanent link">¶</a></h2>
-<p>With the move to a stable 1.0 release line, Comet begins deprecating
older platforms under its
-<a
href="https://datafusion.apache.org/comet/about/versioning_policy.html">versioning
policy</a>:</p>
+<h2 id="planning-correctness">Planning correctness<a
class="headerlink" href="#planning-correctness" title="Permanent
link">¶</a></h2>
<ul>
-<li><strong>JDK 11</strong> is deprecated and scheduled for
removal in Comet 1.1.0.</li>
-<li><strong>Apache Spark 3.4</strong> is deprecated and
scheduled for removal in Comet 1.1.0.</li>
+<li><strong><code>GROUP BY</code>-less aggregates are
preserved</strong> when the scheduler propagates empty stages, so a query
whose
+ input produces no rows still returns the correct single-row aggregate result
rather than nothing.</li>
+<li><strong><code>EmptyExec</code> partition counts
survive stage serialization</strong>, fixing plans whose partitioning
changed as a
+ stage was sent to an executor.</li>
+<li><strong>Task partitions are mapped through
<code>UnionExec</code></strong> when restricting file scans,
so each task scans the
+ files it was assigned.</li>
+<li><strong><code>SortMergeJoinExec</code> broadcast
conversion is now disabled by default.</strong> The static-planner
optimization
+ that broadcasts the small build side of a sort-merge join, added in 54.0.0,
is opt-in in this release while
+ outstanding issues are resolved.</li>
</ul>
-<p>Comet aligns its Spark support window with upstream Apache Spark
maintenance. Spark 3.4 is no longer
-maintained upstream, so under the versioning policy it is deprecated in the
first Comet minor release
-after that point and removed in the following one. Comet 1.0.0 still builds
and publishes Spark 3.4
-binaries.</p>
-<p>Users on these platforms should plan to move to JDK 17+ and Spark 3.5
or later before upgrading to 1.1.0.</p>
-<h2 id="compatibility">Compatibility<a class="headerlink"
href="#compatibility" title="Permanent link">¶</a></h2>
-<p>Supported platforms include:</p>
-<ul>
-<li><strong>Spark 3.4.3</strong> with Java 11/17 and Scala
2.12/2.13 (deprecated, removal in 1.1.0)</li>
-<li><strong>Spark 3.5.9</strong> with Java 11/17 and Scala
2.12/2.13</li>
-<li><strong>Spark 4.0.4</strong> with Java 17 and Scala
2.13</li>
-<li><strong>Spark 4.1.3</strong> with Java 17/21 and Scala
2.13</li>
-<li><strong>Spark 4.2</strong> with Java 17 and Scala 2.13
(experimental, for early evaluation only)</li>
-</ul>
-<p>See the <a
href="https://datafusion.apache.org/comet/user-guide/latest/compatibility/spark-versions.html">Spark
Version Compatibility</a> page for known limitations specific to each
version.</p>
-<p>This release upgrades to <strong>DataFusion 54.1</strong>
and <strong>Arrow 58.4</strong>.</p>
-<h2 id="get-started-with-comet-100">Get Started with Comet 1.0.0<a
class="headerlink" href="#get-started-with-comet-100" title="Permanent
link">¶</a></h2>
-<p>Ready to try it out? Follow the <a
href="https://datafusion.apache.org/comet/user-guide/1.0/installation.html">Comet
1.0.0 Installation Guide</a>
-to get up and running, then point Comet at your existing Spark workloads and
see the speedup for yourself.</p></content><category
term="blog"/></entry><entry><title>Optimizing for Almost Sorted Data: Sort
Pushdown in Apache DataFusion</title><link
href="https://datafusion.apache.org/blog/2026/07/20/sort-pushdown"
rel="alternate"/><published>2026-07-20T00:00:00+00:00</published><updated>2026-07-20T00:00:00+00:00</updated><author><name>Qi
Zhu and Andrew Lamb</name></author><id>tag:da [...]
+<h2 id="thank-you">Thank You<a class="headerlink" href="#thank-you"
title="Permanent link">¶</a></h2>
+<p>Thanks to Andy Grove, Alexander Domenti, and goingforstudying-ctrl
for the commits in this release, and to
+everyone who contributed by filing issues, reviewing PRs, and providing
feedback. Full details are in the
+<a
href="https://github.com/apache/datafusion-ballista/blob/54.1.0/docs/source/changelog/54.1.0.md">changelog</a>.</p></content><category
term="blog"/></entry><entry><title>Optimizing for Almost Sorted Data: Sort
Pushdown in Apache DataFusion</title><link
href="https://datafusion.apache.org/blog/2026/07/20/sort-pushdown"
rel="alternate"/><published>2026-07-20T00:00:00+00:00</published><updated>2026-07-20T00:00:00+00:00</updated><author><name>Qi
Zhu and Andrew Lamb</name [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
diff --git a/blog/feeds/blog.atom.xml b/blog/feeds/blog.atom.xml
index 87629b6..e6e7c24 100644
--- a/blog/feeds/blog.atom.xml
+++ b/blog/feeds/blog.atom.xml
@@ -1,5 +1,5 @@
<?xml version="1.0" encoding="utf-8"?>
-<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog -
blog</title><link href="https://datafusion.apache.org/blog/"
rel="alternate"/><link
href="https://datafusion.apache.org/blog/feeds/blog.atom.xml"
rel="self"/><id>https://datafusion.apache.org/blog/</id><updated>2026-08-07T00:00:00+00:00</updated><entry><title>Apache
DataFusion Comet 1.0.0 Release</title><link
href="https://datafusion.apache.org/blog/2026/08/07/datafusion-comet-1.0.0"
rel="alternate"/><published>202 [...]
+<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog -
blog</title><link href="https://datafusion.apache.org/blog/"
rel="alternate"/><link
href="https://datafusion.apache.org/blog/feeds/blog.atom.xml"
rel="self"/><id>https://datafusion.apache.org/blog/</id><updated>2026-08-09T00:00:00+00:00</updated><entry><title>Apache
DataFusion Ballista 54.1.0 Released</title><link
href="https://datafusion.apache.org/blog/2026/08/09/datafusion-ballista-54.1.0"
rel="alternate"/><publ [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -18,10 +18,10 @@ limitations under the License.
{% endcomment %}
-->
-<p>The Apache DataFusion PMC is pleased to announce version 1.0.0 of the
<a href="https://datafusion.apache.org/comet/">Comet</a>
subproject.</p>
-<p>Comet is an accelerator for Apache Spark that translates Spark
physical plans to DataFusion physical plans for
-improved performance and efficiency without requiring any code
changes.</p>
-<p>This release covers roughly six weeks of development
…</p></summary><content type="html"><!--
+<p>We are pleased to announce version <a
href="https://github.com/apache/datafusion-ballista/blob/54.1.0/docs/source/changelog/54.1.0.md">54.1.0</a>
of <a href="https://datafusion.apache.org/ballista/">Apache DataFusion
Ballista</a>. Ballista is a distributed query
+execution engine that enhances <a
href="https://datafusion.apache.org">Apache DataFusion</a> by enabling
parallel execution of workloads across multiple
+nodes.</p>
+<p>This is a patch release, consisting of 10 commits from 3
contributors. It follows <a
href="/blog/2026/07/12/datafusion-ballista-54.0.0/">54.0.0
…</a></p></summary><content type="html"><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -40,149 +40,58 @@ limitations under the License.
{% endcomment %}
-->
-<p>The Apache DataFusion PMC is pleased to announce version 1.0.0 of the
<a href="https://datafusion.apache.org/comet/">Comet</a>
subproject.</p>
-<p>Comet is an accelerator for Apache Spark that translates Spark
physical plans to DataFusion physical plans for
-improved performance and efficiency without requiring any code
changes.</p>
-<p>This release covers roughly six weeks of development since 0.17.0 and
consists of 244 commits from 23
-contributors. See the <a
href="https://github.com/apache/datafusion-comet/blob/main/docs/source/changelog/1.0.0.md">change
log</a> for the full list of changes.</p>
-<h2 id="the-road-to-10">The Road to 1.0<a class="headerlink"
href="#the-road-to-10" title="Permanent link">¶</a></h2>
-<p>Comet was <a
href="https://datafusion.apache.org/blog/2024/03/06/comet-donation/">donated</a>
to the Apache DataFusion project in March 2024 and cut its first release,
0.1.0, five
-months later with support for 13 operators and 106 expressions. Since then,
the project has shipped 20
-releases and drawn contributions from more than 120 developers, and the
codebase now recognizes over 400
-Spark expressions. Operator coverage has grown alongside it: 1.0 accelerates
each of Spark's four join
-operators, window functions, generators (<code>explode</code>,
<code>explode_outer</code>, <code>posexplode</code>,
and <code>posexplode_outer</code>
-over arrays), sampling, in-memory table scans, and a fully native
shuffle.</p>
-<p>The 1.0 release marks the point at which Comet begins following <a
href="https://datafusion.apache.org/comet/about/versioning_policy.html">semantic
versioning</a>. Users upgrading
-within the 1.x line can expect backward-compatible changes only; features
slated for removal will be
-deprecated in a minor release before being dropped in the next major version.
This is why the deprecations
-of JDK 11 and Spark 3.4 announced below are scheduled for 1.1 rather than
landing in 1.0 itself.</p>
-<h3 id="support-for-spark-40-with-ansi-mode">Support for Spark 4.0+ with
ANSI mode<a class="headerlink" href="#support-for-spark-40-with-ansi-mode"
title="Permanent link">¶</a></h3>
-<p>Comet 1.0.0 supports Spark versions 3.4 through 4.1, with
experimental support for 4.2. Comet fully supports Spark's ANSI mode, which is
enabled by default starting with Spark 4.0.</p>
-<h3 id="correctness-testing">Correctness Testing<a class="headerlink"
href="#correctness-testing" title="Permanent link">¶</a></h3>
-<p>It is important that queries accelerated by Comet produce the same
results as Spark. Correctness checking has always been a large effort in Comet
development, but the approach has evolved over time.</p>
-<ul>
-<li><strong>Upstream Spark tests</strong>: Comet runs
Spark's own test suite with Comet enabled, providing more than 24,000 unit
tests effectively for free. These tests run in Comet's CI for all supported
Spark versions.</li>
-<li><strong>Scala tests</strong>: end-to-end queries that
run with Comet enabled versus disabled, checking that results match.</li>
-<li><strong>Fuzz testing</strong>: many of the Scala tests
generate randomized data to catch regressions around edge cases such as nulls,
NaN, Infinity, and timezone issues.</li>
-<li><strong>Comet SQL tests</strong>: a
sqllogictest-inspired approach that makes end-to-end tests easier to
write.</li>
-<li><strong>Generative AI audits</strong>: agentic skills
sweep every expression, comparing Comet's implementation to Spark's source and
ensuring tests cover important edge cases.</li>
-</ul>
-<h3 id="performance">Performance<a class="headerlink"
href="#performance" title="Permanent link">¶</a></h3>
-<p>The early Comet releases provided a very modest speedup and the
published benchmark results were based on running TPC workloads at small scale
factors on a single node. There are now independent benchmark results published
by AWS Labs that show significant speedups for <a
href="https://awslabs.github.io/data-on-eks/docs/benchmarks/spark-datafusion-comet-benchmark">TPC-DS
@ 3TB running in EKS</a>.</p>
-<h3 id="codegen-dispatch">Codegen Dispatch<a class="headerlink"
href="#codegen-dispatch" title="Permanent link">¶</a></h3>
-<p>Comet 0.17.0 introduced a new approach to filling gaps in expression
coverage. In earlier releases,
-whenever Comet's planner encountered an expression that lacked a native Rust
implementation, it fell back
-to executing an entire subtree of the plan in Spark. That required converting
Arrow columns back to Spark
-rows before the expression ran and back to Arrow after, and the cost was often
enough to erase the speedup
-Comet had bought elsewhere in the plan.</p>
-<p>Codegen dispatch narrows that fallback to the expression itself: the
batch stays in the Comet pipeline
-and Comet invokes Spark's own generated code for just the missing expression,
leaving the rest of the query
-running natively. Four consequences are worth calling out.</p>
-<ul>
-<li><strong>Coverage.</strong> Expressions that would
previously have blocked native execution of a whole subtree are
- now supported immediately, without a Rust port.</li>
-<li><strong>Compatibility.</strong> For categories where a
native reimplementation would inevitably diverge from Spark's
- semantics — regular expressions being the canonical case, given the gap
between Java's regex engine and
- any Rust or C++ equivalent — codegen dispatch delivers bit-for-bit Spark
parity because it <em>is</em> Spark's
- implementation.</li>
-<li><strong>Expression fusion.</strong> A dispatched
expression tree (<em>i.e.</em>, nested expressions) is compiled
into a single method, so the Arrow
- input reads, the expression evaluation, and the Arrow output writes are
fused together. The compiler
- is free to optimize across the whole tree, and no intermediate Arrow
<code>RecordBatch</code> is materialized
- between one expression and the next.</li>
-<li><strong>Scala and Java UDFs.</strong> User-defined
functions are compiled to the same codegen surface as built-in
- expressions, so they can flow through codegen dispatch without any change
from the user. Queries that
- were previously disqualified from acceleration only because they contained a
UDF can now benefit as
- long as the surrounding operators are supported. See the <a
href="https://datafusion.apache.org/comet/user-guide/latest/scala_java_udfs.html">Scala
and Java UDF guide</a> for details.</li>
-</ul>
-<p>Comet 1.0 widens the mechanism in three ways.</p>
-<p>The first is the biggest. An expression that opts into codegen
dispatch previously reached the dispatcher
-only when Comet reported it as <em>incompatible</em> for the given
input; an <em>unsupported</em> report still sent the
-whole subtree back to Spark. In 1.0 both support levels route through the
dispatcher, so an input that
-Spark handles and Comet's native code does not now stays inside the Comet
pipeline.</p>
-<p>Second, casts join the same path. Cast expressions that Comet
declines to run natively — including legacy
-configuration variants such as
<code>spark.sql.legacy.castComplexTypesToString.enabled</code> —
are now dispatched
-rather than falling back, and more string, array, and interval expressions
were opted in as well.</p>
-<p>Third, the path is now visible. Comet's extended explain output
reports native versus codegen-dispatch
-coverage for a plan, so you can see which path each expression actually took
rather than inferring it from
-the absence of a fallback reason.</p>
-<h2 id="improvements-since-0170">Improvements since 0.17.0<a
class="headerlink" href="#improvements-since-0170" title="Permanent
link">¶</a></h2>
-<p>The rest of this post covers what is new since the 0.17.0
release.</p>
-<h3 id="experimental-pyarrow-udf-support">Experimental PyArrow UDF
Support<a class="headerlink" href="#experimental-pyarrow-udf-support"
title="Permanent link">¶</a></h3>
-<p>This release adds experimental support for accelerated PyArrow UDFs,
allowing PyArrow-based user-defined
-functions to participate in native execution instead of forcing a fallback to
Spark. When the feature is
-disabled, Comet now hints at the native PyArrow UDF path in its fallback
reasons so users know the option
-exists. This is an early-stage feature and we welcome feedback from users
experimenting with it.</p>
-<h3 id="new-expression-and-aggregate-support">New Expression and
Aggregate Support<a class="headerlink"
href="#new-expression-and-aggregate-support" title="Permanent
link">¶</a></h3>
-<p>This release expands the set of Spark expressions and aggregates that
are accelerated by Comet:</p>
+<p>We are pleased to announce version <a
href="https://github.com/apache/datafusion-ballista/blob/54.1.0/docs/source/changelog/54.1.0.md">54.1.0</a>
of <a href="https://datafusion.apache.org/ballista/">Apache DataFusion
Ballista</a>. Ballista is a distributed query
+execution engine that enhances <a
href="https://datafusion.apache.org">Apache DataFusion</a> by enabling
parallel execution of workloads across multiple
+nodes.</p>
+<p>This is a patch release, consisting of 10 commits from 3
contributors. It follows <a
href="/blog/2026/07/12/datafusion-ballista-54.0.0/">54.0.0</a> and
contains no
+new features and no breaking changes — only an upgrade to DataFusion 54.1.0
and a set of bug fixes
+backported from <code>main</code>. Users running 54.0.0 are
encouraged to upgrade.</p>
+<h2 id="upgrade-to-datafusion-5410">Upgrade to DataFusion 54.1.0<a
class="headerlink" href="#upgrade-to-datafusion-5410" title="Permanent
link">¶</a></h2>
+<p>Ballista 54.1.0 is built on DataFusion 54.1.0, picking up the fixes
in that patch release. No API changes
+were required, so the <a
href="https://datafusion.apache.org/ballista/upgrading/54.0.0.html">54.0.0
upgrade guide</a> still applies for anyone moving from an earlier
release.</p>
+<h2 id="cluster-stability-fixes">Cluster stability fixes<a
class="headerlink" href="#cluster-stability-fixes" title="Permanent
link">¶</a></h2>
+<p>Three fixes address problems that only appear in a real multi-node
deployment:</p>
<ul>
-<li><strong>Aggregates</strong>:
<code>approx_percentile</code> /
<code>percentile_approx</code>, exact
<code>percentile</code> / <code>median</code>,
- <code>approx_count_distinct</code>, and native
<code>collect_list</code> /
<code>array_agg</code>.</li>
-<li><strong>Cast</strong>: Cast expressions where the native
implementation is marked as incompatible or unsupported
- are now routed through codegen dispatch.</li>
-<li><strong>Grouping</strong>:
<code>grouping()</code> and
<code>grouping_id()</code>.</li>
-<li><strong>Intervals</strong>: interval types via
<code>make_ym_interval</code> and
<code>make_dt_interval</code>,
<code>CalendarIntervalType</code>,
- <code>multiply_dt_interval</code>, and interval codegen dispatch
for nested values and native shuffle.</li>
-<li><strong>String</strong>:
<code>base64</code>, <code>split_part</code> via
<code>StringSplitSQL</code>, native
<code>levenshtein</code>, and native
<code>randstr</code> and
- <code>uuid</code> — both bit-for-bit compatible with Spark for a
given seed.</li>
-<li><strong>Array / map</strong>:
<code>array_prepend</code>, the <code>shuffle()</code>
array function, <code>size()</code> for
<code>MapType</code>, and <code>ElementAt</code>
- over <code>MapType</code>.</li>
-<li><strong>Date/time</strong>: native
<code>TimestampNTZ</code> inputs for <code>hour</code>
/ <code>minute</code> / <code>second</code> and
- <code>PreciseTimestampConversion</code> for native time-window
grouping.</li>
-<li><strong>Windows</strong>: extended native window
function support and Spark 4 decimal window average.</li>
+<li><strong>Executor client pool enabled by
default.</strong> The executor's <code>client_ttl</code>
defaulted to <code>0</code>, which disabled
+ the gRPC client pool entirely. Without a pool, the shuffle-read path opened
and discarded a TCP connection
+ per fetch, so a single shuffle-heavy query could exhaust the host's
ephemeral ports and fail with
+ <code>AddrNotAvailable</code> — as would every query after it,
until those sockets left <code>TIME_WAIT</code>. A single-executor
+ cluster never showed this, because shuffle reads are served from local
files. <code>client_ttl</code> now defaults to
+ 30 seconds, so the pool is on unless explicitly disabled.</li>
+<li><strong>Executor startup race.</strong> The executor
spawned its gRPC server and registered with the scheduler
+ immediately afterwards, but the socket was not bound yet when the scheduler
dialed back. Losing that race
+ produced <code>ECONNREFUSED</code>, a failed registration, and
an executor that died at startup. The listener is now
+ bound before registration can run.</li>
+<li><strong>Executor shutdown.</strong> The shutdown
notification is now sent before the notifier is dropped, so a graceful
+ shutdown is actually observed by the components waiting on it.</li>
</ul>
-<h3 id="faster-parquet-scans">Faster Parquet Scans<a
class="headerlink" href="#faster-parquet-scans" title="Permanent
link">¶</a></h3>
-<p>Parquet reads pick up several improvements as well. Full Parquet
metadata, including the page index, is now
-cached via DataFusion's
<code>CachedParquetFileReaderFactory</code>; identity casts are
unwrapped in the schema adapter
-so Parquet statistics pruning can engage; filter pushdown configuration has
been revised; the native scan
-passes a metadata size hint so a single read usually captures the footer; and
the native Parquet scan seeds
-its reader options from the session config so Parquet settings you already set
take effect.</p>
-<h3 id="iceberg-table-format-v3">Iceberg Table Format V3<a
class="headerlink" href="#iceberg-table-format-v3" title="Permanent
link">¶</a></h3>
-<p>Comet now supports Iceberg 1.11 and its first Iceberg table format V3
feature: full table encryption. Other V3 features like deletion vectors and new
data types (<em>e.g.</em>, <code>VARIANT</code>) fall
back gracefully. The native Iceberg scan supports the
<code>_pos</code>, <code>_spec</code>,
<code>_file</code>, and
-<code>_partition</code> metadata columns, sizes delete files
correctly to avoid dropped deletes, disambiguates scans that
-share a <code>metadata_location</code>, and dedupes residuals and
delete files in the native scan serde. A prior case
-where Iceberg native scan exchange reuse with different pushed filters could
produce wrong results is also
-fixed.</p>
-<h3 id="native-expression-performance">Native Expression
Performance<a class="headerlink" href="#native-expression-performance"
title="Permanent link">¶</a></h3>
-<p>Many native expression implementations have been optimized to more
efficiently leverage Arrow kernels or to avoid per-row builders.</p>
+<h2 id="task-retry-and-shuffle-memory">Task retry and shuffle
memory<a class="headerlink" href="#task-retry-and-shuffle-memory"
title="Permanent link">¶</a></h2>
<ul>
-<li><strong>Casts</strong> between numeric, string, decimal,
and date types, including a faster float-to-decimal cast, an
- optimized integer-to-integer cast, shared no-overflow fast paths in
<code>CheckOverflow</code> and
- <code>DecimalRescaleCheckOverflow</code>, and a
<code>cast_binary_to_string</code> that is up to 27x faster on
binary-format
- styles.</li>
-<li><strong>JSON, regex, and URL parsing</strong>:
<code>get_json_object</code>,
<code>regexp_extract</code>, and
<code>parse_url</code>.</li>
-<li><strong>Date/time and decimal kernels</strong>:
<code>date_trunc</code>, <code>spark_ceil</code>, and a
vectorized <code>spark_unscaled_value</code>.</li>
-<li><strong>String and array kernels</strong>:
<code>lpad</code>, <code>unhex</code>,
<code>size</code>, <code>arrays_overlap</code>,
<code>escape_string</code>, and the <code>try_*</code>
- arithmetic kernel.</li>
+<li><strong>Retryable IO errors on a join's build
side.</strong> A retryable IO error raised on a join's shared build side
+ arrives at the failed-task classifier wrapped in
<code>DataFusionError::Shared</code>, so a match on the outermost
+ variant never saw the underlying <code>IoError</code> and the
task was marked non-retryable. Classification now
+ operates on the root error, regardless of wrapping.</li>
+<li><strong>Sort shuffle writer respects memory pool
pressure.</strong> The sort shuffle writer discarded the result of
+ <code>try_grow</code> on its memory reservation, so a rejected
grow produced no spill: the batch stayed buffered while
+ the shared <code>MemoryPool</code> had not accounted for it, and
other consumers saw memory as free that was not. A
+ rejected grow is now treated as a spill trigger alongside the existing
per-task budget.</li>
</ul>
-<p>To make this kind of work repeatable, the release also adds a <a
href="https://datafusion.apache.org/comet/contributor-guide/optimizing_expressions.html">scalar
expression optimization guide</a>
-documenting how to benchmark a kernel, keep its output bit-identical to Spark,
and gate changes on a
-no-regression check.</p>
-<h2 id="deprecation-notice">Deprecation Notice<a class="headerlink"
href="#deprecation-notice" title="Permanent link">¶</a></h2>
-<p>With the move to a stable 1.0 release line, Comet begins deprecating
older platforms under its
-<a
href="https://datafusion.apache.org/comet/about/versioning_policy.html">versioning
policy</a>:</p>
+<h2 id="planning-correctness">Planning correctness<a
class="headerlink" href="#planning-correctness" title="Permanent
link">¶</a></h2>
<ul>
-<li><strong>JDK 11</strong> is deprecated and scheduled for
removal in Comet 1.1.0.</li>
-<li><strong>Apache Spark 3.4</strong> is deprecated and
scheduled for removal in Comet 1.1.0.</li>
+<li><strong><code>GROUP BY</code>-less aggregates are
preserved</strong> when the scheduler propagates empty stages, so a query
whose
+ input produces no rows still returns the correct single-row aggregate result
rather than nothing.</li>
+<li><strong><code>EmptyExec</code> partition counts
survive stage serialization</strong>, fixing plans whose partitioning
changed as a
+ stage was sent to an executor.</li>
+<li><strong>Task partitions are mapped through
<code>UnionExec</code></strong> when restricting file scans,
so each task scans the
+ files it was assigned.</li>
+<li><strong><code>SortMergeJoinExec</code> broadcast
conversion is now disabled by default.</strong> The static-planner
optimization
+ that broadcasts the small build side of a sort-merge join, added in 54.0.0,
is opt-in in this release while
+ outstanding issues are resolved.</li>
</ul>
-<p>Comet aligns its Spark support window with upstream Apache Spark
maintenance. Spark 3.4 is no longer
-maintained upstream, so under the versioning policy it is deprecated in the
first Comet minor release
-after that point and removed in the following one. Comet 1.0.0 still builds
and publishes Spark 3.4
-binaries.</p>
-<p>Users on these platforms should plan to move to JDK 17+ and Spark 3.5
or later before upgrading to 1.1.0.</p>
-<h2 id="compatibility">Compatibility<a class="headerlink"
href="#compatibility" title="Permanent link">¶</a></h2>
-<p>Supported platforms include:</p>
-<ul>
-<li><strong>Spark 3.4.3</strong> with Java 11/17 and Scala
2.12/2.13 (deprecated, removal in 1.1.0)</li>
-<li><strong>Spark 3.5.9</strong> with Java 11/17 and Scala
2.12/2.13</li>
-<li><strong>Spark 4.0.4</strong> with Java 17 and Scala
2.13</li>
-<li><strong>Spark 4.1.3</strong> with Java 17/21 and Scala
2.13</li>
-<li><strong>Spark 4.2</strong> with Java 17 and Scala 2.13
(experimental, for early evaluation only)</li>
-</ul>
-<p>See the <a
href="https://datafusion.apache.org/comet/user-guide/latest/compatibility/spark-versions.html">Spark
Version Compatibility</a> page for known limitations specific to each
version.</p>
-<p>This release upgrades to <strong>DataFusion 54.1</strong>
and <strong>Arrow 58.4</strong>.</p>
-<h2 id="get-started-with-comet-100">Get Started with Comet 1.0.0<a
class="headerlink" href="#get-started-with-comet-100" title="Permanent
link">¶</a></h2>
-<p>Ready to try it out? Follow the <a
href="https://datafusion.apache.org/comet/user-guide/1.0/installation.html">Comet
1.0.0 Installation Guide</a>
-to get up and running, then point Comet at your existing Spark workloads and
see the speedup for yourself.</p></content><category
term="blog"/></entry><entry><title>Optimizing for Almost Sorted Data: Sort
Pushdown in Apache DataFusion</title><link
href="https://datafusion.apache.org/blog/2026/07/20/sort-pushdown"
rel="alternate"/><published>2026-07-20T00:00:00+00:00</published><updated>2026-07-20T00:00:00+00:00</updated><author><name>Qi
Zhu and Andrew Lamb</name></author><id>tag:da [...]
+<h2 id="thank-you">Thank You<a class="headerlink" href="#thank-you"
title="Permanent link">¶</a></h2>
+<p>Thanks to Andy Grove, Alexander Domenti, and goingforstudying-ctrl
for the commits in this release, and to
+everyone who contributed by filing issues, reviewing PRs, and providing
feedback. Full details are in the
+<a
href="https://github.com/apache/datafusion-ballista/blob/54.1.0/docs/source/changelog/54.1.0.md">changelog</a>.</p></content><category
term="blog"/></entry><entry><title>Optimizing for Almost Sorted Data: Sort
Pushdown in Apache DataFusion</title><link
href="https://datafusion.apache.org/blog/2026/07/20/sort-pushdown"
rel="alternate"/><published>2026-07-20T00:00:00+00:00</published><updated>2026-07-20T00:00:00+00:00</updated><author><name>Qi
Zhu and Andrew Lamb</name [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
diff --git a/blog/feeds/pmc.atom.xml b/blog/feeds/pmc.atom.xml
index 8509a6b..209358e 100644
--- a/blog/feeds/pmc.atom.xml
+++ b/blog/feeds/pmc.atom.xml
@@ -1,5 +1,5 @@
<?xml version="1.0" encoding="utf-8"?>
-<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog -
pmc</title><link href="https://datafusion.apache.org/blog/"
rel="alternate"/><link
href="https://datafusion.apache.org/blog/feeds/pmc.atom.xml"
rel="self"/><id>https://datafusion.apache.org/blog/</id><updated>2026-08-07T00:00:00+00:00</updated><entry><title>Apache
DataFusion Comet 1.0.0 Release</title><link
href="https://datafusion.apache.org/blog/2026/08/07/datafusion-comet-1.0.0"
rel="alternate"/><published>2026- [...]
+<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog -
pmc</title><link href="https://datafusion.apache.org/blog/"
rel="alternate"/><link
href="https://datafusion.apache.org/blog/feeds/pmc.atom.xml"
rel="self"/><id>https://datafusion.apache.org/blog/</id><updated>2026-08-09T00:00:00+00:00</updated><entry><title>Apache
DataFusion Ballista 54.1.0 Released</title><link
href="https://datafusion.apache.org/blog/2026/08/09/datafusion-ballista-54.1.0"
rel="alternate"/><publis [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -18,10 +18,10 @@ limitations under the License.
{% endcomment %}
-->
-<p>The Apache DataFusion PMC is pleased to announce version 1.0.0 of the
<a href="https://datafusion.apache.org/comet/">Comet</a>
subproject.</p>
-<p>Comet is an accelerator for Apache Spark that translates Spark
physical plans to DataFusion physical plans for
-improved performance and efficiency without requiring any code
changes.</p>
-<p>This release covers roughly six weeks of development
…</p></summary><content type="html"><!--
+<p>We are pleased to announce version <a
href="https://github.com/apache/datafusion-ballista/blob/54.1.0/docs/source/changelog/54.1.0.md">54.1.0</a>
of <a href="https://datafusion.apache.org/ballista/">Apache DataFusion
Ballista</a>. Ballista is a distributed query
+execution engine that enhances <a
href="https://datafusion.apache.org">Apache DataFusion</a> by enabling
parallel execution of workloads across multiple
+nodes.</p>
+<p>This is a patch release, consisting of 10 commits from 3
contributors. It follows <a
href="/blog/2026/07/12/datafusion-ballista-54.0.0/">54.0.0
…</a></p></summary><content type="html"><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -40,149 +40,58 @@ limitations under the License.
{% endcomment %}
-->
-<p>The Apache DataFusion PMC is pleased to announce version 1.0.0 of the
<a href="https://datafusion.apache.org/comet/">Comet</a>
subproject.</p>
-<p>Comet is an accelerator for Apache Spark that translates Spark
physical plans to DataFusion physical plans for
-improved performance and efficiency without requiring any code
changes.</p>
-<p>This release covers roughly six weeks of development since 0.17.0 and
consists of 244 commits from 23
-contributors. See the <a
href="https://github.com/apache/datafusion-comet/blob/main/docs/source/changelog/1.0.0.md">change
log</a> for the full list of changes.</p>
-<h2 id="the-road-to-10">The Road to 1.0<a class="headerlink"
href="#the-road-to-10" title="Permanent link">¶</a></h2>
-<p>Comet was <a
href="https://datafusion.apache.org/blog/2024/03/06/comet-donation/">donated</a>
to the Apache DataFusion project in March 2024 and cut its first release,
0.1.0, five
-months later with support for 13 operators and 106 expressions. Since then,
the project has shipped 20
-releases and drawn contributions from more than 120 developers, and the
codebase now recognizes over 400
-Spark expressions. Operator coverage has grown alongside it: 1.0 accelerates
each of Spark's four join
-operators, window functions, generators (<code>explode</code>,
<code>explode_outer</code>, <code>posexplode</code>,
and <code>posexplode_outer</code>
-over arrays), sampling, in-memory table scans, and a fully native
shuffle.</p>
-<p>The 1.0 release marks the point at which Comet begins following <a
href="https://datafusion.apache.org/comet/about/versioning_policy.html">semantic
versioning</a>. Users upgrading
-within the 1.x line can expect backward-compatible changes only; features
slated for removal will be
-deprecated in a minor release before being dropped in the next major version.
This is why the deprecations
-of JDK 11 and Spark 3.4 announced below are scheduled for 1.1 rather than
landing in 1.0 itself.</p>
-<h3 id="support-for-spark-40-with-ansi-mode">Support for Spark 4.0+ with
ANSI mode<a class="headerlink" href="#support-for-spark-40-with-ansi-mode"
title="Permanent link">¶</a></h3>
-<p>Comet 1.0.0 supports Spark versions 3.4 through 4.1, with
experimental support for 4.2. Comet fully supports Spark's ANSI mode, which is
enabled by default starting with Spark 4.0.</p>
-<h3 id="correctness-testing">Correctness Testing<a class="headerlink"
href="#correctness-testing" title="Permanent link">¶</a></h3>
-<p>It is important that queries accelerated by Comet produce the same
results as Spark. Correctness checking has always been a large effort in Comet
development, but the approach has evolved over time.</p>
-<ul>
-<li><strong>Upstream Spark tests</strong>: Comet runs
Spark's own test suite with Comet enabled, providing more than 24,000 unit
tests effectively for free. These tests run in Comet's CI for all supported
Spark versions.</li>
-<li><strong>Scala tests</strong>: end-to-end queries that
run with Comet enabled versus disabled, checking that results match.</li>
-<li><strong>Fuzz testing</strong>: many of the Scala tests
generate randomized data to catch regressions around edge cases such as nulls,
NaN, Infinity, and timezone issues.</li>
-<li><strong>Comet SQL tests</strong>: a
sqllogictest-inspired approach that makes end-to-end tests easier to
write.</li>
-<li><strong>Generative AI audits</strong>: agentic skills
sweep every expression, comparing Comet's implementation to Spark's source and
ensuring tests cover important edge cases.</li>
-</ul>
-<h3 id="performance">Performance<a class="headerlink"
href="#performance" title="Permanent link">¶</a></h3>
-<p>The early Comet releases provided a very modest speedup and the
published benchmark results were based on running TPC workloads at small scale
factors on a single node. There are now independent benchmark results published
by AWS Labs that show significant speedups for <a
href="https://awslabs.github.io/data-on-eks/docs/benchmarks/spark-datafusion-comet-benchmark">TPC-DS
@ 3TB running in EKS</a>.</p>
-<h3 id="codegen-dispatch">Codegen Dispatch<a class="headerlink"
href="#codegen-dispatch" title="Permanent link">¶</a></h3>
-<p>Comet 0.17.0 introduced a new approach to filling gaps in expression
coverage. In earlier releases,
-whenever Comet's planner encountered an expression that lacked a native Rust
implementation, it fell back
-to executing an entire subtree of the plan in Spark. That required converting
Arrow columns back to Spark
-rows before the expression ran and back to Arrow after, and the cost was often
enough to erase the speedup
-Comet had bought elsewhere in the plan.</p>
-<p>Codegen dispatch narrows that fallback to the expression itself: the
batch stays in the Comet pipeline
-and Comet invokes Spark's own generated code for just the missing expression,
leaving the rest of the query
-running natively. Four consequences are worth calling out.</p>
-<ul>
-<li><strong>Coverage.</strong> Expressions that would
previously have blocked native execution of a whole subtree are
- now supported immediately, without a Rust port.</li>
-<li><strong>Compatibility.</strong> For categories where a
native reimplementation would inevitably diverge from Spark's
- semantics — regular expressions being the canonical case, given the gap
between Java's regex engine and
- any Rust or C++ equivalent — codegen dispatch delivers bit-for-bit Spark
parity because it <em>is</em> Spark's
- implementation.</li>
-<li><strong>Expression fusion.</strong> A dispatched
expression tree (<em>i.e.</em>, nested expressions) is compiled
into a single method, so the Arrow
- input reads, the expression evaluation, and the Arrow output writes are
fused together. The compiler
- is free to optimize across the whole tree, and no intermediate Arrow
<code>RecordBatch</code> is materialized
- between one expression and the next.</li>
-<li><strong>Scala and Java UDFs.</strong> User-defined
functions are compiled to the same codegen surface as built-in
- expressions, so they can flow through codegen dispatch without any change
from the user. Queries that
- were previously disqualified from acceleration only because they contained a
UDF can now benefit as
- long as the surrounding operators are supported. See the <a
href="https://datafusion.apache.org/comet/user-guide/latest/scala_java_udfs.html">Scala
and Java UDF guide</a> for details.</li>
-</ul>
-<p>Comet 1.0 widens the mechanism in three ways.</p>
-<p>The first is the biggest. An expression that opts into codegen
dispatch previously reached the dispatcher
-only when Comet reported it as <em>incompatible</em> for the given
input; an <em>unsupported</em> report still sent the
-whole subtree back to Spark. In 1.0 both support levels route through the
dispatcher, so an input that
-Spark handles and Comet's native code does not now stays inside the Comet
pipeline.</p>
-<p>Second, casts join the same path. Cast expressions that Comet
declines to run natively — including legacy
-configuration variants such as
<code>spark.sql.legacy.castComplexTypesToString.enabled</code> —
are now dispatched
-rather than falling back, and more string, array, and interval expressions
were opted in as well.</p>
-<p>Third, the path is now visible. Comet's extended explain output
reports native versus codegen-dispatch
-coverage for a plan, so you can see which path each expression actually took
rather than inferring it from
-the absence of a fallback reason.</p>
-<h2 id="improvements-since-0170">Improvements since 0.17.0<a
class="headerlink" href="#improvements-since-0170" title="Permanent
link">¶</a></h2>
-<p>The rest of this post covers what is new since the 0.17.0
release.</p>
-<h3 id="experimental-pyarrow-udf-support">Experimental PyArrow UDF
Support<a class="headerlink" href="#experimental-pyarrow-udf-support"
title="Permanent link">¶</a></h3>
-<p>This release adds experimental support for accelerated PyArrow UDFs,
allowing PyArrow-based user-defined
-functions to participate in native execution instead of forcing a fallback to
Spark. When the feature is
-disabled, Comet now hints at the native PyArrow UDF path in its fallback
reasons so users know the option
-exists. This is an early-stage feature and we welcome feedback from users
experimenting with it.</p>
-<h3 id="new-expression-and-aggregate-support">New Expression and
Aggregate Support<a class="headerlink"
href="#new-expression-and-aggregate-support" title="Permanent
link">¶</a></h3>
-<p>This release expands the set of Spark expressions and aggregates that
are accelerated by Comet:</p>
-<ul>
-<li><strong>Aggregates</strong>:
<code>approx_percentile</code> /
<code>percentile_approx</code>, exact
<code>percentile</code> / <code>median</code>,
- <code>approx_count_distinct</code>, and native
<code>collect_list</code> /
<code>array_agg</code>.</li>
-<li><strong>Cast</strong>: Cast expressions where the native
implementation is marked as incompatible or unsupported
- are now routed through codegen dispatch.</li>
-<li><strong>Grouping</strong>:
<code>grouping()</code> and
<code>grouping_id()</code>.</li>
-<li><strong>Intervals</strong>: interval types via
<code>make_ym_interval</code> and
<code>make_dt_interval</code>,
<code>CalendarIntervalType</code>,
- <code>multiply_dt_interval</code>, and interval codegen dispatch
for nested values and native shuffle.</li>
-<li><strong>String</strong>:
<code>base64</code>, <code>split_part</code> via
<code>StringSplitSQL</code>, native
<code>levenshtein</code>, and native
<code>randstr</code> and
- <code>uuid</code> — both bit-for-bit compatible with Spark for a
given seed.</li>
-<li><strong>Array / map</strong>:
<code>array_prepend</code>, the <code>shuffle()</code>
array function, <code>size()</code> for
<code>MapType</code>, and <code>ElementAt</code>
- over <code>MapType</code>.</li>
-<li><strong>Date/time</strong>: native
<code>TimestampNTZ</code> inputs for <code>hour</code>
/ <code>minute</code> / <code>second</code> and
- <code>PreciseTimestampConversion</code> for native time-window
grouping.</li>
-<li><strong>Windows</strong>: extended native window
function support and Spark 4 decimal window average.</li>
-</ul>
-<h3 id="faster-parquet-scans">Faster Parquet Scans<a
class="headerlink" href="#faster-parquet-scans" title="Permanent
link">¶</a></h3>
-<p>Parquet reads pick up several improvements as well. Full Parquet
metadata, including the page index, is now
-cached via DataFusion's
<code>CachedParquetFileReaderFactory</code>; identity casts are
unwrapped in the schema adapter
-so Parquet statistics pruning can engage; filter pushdown configuration has
been revised; the native scan
-passes a metadata size hint so a single read usually captures the footer; and
the native Parquet scan seeds
-its reader options from the session config so Parquet settings you already set
take effect.</p>
-<h3 id="iceberg-table-format-v3">Iceberg Table Format V3<a
class="headerlink" href="#iceberg-table-format-v3" title="Permanent
link">¶</a></h3>
-<p>Comet now supports Iceberg 1.11 and its first Iceberg table format V3
feature: full table encryption. Other V3 features like deletion vectors and new
data types (<em>e.g.</em>, <code>VARIANT</code>) fall
back gracefully. The native Iceberg scan supports the
<code>_pos</code>, <code>_spec</code>,
<code>_file</code>, and
-<code>_partition</code> metadata columns, sizes delete files
correctly to avoid dropped deletes, disambiguates scans that
-share a <code>metadata_location</code>, and dedupes residuals and
delete files in the native scan serde. A prior case
-where Iceberg native scan exchange reuse with different pushed filters could
produce wrong results is also
-fixed.</p>
-<h3 id="native-expression-performance">Native Expression
Performance<a class="headerlink" href="#native-expression-performance"
title="Permanent link">¶</a></h3>
-<p>Many native expression implementations have been optimized to more
efficiently leverage Arrow kernels or to avoid per-row builders.</p>
-<ul>
-<li><strong>Casts</strong> between numeric, string, decimal,
and date types, including a faster float-to-decimal cast, an
- optimized integer-to-integer cast, shared no-overflow fast paths in
<code>CheckOverflow</code> and
- <code>DecimalRescaleCheckOverflow</code>, and a
<code>cast_binary_to_string</code> that is up to 27x faster on
binary-format
- styles.</li>
-<li><strong>JSON, regex, and URL parsing</strong>:
<code>get_json_object</code>,
<code>regexp_extract</code>, and
<code>parse_url</code>.</li>
-<li><strong>Date/time and decimal kernels</strong>:
<code>date_trunc</code>, <code>spark_ceil</code>, and a
vectorized <code>spark_unscaled_value</code>.</li>
-<li><strong>String and array kernels</strong>:
<code>lpad</code>, <code>unhex</code>,
<code>size</code>, <code>arrays_overlap</code>,
<code>escape_string</code>, and the <code>try_*</code>
- arithmetic kernel.</li>
-</ul>
-<p>To make this kind of work repeatable, the release also adds a <a
href="https://datafusion.apache.org/comet/contributor-guide/optimizing_expressions.html">scalar
expression optimization guide</a>
-documenting how to benchmark a kernel, keep its output bit-identical to Spark,
and gate changes on a
-no-regression check.</p>
-<h2 id="deprecation-notice">Deprecation Notice<a class="headerlink"
href="#deprecation-notice" title="Permanent link">¶</a></h2>
-<p>With the move to a stable 1.0 release line, Comet begins deprecating
older platforms under its
-<a
href="https://datafusion.apache.org/comet/about/versioning_policy.html">versioning
policy</a>:</p>
-<ul>
-<li><strong>JDK 11</strong> is deprecated and scheduled for
removal in Comet 1.1.0.</li>
-<li><strong>Apache Spark 3.4</strong> is deprecated and
scheduled for removal in Comet 1.1.0.</li>
-</ul>
-<p>Comet aligns its Spark support window with upstream Apache Spark
maintenance. Spark 3.4 is no longer
-maintained upstream, so under the versioning policy it is deprecated in the
first Comet minor release
-after that point and removed in the following one. Comet 1.0.0 still builds
and publishes Spark 3.4
-binaries.</p>
-<p>Users on these platforms should plan to move to JDK 17+ and Spark 3.5
or later before upgrading to 1.1.0.</p>
-<h2 id="compatibility">Compatibility<a class="headerlink"
href="#compatibility" title="Permanent link">¶</a></h2>
-<p>Supported platforms include:</p>
-<ul>
-<li><strong>Spark 3.4.3</strong> with Java 11/17 and Scala
2.12/2.13 (deprecated, removal in 1.1.0)</li>
-<li><strong>Spark 3.5.9</strong> with Java 11/17 and Scala
2.12/2.13</li>
-<li><strong>Spark 4.0.4</strong> with Java 17 and Scala
2.13</li>
-<li><strong>Spark 4.1.3</strong> with Java 17/21 and Scala
2.13</li>
-<li><strong>Spark 4.2</strong> with Java 17 and Scala 2.13
(experimental, for early evaluation only)</li>
+<p>We are pleased to announce version <a
href="https://github.com/apache/datafusion-ballista/blob/54.1.0/docs/source/changelog/54.1.0.md">54.1.0</a>
of <a href="https://datafusion.apache.org/ballista/">Apache DataFusion
Ballista</a>. Ballista is a distributed query
+execution engine that enhances <a
href="https://datafusion.apache.org">Apache DataFusion</a> by enabling
parallel execution of workloads across multiple
+nodes.</p>
+<p>This is a patch release, consisting of 10 commits from 3
contributors. It follows <a
href="/blog/2026/07/12/datafusion-ballista-54.0.0/">54.0.0</a> and
contains no
+new features and no breaking changes — only an upgrade to DataFusion 54.1.0
and a set of bug fixes
+backported from <code>main</code>. Users running 54.0.0 are
encouraged to upgrade.</p>
+<h2 id="upgrade-to-datafusion-5410">Upgrade to DataFusion 54.1.0<a
class="headerlink" href="#upgrade-to-datafusion-5410" title="Permanent
link">¶</a></h2>
+<p>Ballista 54.1.0 is built on DataFusion 54.1.0, picking up the fixes
in that patch release. No API changes
+were required, so the <a
href="https://datafusion.apache.org/ballista/upgrading/54.0.0.html">54.0.0
upgrade guide</a> still applies for anyone moving from an earlier
release.</p>
+<h2 id="cluster-stability-fixes">Cluster stability fixes<a
class="headerlink" href="#cluster-stability-fixes" title="Permanent
link">¶</a></h2>
+<p>Three fixes address problems that only appear in a real multi-node
deployment:</p>
+<ul>
+<li><strong>Executor client pool enabled by
default.</strong> The executor's <code>client_ttl</code>
defaulted to <code>0</code>, which disabled
+ the gRPC client pool entirely. Without a pool, the shuffle-read path opened
and discarded a TCP connection
+ per fetch, so a single shuffle-heavy query could exhaust the host's
ephemeral ports and fail with
+ <code>AddrNotAvailable</code> — as would every query after it,
until those sockets left <code>TIME_WAIT</code>. A single-executor
+ cluster never showed this, because shuffle reads are served from local
files. <code>client_ttl</code> now defaults to
+ 30 seconds, so the pool is on unless explicitly disabled.</li>
+<li><strong>Executor startup race.</strong> The executor
spawned its gRPC server and registered with the scheduler
+ immediately afterwards, but the socket was not bound yet when the scheduler
dialed back. Losing that race
+ produced <code>ECONNREFUSED</code>, a failed registration, and
an executor that died at startup. The listener is now
+ bound before registration can run.</li>
+<li><strong>Executor shutdown.</strong> The shutdown
notification is now sent before the notifier is dropped, so a graceful
+ shutdown is actually observed by the components waiting on it.</li>
+</ul>
+<h2 id="task-retry-and-shuffle-memory">Task retry and shuffle
memory<a class="headerlink" href="#task-retry-and-shuffle-memory"
title="Permanent link">¶</a></h2>
+<ul>
+<li><strong>Retryable IO errors on a join's build
side.</strong> A retryable IO error raised on a join's shared build side
+ arrives at the failed-task classifier wrapped in
<code>DataFusionError::Shared</code>, so a match on the outermost
+ variant never saw the underlying <code>IoError</code> and the
task was marked non-retryable. Classification now
+ operates on the root error, regardless of wrapping.</li>
+<li><strong>Sort shuffle writer respects memory pool
pressure.</strong> The sort shuffle writer discarded the result of
+ <code>try_grow</code> on its memory reservation, so a rejected
grow produced no spill: the batch stayed buffered while
+ the shared <code>MemoryPool</code> had not accounted for it, and
other consumers saw memory as free that was not. A
+ rejected grow is now treated as a spill trigger alongside the existing
per-task budget.</li>
+</ul>
+<h2 id="planning-correctness">Planning correctness<a
class="headerlink" href="#planning-correctness" title="Permanent
link">¶</a></h2>
+<ul>
+<li><strong><code>GROUP BY</code>-less aggregates are
preserved</strong> when the scheduler propagates empty stages, so a query
whose
+ input produces no rows still returns the correct single-row aggregate result
rather than nothing.</li>
+<li><strong><code>EmptyExec</code> partition counts
survive stage serialization</strong>, fixing plans whose partitioning
changed as a
+ stage was sent to an executor.</li>
+<li><strong>Task partitions are mapped through
<code>UnionExec</code></strong> when restricting file scans,
so each task scans the
+ files it was assigned.</li>
+<li><strong><code>SortMergeJoinExec</code> broadcast
conversion is now disabled by default.</strong> The static-planner
optimization
+ that broadcasts the small build side of a sort-merge join, added in 54.0.0,
is opt-in in this release while
+ outstanding issues are resolved.</li>
</ul>
-<p>See the <a
href="https://datafusion.apache.org/comet/user-guide/latest/compatibility/spark-versions.html">Spark
Version Compatibility</a> page for known limitations specific to each
version.</p>
-<p>This release upgrades to <strong>DataFusion 54.1</strong>
and <strong>Arrow 58.4</strong>.</p>
-<h2 id="get-started-with-comet-100">Get Started with Comet 1.0.0<a
class="headerlink" href="#get-started-with-comet-100" title="Permanent
link">¶</a></h2>
-<p>Ready to try it out? Follow the <a
href="https://datafusion.apache.org/comet/user-guide/1.0/installation.html">Comet
1.0.0 Installation Guide</a>
-to get up and running, then point Comet at your existing Spark workloads and
see the speedup for yourself.</p></content><category
term="blog"/></entry><entry><title>Apache DataFusion Ballista 54.0.0
Released</title><link
href="https://datafusion.apache.org/blog/2026/07/12/datafusion-ballista-54.0.0"
rel="alternate"/><published>2026-07-12T00:00:00+00:00</published><updated>2026-07-12T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2026-07-12:/b
[...]
+<h2 id="thank-you">Thank You<a class="headerlink" href="#thank-you"
title="Permanent link">¶</a></h2>
+<p>Thanks to Andy Grove, Alexander Domenti, and goingforstudying-ctrl
for the commits in this release, and to
+everyone who contributed by filing issues, reviewing PRs, and providing
feedback. Full details are in the
+<a
href="https://github.com/apache/datafusion-ballista/blob/54.1.0/docs/source/changelog/54.1.0.md">changelog</a>.</p></content><category
term="blog"/></entry><entry><title>Apache DataFusion Ballista 54.0.0
Released</title><link
href="https://datafusion.apache.org/blog/2026/07/12/datafusion-ballista-54.0.0"
rel="alternate"/><published>2026-07-12T00:00:00+00:00</published><updated>2026-07-12T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apac
[...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
diff --git a/blog/feeds/pmc.rss.xml b/blog/feeds/pmc.rss.xml
index 9633359..ff84c71 100644
--- a/blog/feeds/pmc.rss.xml
+++ b/blog/feeds/pmc.rss.xml
@@ -1,5 +1,5 @@
<?xml version="1.0" encoding="utf-8"?>
-<rss version="2.0"
xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Apache DataFusion Blog
-
pmc</title><link>https://datafusion.apache.org/blog/</link><description/><atom:link
href="https://datafusion.apache.org/blog/feeds/pmc.rss.xml"
rel="self"/><lastBuildDate>Fri, 07 Aug 2026 00:00:00
+0000</lastBuildDate><item><title>Apache DataFusion Comet 1.0.0
Release</title><link>https://datafusion.apache.org/blog/2026/08/07/datafusion-comet-1.0.0</link><description><!--
+<rss version="2.0"
xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Apache DataFusion Blog
-
pmc</title><link>https://datafusion.apache.org/blog/</link><description/><atom:link
href="https://datafusion.apache.org/blog/feeds/pmc.rss.xml"
rel="self"/><lastBuildDate>Sun, 09 Aug 2026 00:00:00
+0000</lastBuildDate><item><title>Apache DataFusion Ballista 54.1.0
Released</title><link>https://datafusion.apache.org/blog/2026/08/09/datafusion-ballista-54.1.0</link><description><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -18,10 +18,10 @@ limitations under the License.
{% endcomment %}
-->
-<p>The Apache DataFusion PMC is pleased to announce version 1.0.0 of the
<a href="https://datafusion.apache.org/comet/">Comet</a>
subproject.</p>
-<p>Comet is an accelerator for Apache Spark that translates Spark
physical plans to DataFusion physical plans for
-improved performance and efficiency without requiring any code
changes.</p>
-<p>This release covers roughly six weeks of development
…</p></description><dc:creator
xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Fri, 07
Aug 2026 00:00:00
+0000</pubDate><guid>tag:datafusion.apache.org,2026-08-07:/blog/2026/08/07/datafusion-comet-1.0.0</guid><category>blog</category></item><item><title>Apache
DataFusion Ballista 54.0.0
Released</title><link>https://datafusion.apache.org/blog/2026/07/12/datafusion-ballista-54.0.0</link><description><!--
+<p>We are pleased to announce version <a
href="https://github.com/apache/datafusion-ballista/blob/54.1.0/docs/source/changelog/54.1.0.md">54.1.0</a>
of <a href="https://datafusion.apache.org/ballista/">Apache DataFusion
Ballista</a>. Ballista is a distributed query
+execution engine that enhances <a
href="https://datafusion.apache.org">Apache DataFusion</a> by enabling
parallel execution of workloads across multiple
+nodes.</p>
+<p>This is a patch release, consisting of 10 commits from 3
contributors. It follows <a
href="/blog/2026/07/12/datafusion-ballista-54.0.0/">54.0.0
…</a></p></description><dc:creator
xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Sun, 09
Aug 2026 00:00:00
+0000</pubDate><guid>tag:datafusion.apache.org,2026-08-09:/blog/2026/08/09/datafusion-ballista-54.1.0</guid><category>blog</category></item><item><title>Apache
DataFusion Ballista 54.0.0 Rele [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
diff --git a/blog/index.html b/blog/index.html
index a1c79ae..49ee0ba 100644
--- a/blog/index.html
+++ b/blog/index.html
@@ -59,8 +59,8 @@
<article class="post">
<header>
<div class="title">
- <h1><a
href="/blog/2026/08/07/datafusion-comet-1.0.0">Apache DataFusion Comet 1.0.0
Release</a></h1>
- <p>Posted on: Fri 07 August 2026 by pmc</p>
+ <h1><a
href="/blog/2026/08/09/datafusion-ballista-54.1.0">Apache DataFusion Ballista
54.1.0 Released</a></h1>
+ <p>Posted on: Sun 09 August 2026 by pmc</p>
<p><!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
@@ -80,13 +80,13 @@ limitations under the License.
{% endcomment %}
-->
-<p>The Apache DataFusion PMC is pleased to announce version 1.0.0 of the <a
href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
-<p>Comet is an accelerator for Apache Spark that translates Spark physical
plans to DataFusion physical plans for
-improved performance and efficiency without requiring any code changes.</p>
-<p>This release covers roughly six weeks of development …</p></p>
+<p>We are pleased to announce version <a
href="https://github.com/apache/datafusion-ballista/blob/54.1.0/docs/source/changelog/54.1.0.md">54.1.0</a>
of <a href="https://datafusion.apache.org/ballista/">Apache DataFusion
Ballista</a>. Ballista is a distributed query
+execution engine that enhances <a href="https://datafusion.apache.org">Apache
DataFusion</a> by enabling parallel execution of workloads across multiple
+nodes.</p>
+<p>This is a patch release, consisting of 10 commits from 3 contributors. It
follows <a href="/blog/2026/07/12/datafusion-ballista-54.0.0/">54.0.0
…</a></p></p>
<footer>
<ul class="actions">
- <div style="text-align: right"><a
href="/blog/2026/08/07/datafusion-comet-1.0.0" class="button medium">Continue
Reading</a></div>
+ <div style="text-align: right"><a
href="/blog/2026/08/09/datafusion-ballista-54.1.0" class="button
medium">Continue Reading</a></div>
</ul>
<ul class="stats">
</ul>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]