This is an automated email from the ASF dual-hosted git repository.
andygrove pushed a commit to branch branch-54
in repository https://gitbox.apache.org/repos/asf/datafusion-ballista.git
The following commit(s) were added to refs/heads/branch-54 by this push:
new a15c86be6 change log (#2005)
a15c86be6 is described below
commit a15c86be685839a5eecee2567cc46e958e4527df
Author: Andy Grove <[email protected]>
AuthorDate: Sun Jul 12 10:24:12 2026 -0600
change log (#2005)
---
docs/source/changelog/54.0.0.md | 150 ++++++++++++++++++++++++++++++++++++++++
1 file changed, 150 insertions(+)
diff --git a/docs/source/changelog/54.0.0.md b/docs/source/changelog/54.0.0.md
new file mode 100644
index 000000000..6148de32f
--- /dev/null
+++ b/docs/source/changelog/54.0.0.md
@@ -0,0 +1,150 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Apache DataFusion Ballista 54.0.0 Changelog
+
+This release consists of 171 commits from 11 contributors. See credits at the
end of this changelog for more information.
+
+**Fixed bugs:**
+
+- fix(python): clear latent lint debt and gate wheel jobs on lint/tests
[#1749](https://github.com/apache/datafusion-ballista/pull/1749) (andygrove)
+- fix(python): exclude mimalloc from Python wheel to avoid macOS segfault
[#1769](https://github.com/apache/datafusion-ballista/pull/1769) (andygrove)
+- fix: handle task execution errors and panics in executor
[#1793](https://github.com/apache/datafusion-ballista/pull/1793) (milenkovicm)
+- fix: downgrade `time` version to fix build issue
[#1862](https://github.com/apache/datafusion-ballista/pull/1862) (milenkovicm)
+- fix: use DataFrame.**name** instead of hardcoded string in metaclass method
wrapping [#1866](https://github.com/apache/datafusion-ballista/pull/1866)
(djanand)
+- fix: honor client datafusion.\* session config overrides in scheduler
planning [#1902](https://github.com/apache/datafusion-ballista/pull/1902)
(andygrove)
+- fix: Adaptive query planning fails with remote object stores: "No suitable
object store found"
[#1962](https://github.com/apache/datafusion-ballista/pull/1962) (milenkovicm)
+- fix: satisfy clippy 1.97 useless_borrows_in_formatting
[#1978](https://github.com/apache/datafusion-ballista/pull/1978) (andygrove)
+
+**Performance related:**
+
+- perf: avoid redundant TaskStatus clones in scheduler update_task_status
[#1983](https://github.com/apache/datafusion-ballista/pull/1983) (andygrove)
+- perf: prune irrelevant partitions task
[#1911](https://github.com/apache/datafusion-ballista/pull/1911) (coderfender)
+
+**Implemented enhancements:**
+
+- feat: task_manager delegats physical plan creation to execution graph
[#1726](https://github.com/apache/datafusion-ballista/pull/1726) (milenkovicm)
+- feat: ShuffleReader shows `upstream_stage`
[#1767](https://github.com/apache/datafusion-ballista/pull/1767) (milenkovicm)
+- feat: runtime stage's metrics rendering
[#1778](https://github.com/apache/datafusion-ballista/pull/1778) (sandugood)
+- feat: add `/api/job/{job_id}/config` endpoint to retrieve session
configuration [#1792](https://github.com/apache/datafusion-ballista/pull/1792)
(killzoner)
+- feat(aqe): delay join decision & introduce broadcast join in AQE
[#1752](https://github.com/apache/datafusion-ballista/pull/1752) (milenkovicm)
+- feat: Add web rendering of the TUI application
[#1742](https://github.com/apache/datafusion-ballista/pull/1742) (martin-g)
+- feat(aqe): chaos monkey testing for ballista execution
[#1802](https://github.com/apache/datafusion-ballista/pull/1802) (milenkovicm)
+- feat(aqe): empty join handling
[#1723](https://github.com/apache/datafusion-ballista/pull/1723) (sandugood)
+- feat(tui): Show shimmer animation for "Queued" and "Running" jobs
[#1808](https://github.com/apache/datafusion-ballista/pull/1808) (martin-g)
+- feat(cli): use local time in tracing subscriber logging for ballista-cli
[#1824](https://github.com/apache/datafusion-ballista/pull/1824) (jiengup)
+- feat(scheduler): Use command line options for CORS origins/methods instead
of env vars [#1818](https://github.com/apache/datafusion-ballista/pull/1818)
(martin-g)
+- feat(tui): Add configuration popup for jobs
[#1825](https://github.com/apache/datafusion-ballista/pull/1825) (jiengup)
+- feat(TUI): enable various plan rendering formats
[#1828](https://github.com/apache/datafusion-ballista/pull/1828) (sandugood)
+- feat(tui): Add theming support
[#1850](https://github.com/apache/datafusion-ballista/pull/1850) (martin-g)
+- feat: NewType pattern for `job_id`
[#1852](https://github.com/apache/datafusion-ballista/pull/1852) (JarroVGIT)
+- feat(tui): Show job failed status below the Jobs table
[#1871](https://github.com/apache/datafusion-ballista/pull/1871) (martin-g)
+- feat: broadcast small build sides via non-zero CollectLeft thresholds
[#1900](https://github.com/apache/datafusion-ballista/pull/1900) (andygrove)
+- feat: broadcast small build side of SortMergeJoinExec in the static planner
[#1904](https://github.com/apache/datafusion-ballista/pull/1904) (andygrove)
+- feat: Allow submitting a pre-built physical plan to the scheduler
[#1924](https://github.com/apache/datafusion-ballista/pull/1924) (andygrove)
+- feat: cancel running stages and tasks on job failure or cancellation
[#1903](https://github.com/apache/datafusion-ballista/pull/1903) (killzoner)
+- feat: support running TPC-H benchmarks against S3-compatible object storage
[#1942](https://github.com/apache/datafusion-ballista/pull/1942) (andygrove)
+- feat: bound shuffle fetch with a reduce-side in-flight governor
[#1951](https://github.com/apache/datafusion-ballista/pull/1951) (andygrove)
+- feat: improve session state creation for AQE planner
[#1963](https://github.com/apache/datafusion-ballista/pull/1963) (milenkovicm)
+- feat: delete intermediate shuffle files on job success
[#1982](https://github.com/apache/datafusion-ballista/pull/1982) (andygrove)
+- feat: Share read-side runtime state across a session's tasks on the executor
[#1995](https://github.com/apache/datafusion-ballista/pull/1995) (andygrove)
+
+**Documentation updates:**
+
+- docs(release): document PyPI publish process for python client
[#1610](https://github.com/apache/datafusion-ballista/pull/1610) (andygrove)
+- docs(release): cover TestPyPI verification and fix stale wheel-download
notes [#1760](https://github.com/apache/datafusion-ballista/pull/1760)
(andygrove)
+- refactor: make use of `async_trait::async_trait` consistent across project
[#1848](https://github.com/apache/datafusion-ballista/pull/1848) (JarroVGIT)
+- chore(docs): Update docs website to bring inline with Datafusion style
[#1857](https://github.com/apache/datafusion-ballista/pull/1857) (JarroVGIT)
+- Improve changelog generator and split CHANGELOG.md into per-version files
[#1768](https://github.com/apache/datafusion-ballista/pull/1768) (andygrove)
+- docs: add versioned upgrade guide (54.0.0)
[#1939](https://github.com/apache/datafusion-ballista/pull/1939) (andygrove)
+- docs: document TPC-H plan-stability golden files and regeneration
[#1979](https://github.com/apache/datafusion-ballista/pull/1979) (andygrove)
+- docs: update companies using ballista
[#1992](https://github.com/apache/datafusion-ballista/pull/1992) (coderfender)
+- docs: link AQE limitations to tracking issues and add AQE contributor guide
section [#1990](https://github.com/apache/datafusion-ballista/pull/1990)
(andygrove)
+
+**Other:**
+
+- chore(deps): bump idna from 3.11 to 3.15 in /python
[#1737](https://github.com/apache/datafusion-ballista/pull/1737)
(dependabot[bot])
+- chore(ci): Execute the CI workflows only when this is a push to main or a PR
against any branch
[#1738](https://github.com/apache/datafusion-ballista/pull/1738) (martin-g)
+- minor: move aqe executions plans to a directory
[#1744](https://github.com/apache/datafusion-ballista/pull/1744) (milenkovicm)
+- ci: pin all third-party actions to ASF-allowlisted SHAs
[#1748](https://github.com/apache/datafusion-ballista/pull/1748) (andygrove)
+- ci: pin Windows wheel builder to windows-2022
[#1751](https://github.com/apache/datafusion-ballista/pull/1751) (andygrove)
+- chore: bump python crate to 53.0.0 and use path deps for ballista crates
[#1750](https://github.com/apache/datafusion-ballista/pull/1750) (andygrove)
+- ci: produce real aarch64 wheel and upload sdist artifact
[#1753](https://github.com/apache/datafusion-ballista/pull/1753) (andygrove)
+- ci: only run Python Release Build on RC tags and matching PRs
[#1754](https://github.com/apache/datafusion-ballista/pull/1754) (andygrove)
+- ci: gate TPC-H workflow on rust-only path allow list
[#1755](https://github.com/apache/datafusion-ballista/pull/1755) (andygrove)
+- ci: gate Docker workflow on rust-only path allow list
[#1756](https://github.com/apache/datafusion-ballista/pull/1756) (andygrove)
+- ci: gate Rust workflow on workspace path allow list and rename to Ballista
Rust [#1757](https://github.com/apache/datafusion-ballista/pull/1757)
(andygrove)
+- ci: use maturin sdist subcommand to avoid building a wheel
[#1759](https://github.com/apache/datafusion-ballista/pull/1759) (andygrove)
+- ci: gate CodeQL workflow on .github path changes
[#1758](https://github.com/apache/datafusion-ballista/pull/1758) (andygrove)
+- minor: Ballista `showplan` helper script
[#1761](https://github.com/apache/datafusion-ballista/pull/1761) (milenkovicm)
+- chore(deps): bump requests from 2.32.5 to 2.33.0 in /python
[#1781](https://github.com/apache/datafusion-ballista/pull/1781)
(dependabot[bot])
+- chore(ci): Execute the "Dependencies" workflow on push to main and for PRs
[#1782](https://github.com/apache/datafusion-ballista/pull/1782) (martin-g)
+- chore(deps): bump astral-tokio-tar from 0.6.1 to 0.6.2
[#1794](https://github.com/apache/datafusion-ballista/pull/1794)
(dependabot[bot])
+- chore(deps): bump pygments from 2.19.2 to 2.20.0 in /python
[#1800](https://github.com/apache/datafusion-ballista/pull/1800)
(dependabot[bot])
+- minor: code cleanup and log level changes
[#1805](https://github.com/apache/datafusion-ballista/pull/1805) (milenkovicm)
+- minor(aqe): backport few datafusion physical rules identified as not
idempotent [#1806](https://github.com/apache/datafusion-ballista/pull/1806)
(milenkovicm)
+- minor(aqe): add `plan_id` to dynamic join
[#1809](https://github.com/apache/datafusion-ballista/pull/1809) (milenkovicm)
+- chore(docker): Use yellow_tripdata_2022-01.parquet from nightlies.apache.org
[#1817](https://github.com/apache/datafusion-ballista/pull/1817) (martin-g)
+- chore(ci): Use MacOS Github hosted runners for the small CI checks
[#1810](https://github.com/apache/datafusion-ballista/pull/1810) (martin-g)
+- chore(ci): Fix the Rust installer for MacOS in dependencies.yml
[#1833](https://github.com/apache/datafusion-ballista/pull/1833) (martin-g)
+- chore(ci): Drop Windows CI runs
[#1801](https://github.com/apache/datafusion-ballista/pull/1801) (martin-g)
+- chore(ci): Fix the name of a CI job dependency
[#1834](https://github.com/apache/datafusion-ballista/pull/1834) (martin-g)
+- chore(github): Enable "Update branch" button for PRs
[#1839](https://github.com/apache/datafusion-ballista/pull/1839) (martin-g)
+- chore(ci): Download macos aarch64 build of protoc
[#1835](https://github.com/apache/datafusion-ballista/pull/1835) (martin-g)
+- chore(tui): Upgrade Ratatui to 0.30.1 / Ratzilla to 0.3.1
[#1840](https://github.com/apache/datafusion-ballista/pull/1840) (martin-g)
+- chore(ci): Run `cargo check` and `cargo doc` CI jobs on MacOS
[#1846](https://github.com/apache/datafusion-ballista/pull/1846) (martin-g)
+- chore: re enable approx distinct test
[#1844](https://github.com/apache/datafusion-ballista/pull/1844) (coderfender)
+- bug: fix empty projection serde ambiguity bug
[#1863](https://github.com/apache/datafusion-ballista/pull/1863) (coderfender)
+- chore(tui): Use lighter foreground colors for the dark theme. For better
contrast [#1872](https://github.com/apache/datafusion-ballista/pull/1872)
(martin-g)
+- minor: remove redundant scheduler info loggers
[#1879](https://github.com/apache/datafusion-ballista/pull/1879) (milenkovicm)
+- chore(ci): Deploy WebTUI to nightlies.a.o
[#1888](https://github.com/apache/datafusion-ballista/pull/1888) (martin-g)
+- ci: run TPC-H SF10 under AQE off + on, and build it faster
[#1913](https://github.com/apache/datafusion-ballista/pull/1913) (andygrove)
+- chore(deps): upgrade to DataFusion 54
[#1906](https://github.com/apache/datafusion-ballista/pull/1906) (andygrove)
+- ci: verify TPC-H query results against single-process DataFusion
[#1917](https://github.com/apache/datafusion-ballista/pull/1917) (andygrove)
+- chore(deps): upgrade pyballista to DataFusion 54
[#1919](https://github.com/apache/datafusion-ballista/pull/1919) (djanand)
+- Fix/python cargo lock drift
[#1899](https://github.com/apache/datafusion-ballista/pull/1899) (djanand)
+- refactor(scheduler): keep submit_job non-breaking, add submit_physical_plan
[#1941](https://github.com/apache/datafusion-ballista/pull/1941) (andygrove)
+- Add TPC-H distributed plan-stability test suite
[#1966](https://github.com/apache/datafusion-ballista/pull/1966) (andygrove)
+- Display input stage id on UnresolvedShuffleExec
[#1969](https://github.com/apache/datafusion-ballista/pull/1969) (andygrove)
+- chore(deps): bump pyarrow from 23.0.0 to 23.0.1 in /python
[#1961](https://github.com/apache/datafusion-ballista/pull/1961)
(dependabot[bot])
+- chore: add script and CI check to sync vendored DataFusion protos
[#1991](https://github.com/apache/datafusion-ballista/pull/1991) (andygrove)
+- feat(tui + rest): surface failed tasks
[#1949](https://github.com/apache/datafusion-ballista/pull/1949) (killzoner)
+- Add shuffle-read fetch metrics and surface per-operator metrics in plan
display [#1968](https://github.com/apache/datafusion-ballista/pull/1968)
(andygrove)
+- Log task execution duration in the executor "Finished task" message
[#1999](https://github.com/apache/datafusion-ballista/pull/1999) (andygrove)
+- chore: bump Ballista version to 54.0.0
[#2004](https://github.com/apache/datafusion-ballista/pull/2004) (andygrove)
+
+## Credits
+
+Thank you to everyone who contributed to this release. Here is a breakdown of
commits (PRs merged) per contributor.
+
+```
+ 83 dependabot[bot]
+ 38 Andy Grove
+ 18 Martin Grigorov
+ 14 Marko Milenković
+ 4 Bhargava Vadlamani
+ 3 Jarro van Ginkel
+ 3 alexander domenti
+ 3 jgrim
+ 2 Gunther Xing
+ 2 Jogesh A Dinavahi
+ 1 Jogesh Dinavahi
+```
+
+Thank you also to everyone who contributed in other ways such as filing
issues, reviewing PRs, and providing feedback on this release.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
