This is an automated email from the ASF dual-hosted git repository.
milenkovicm pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-ballista.git
The following commit(s) were added to refs/heads/main by this push:
new e0a78666b chore: generate changelog for ballista 53 (#1735)
e0a78666b is described below
commit e0a78666b36eb72d78455d6cfbdf5dcc2972e16d
Author: Marko Milenković <[email protected]>
AuthorDate: Tue May 19 19:01:33 2026 +0100
chore: generate changelog for ballista 53 (#1735)
* chore: generate changelog for ballista 53
* add full changelog
---
CHANGELOG.md | 172 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 172 insertions(+)
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 4baa81564..dfab6fb9f 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -19,6 +19,178 @@
# Changelog
+## [53.0.0](https://github.com/apache/datafusion-ballista/tree/53.0.0)
(2026-05-19)
+
+[Full
Changelog](https://github.com/apache/datafusion-ballista/compare/52.0.0...53.0.0)
+
+**Implemented enhancements:**
+
+- [REST]: Cancelling a completed/failed job should return "false"
[#1494](https://github.com/apache/datafusion-ballista/pull/1494) (martin-g)
+- feat: Add plain "status" field to the JobResponse
[#1497](https://github.com/apache/datafusion-ballista/pull/1497) (martin-g)
+- feat: Expose Logical and Physical plan details in the REST API
[#1498](https://github.com/apache/datafusion-ballista/pull/1498) (milenkovicm)
+- feat: (remote) shuffle reader cleanup
[#1503](https://github.com/apache/datafusion-ballista/pull/1503) (milenkovicm)
+- feat: enable scheduler rest api by default
[#1506](https://github.com/apache/datafusion-ballista/pull/1506) (milenkovicm)
+- feat: [REST] Return the job's start and end times
[#1511](https://github.com/apache/datafusion-ballista/pull/1511) (martin-g)
+- feat: remove hash policy task distribution as not used
[#1526](https://github.com/apache/datafusion-ballista/pull/1526) (eachsaj)
+- feat: remove full path from partition locations
[#1527](https://github.com/apache/datafusion-ballista/pull/1527) (milenkovicm)
+- feat: add config to ExecutorEngine::create_query_stage_exec
[#1542](https://github.com/apache/datafusion-ballista/pull/1542) (milenkovicm)
+- feat: Improve REST API adding task related info to job stages
[#1543](https://github.com/apache/datafusion-ballista/pull/1543) (milenkovicm)
+- feat: Ballista Text User Interface
[#1436](https://github.com/apache/datafusion-ballista/pull/1436) (martin-g)
+- feat: jupyter notebook support
[#1513](https://github.com/apache/datafusion-ballista/pull/1513) (sandugood)
+- feat: ExecutionEngine::create_query_stage_exec accepts partition_id
[#1556](https://github.com/apache/datafusion-ballista/pull/1556) (milenkovicm)
+- feat: make ballista client retry policy configurable
[#1577](https://github.com/apache/datafusion-ballista/pull/1577) (eachsaj)
+- feat: Executor system & process metrics reporting
[#1547](https://github.com/apache/datafusion-ballista/pull/1547) (sandugood)
+- feat: Scheduler config update
[#1597](https://github.com/apache/datafusion-ballista/pull/1597) (sandugood)
+- feat: Support `EXPLAIN ANALYZE` in Ballista
[#1567](https://github.com/apache/datafusion-ballista/pull/1567)
(danielhumanmod)
+- feat: Add standalone shuffle writer benchmark that shuffles real Parquet
input [#1600](https://github.com/apache/datafusion-ballista/pull/1600)
(andygrove)
+- feat: defer sort-shuffle materialization with interleave_record_batch
[#1598](https://github.com/apache/datafusion-ballista/pull/1598) (andygrove)
+- feat(executor): bound executor memory via --memory-pool-size
[#1624](https://github.com/apache/datafusion-ballista/pull/1624) (andygrove)
+- feat(bench): show TPC-H query timings in seconds and add total time
[#1641](https://github.com/apache/datafusion-ballista/pull/1641) (andygrove)
+- feat(aqe): Support sort-based shuffle writer in AQE
[#1640](https://github.com/apache/datafusion-ballista/pull/1640)
(danielhumanmod)
+- feat: rest api supports plan tree rendering
[#1650](https://github.com/apache/datafusion-ballista/pull/1650) (sandugood)
+- feat: Cache ballista clients on executor
[#1578](https://github.com/apache/datafusion-ballista/pull/1578) (milenkovicm)
+- feat(aqe): Lazy stage evaluation in AQE
[#1649](https://github.com/apache/datafusion-ballista/pull/1649) (milenkovicm)
+- feat: move shuffle writer disk I/O off tokio worker threads
[#1537](https://github.com/apache/datafusion-ballista/pull/1537) (hcrosse)
+- feat(scheduler): broadcast-style hash join for small-side joins
[#1647](https://github.com/apache/datafusion-ballista/pull/1647) (andygrove)
+- feat: default to sort-merge join
[#1651](https://github.com/apache/datafusion-ballista/pull/1651) (andygrove)
+- feat: TUI shows running job information
[#1717](https://github.com/apache/datafusion-ballista/pull/1717) (milenkovicm)
+- feat(aqe): CoalescePartitionsRule — shuffle-partition coalescing on resolved
stats [#1684](https://github.com/apache/datafusion-ballista/pull/1684)
(metegenez)
+- feat: TUI make task popup scrollable
[#1725](https://github.com/apache/datafusion-ballista/pull/1725) (milenkovicm)
+- feat(tui): Use separate areas for the table and its associated scrollbar
[#1729](https://github.com/apache/datafusion-ballista/pull/1729) (martin-g)
+
+**Fixed bugs:**
+
+- fix: remove unwrap from executor and improve task error handling
[#1540](https://github.com/apache/datafusion-ballista/pull/1540) (eachsaj)
+- fix: handle None task slot in update_task_info after executor lost
[#1523](https://github.com/apache/datafusion-ballista/pull/1523) (milenkovicm)
+- fix: [Python] serialize optimized logical plan to fix subquery support
[#1586](https://github.com/apache/datafusion-ballista/pull/1586) (andygrove)
+- fix: allow S3 access without explicit credentials
[#1584](https://github.com/apache/datafusion-ballista/pull/1584) (andygrove)
+- fix: route collect/show/to_pandas through Ballista cluster
[#1585](https://github.com/apache/datafusion-ballista/pull/1585) (andygrove)
+- fix: propagate session config from Python client to Ballista cluster
[#1592](https://github.com/apache/datafusion-ballista/pull/1592) (andygrove)
+- fix(python): ignore ballista-namespaced cluster_config keys locally
[#1613](https://github.com/apache/datafusion-ballista/pull/1613) (andygrove)
+- fix: `df.write_` fix as it was broken after update
[#1625](https://github.com/apache/datafusion-ballista/pull/1625) (milenkovicm)
+- fix(rest): remove unwrap and return 404 if executor does not exist
[#1628](https://github.com/apache/datafusion-ballista/pull/1628) (milenkovicm)
+- fix(metrics): avoid stage metrics inflation by tracking partition snapshots
[#1652](https://github.com/apache/datafusion-ballista/pull/1652)
(danielhumanmod)
+- fix: compilation issue after merge
[#1658](https://github.com/apache/datafusion-ballista/pull/1658) (milenkovicm)
+- fix: rest api calculates stage running time correctly
[#1675](https://github.com/apache/datafusion-ballista/pull/1675) (milenkovicm)
+- fix: REST API does not show running jobs
[#1703](https://github.com/apache/datafusion-ballista/pull/1703) (gittihub-jpg)
+- fix: no executor warning, correct prometheus feature name in TUI
[#1698](https://github.com/apache/datafusion-ballista/pull/1698) (killzoner)
+
+**Documentation updates:**
+
+- docs: document hash-based and sort-based shuffle implementations
[#1595](https://github.com/apache/datafusion-ballista/pull/1595) (andygrove)
+- docs: improve Python documentation structure
[#1579](https://github.com/apache/datafusion-ballista/pull/1579) (andygrove)
+- docs: add Python client build-from-source section to contributor guide
[#1620](https://github.com/apache/datafusion-ballista/pull/1620) (andygrove)
+- chore: remove NYC Taxi benchmark
[#1644](https://github.com/apache/datafusion-ballista/pull/1644) (andygrove)
+- docs: document experimental Adaptive Query Execution in tuning guide
[#1645](https://github.com/apache/datafusion-ballista/pull/1645) (andygrove)
+- build(bench): rework docker-compose TPC-H stack
[#1646](https://github.com/apache/datafusion-ballista/pull/1646) (andygrove)
+- docs: add Ballista TUI documentation
[#1593](https://github.com/apache/datafusion-ballista/pull/1593)
(goingforstudying-ctrl)
+- [TUI] Show executor's details in a popup
[#1670](https://github.com/apache/datafusion-ballista/pull/1670) (martin-g)
+- [TUI] Add a config setting for rendering job stage's plan as a tree
[#1704](https://github.com/apache/datafusion-ballista/pull/1704) (martin-g)
+- [TUI] Add support for horizontal scrolling to the job/stage plan popups
[#1711](https://github.com/apache/datafusion-ballista/pull/1711) (martin-g)
+- [TUI] Add screenshots of the TUI application in README/cli.md
[#1714](https://github.com/apache/datafusion-ballista/pull/1714) (martin-g)
+
+**Merged pull requests:**
+
+- CI: Add CodeQL workflow for GitHub Actions security scanning
[#1484](https://github.com/apache/datafusion-ballista/pull/1484) (kevinjqliu)
+- ci: Harden labeler workflow, remove unnecessary checkout from
pull_request_target job
[#1487](https://github.com/apache/datafusion-ballista/pull/1487) (kevinjqliu)
+- ci: add take and stale
[#1488](https://github.com/apache/datafusion-ballista/pull/1488) (kevinjqliu)
+- [REST]: Cancelling a completed/failed job should return "false"
[#1494](https://github.com/apache/datafusion-ballista/pull/1494) (martin-g)
+- feat: Add plain "status" field to the JobResponse
[#1497](https://github.com/apache/datafusion-ballista/pull/1497) (martin-g)
+- feat: Expose Logical and Physical plan details in the REST API
[#1498](https://github.com/apache/datafusion-ballista/pull/1498) (milenkovicm)
+- chore(deps): cargo update deps
[#1502](https://github.com/apache/datafusion-ballista/pull/1502) (milenkovicm)
+- feat: (remote) shuffle reader cleanup
[#1503](https://github.com/apache/datafusion-ballista/pull/1503) (milenkovicm)
+- minor: task scheduling policy config cleanup
[#1507](https://github.com/apache/datafusion-ballista/pull/1507) (milenkovicm)
+- minor: address task policy config comments
[#1508](https://github.com/apache/datafusion-ballista/pull/1508) (milenkovicm)
+- feat: enable scheduler rest api by default
[#1506](https://github.com/apache/datafusion-ballista/pull/1506) (milenkovicm)
+- feat: [REST] Return the job's start and end times
[#1511](https://github.com/apache/datafusion-ballista/pull/1511) (martin-g)
+- minor: [REST] add datafusion version info
[#1512](https://github.com/apache/datafusion-ballista/pull/1512) (milenkovicm)
+- ci: pin third-party actions to Apache-approved SHAs
[#1516](https://github.com/apache/datafusion-ballista/pull/1516) (kevinjqliu)
+- chore(deps): update to datafusion v.53
[#1486](https://github.com/apache/datafusion-ballista/pull/1486) (milenkovicm)
+- feat: remove hash policy task distribution as not used
[#1526](https://github.com/apache/datafusion-ballista/pull/1526) (eachsaj)
+- chore: update datafusion proto
[#1528](https://github.com/apache/datafusion-ballista/pull/1528) (milenkovicm)
+- minor: cleanup scheduler clap configuration
[#1529](https://github.com/apache/datafusion-ballista/pull/1529) (milenkovicm)
+- fix: remove unwrap from executor and improve task error handling
[#1540](https://github.com/apache/datafusion-ballista/pull/1540) (eachsaj)
+- feat: remove full path from partition locations
[#1527](https://github.com/apache/datafusion-ballista/pull/1527) (milenkovicm)
+- fix: handle None task slot in update_task_info after executor lost
[#1523](https://github.com/apache/datafusion-ballista/pull/1523) (milenkovicm)
+- feat: add config to ExecutorEngine::create_query_stage_exec
[#1542](https://github.com/apache/datafusion-ballista/pull/1542) (milenkovicm)
+- feat: Improve REST API adding task related info to job stages
[#1543](https://github.com/apache/datafusion-ballista/pull/1543) (milenkovicm)
+- feat: Ballista Text User Interface
[#1436](https://github.com/apache/datafusion-ballista/pull/1436) (martin-g)
+- feat: jupyter notebook support
[#1513](https://github.com/apache/datafusion-ballista/pull/1513) (sandugood)
+- feat: ExecutionEngine::create_query_stage_exec accepts partition_id
[#1556](https://github.com/apache/datafusion-ballista/pull/1556) (milenkovicm)
+- chore: Fix Clippy issues with Rust 1.95.0
[#1558](https://github.com/apache/datafusion-ballista/pull/1558) (martin-g)
+- chore: update deps
[#1561](https://github.com/apache/datafusion-ballista/pull/1561) (milenkovicm)
+- [TUI] Change the key binding for job's plans
[#1573](https://github.com/apache/datafusion-ballista/pull/1573) (martin-g)
+- feat: make ballista client retry policy configurable
[#1577](https://github.com/apache/datafusion-ballista/pull/1577) (eachsaj)
+- feat: Executor system & process metrics reporting
[#1547](https://github.com/apache/datafusion-ballista/pull/1547) (sandugood)
+- fix: [Python] serialize optimized logical plan to fix subquery support
[#1586](https://github.com/apache/datafusion-ballista/pull/1586) (andygrove)
+- fix: allow S3 access without explicit credentials
[#1584](https://github.com/apache/datafusion-ballista/pull/1584) (andygrove)
+- fix: route collect/show/to_pandas through Ballista cluster
[#1585](https://github.com/apache/datafusion-ballista/pull/1585) (andygrove)
+- fix: propagate session config from Python client to Ballista cluster
[#1592](https://github.com/apache/datafusion-ballista/pull/1592) (andygrove)
+- minor: make config naming convention consistent
[#1580](https://github.com/apache/datafusion-ballista/pull/1580) (milenkovicm)
+- chore: Remove unnecessary optimizer rules due to datafusion upgrade to v53
[#1594](https://github.com/apache/datafusion-ballista/pull/1594) (sandugood)
+- docs: document hash-based and sort-based shuffle implementations
[#1595](https://github.com/apache/datafusion-ballista/pull/1595) (andygrove)
+- feat: Scheduler config update
[#1597](https://github.com/apache/datafusion-ballista/pull/1597) (sandugood)
+- feat: Support `EXPLAIN ANALYZE` in Ballista
[#1567](https://github.com/apache/datafusion-ballista/pull/1567)
(danielhumanmod)
+- docs: improve Python documentation structure
[#1579](https://github.com/apache/datafusion-ballista/pull/1579) (andygrove)
+- feat: Add standalone shuffle writer benchmark that shuffles real Parquet
input [#1600](https://github.com/apache/datafusion-ballista/pull/1600)
(andygrove)
+- feat: defer sort-shuffle materialization with interleave_record_batch
[#1598](https://github.com/apache/datafusion-ballista/pull/1598) (andygrove)
+- ci: drop Intel macOS Python wheel build
[#1612](https://github.com/apache/datafusion-ballista/pull/1612) (andygrove)
+- fix(python): ignore ballista-namespaced cluster_config keys locally
[#1613](https://github.com/apache/datafusion-ballista/pull/1613) (andygrove)
+- chore: update python deps to ballista and datafusion 52
[#1590](https://github.com/apache/datafusion-ballista/pull/1590) (andygrove)
+- feat(sort-shuffle): byte-copy spill files and enable block-IO transport
[#1615](https://github.com/apache/datafusion-ballista/pull/1615) (andygrove)
+- Use BallistaSessionContext instead of BallistaBuilder in tpch.py
[#1621](https://github.com/apache/datafusion-ballista/pull/1621) (martin-g)
+- docs: add Python client build-from-source section to contributor guide
[#1620](https://github.com/apache/datafusion-ballista/pull/1620) (andygrove)
+- feat(executor): bound executor memory via --memory-pool-size
[#1624](https://github.com/apache/datafusion-ballista/pull/1624) (andygrove)
+- fix: `df.write_` fix as it was broken after update
[#1625](https://github.com/apache/datafusion-ballista/pull/1625) (milenkovicm)
+- feat(sort-shuffle): enable sort-based shuffle by default
[#1623](https://github.com/apache/datafusion-ballista/pull/1623) (andygrove)
+- fix(rest): remove unwrap and return 404 if executor does not exist
[#1628](https://github.com/apache/datafusion-ballista/pull/1628) (milenkovicm)
+- perf(sort-shuffle): fix performance regression caused by datafusion upgrade
[#1626](https://github.com/apache/datafusion-ballista/pull/1626) (andygrove)
+- fix(sort-shuffle): bound writer memory with per-task spill threshold
[#1636](https://github.com/apache/datafusion-ballista/pull/1636) (andygrove)
+- chore: remove NYC Taxi benchmark
[#1644](https://github.com/apache/datafusion-ballista/pull/1644) (andygrove)
+- docs: document experimental Adaptive Query Execution in tuning guide
[#1645](https://github.com/apache/datafusion-ballista/pull/1645) (andygrove)
+- feat(bench): show TPC-H query timings in seconds and add total time
[#1641](https://github.com/apache/datafusion-ballista/pull/1641) (andygrove)
+- build(bench): rework docker-compose TPC-H stack
[#1646](https://github.com/apache/datafusion-ballista/pull/1646) (andygrove)
+- feat(aqe): Support sort-based shuffle writer in AQE
[#1640](https://github.com/apache/datafusion-ballista/pull/1640)
(danielhumanmod)
+- fix(metrics): avoid stage metrics inflation by tracking partition snapshots
[#1652](https://github.com/apache/datafusion-ballista/pull/1652)
(danielhumanmod)
+- feat: rest api supports plan tree rendering
[#1650](https://github.com/apache/datafusion-ballista/pull/1650) (sandugood)
+- feat: Cache ballista clients on executor
[#1578](https://github.com/apache/datafusion-ballista/pull/1578) (milenkovicm)
+- fix: compilation issue after merge
[#1658](https://github.com/apache/datafusion-ballista/pull/1658) (milenkovicm)
+- [TUI] Show job's stages and their tasks
[#1574](https://github.com/apache/datafusion-ballista/pull/1574) (martin-g)
+- [TUI] Configurable tick interval
[#1669](https://github.com/apache/datafusion-ballista/pull/1669) (martin-g)
+- feat(aqe): Lazy stage evaluation in AQE
[#1649](https://github.com/apache/datafusion-ballista/pull/1649) (milenkovicm)
+- docs: add Ballista TUI documentation
[#1593](https://github.com/apache/datafusion-ballista/pull/1593)
(goingforstudying-ctrl)
+- feat: move shuffle writer disk I/O off tokio worker threads
[#1537](https://github.com/apache/datafusion-ballista/pull/1537) (hcrosse)
+- feat(scheduler): broadcast-style hash join for small-side joins
[#1647](https://github.com/apache/datafusion-ballista/pull/1647) (andygrove)
+- feat: default to sort-merge join
[#1651](https://github.com/apache/datafusion-ballista/pull/1651) (andygrove)
+- minor: add pending stage indicator for `AdaptiveDatafusionExec`
[#1672](https://github.com/apache/datafusion-ballista/pull/1672) (milenkovicm)
+- fix: rest api calculates stage running time correctly
[#1675](https://github.com/apache/datafusion-ballista/pull/1675) (milenkovicm)
+- minor: change parameter ordering in
`AdaptivePlanner::try_new_with_optimizers`
[#1687](https://github.com/apache/datafusion-ballista/pull/1687) (milenkovicm)
+- [TUI] Show executor's details in a popup
[#1670](https://github.com/apache/datafusion-ballista/pull/1670) (martin-g)
+- Fix REST API panic on job list/detail when end_time < start_time
[#1693](https://github.com/apache/datafusion-ballista/pull/1693)
(abhinavgautam01)
+- Right align all numeric columns in the TUI tables
[#1695](https://github.com/apache/datafusion-ballista/pull/1695) (martin-g)
+- fix(join-selection): guard CollectLeft swap when right has multiple
partitions [#1691](https://github.com/apache/datafusion-ballista/pull/1691)
(andygrove)
+- Minor improvements for #1675
[#1686](https://github.com/apache/datafusion-ballista/pull/1686) (martin-g)
+- [CLI/TUI] Use only tracing crate for logging in CLI and TUI
[#1697](https://github.com/apache/datafusion-ballista/pull/1697) (martin-g)
+- minor: change log level for few statements
[#1706](https://github.com/apache/datafusion-ballista/pull/1706) (milenkovicm)
+- Saturate scheduler job elapsed time
[#1708](https://github.com/apache/datafusion-ballista/pull/1708) (MukundaKatta)
+- [TUI] Add a config setting for rendering job stage's plan as a tree
[#1704](https://github.com/apache/datafusion-ballista/pull/1704) (martin-g)
+- [TUI] Add support for horizontal scrolling to the job/stage plan popups
[#1711](https://github.com/apache/datafusion-ballista/pull/1711) (martin-g)
+- [TUI] Executor's id is not a numeric column. It should be center aligned
[#1713](https://github.com/apache/datafusion-ballista/pull/1713) (martin-g)
+- [TUI] Add screenshots of the TUI application in README/cli.md
[#1714](https://github.com/apache/datafusion-ballista/pull/1714) (martin-g)
+- Make use of Swatinem/rust-cache to make the CI workflows faster
[#1705](https://github.com/apache/datafusion-ballista/pull/1705) (martin-g)
+- Merge Executor's brief and extended properties
[#1716](https://github.com/apache/datafusion-ballista/pull/1716) (martin-g)
+- fix: REST API does not show running jobs
[#1703](https://github.com/apache/datafusion-ballista/pull/1703) (gittihub-jpg)
+- [INFRA] Set up default rulesets for default and release branches
[#1715](https://github.com/apache/datafusion-ballista/pull/1715)
(asf-gitbox-commits)
+- feat: TUI shows running job information
[#1717](https://github.com/apache/datafusion-ballista/pull/1717) (milenkovicm)
+- feat(aqe): CoalescePartitionsRule — shuffle-partition coalescing on resolved
stats [#1684](https://github.com/apache/datafusion-ballista/pull/1684)
(metegenez)
+- feat: TUI make task popup scrollable
[#1725](https://github.com/apache/datafusion-ballista/pull/1725) (milenkovicm)
+- ci: add TPC-H SF10 workflow
[#1688](https://github.com/apache/datafusion-ballista/pull/1688) (andygrove)
+- feat(tui): Use separate areas for the table and its associated scrollbar
[#1729](https://github.com/apache/datafusion-ballista/pull/1729) (martin-g)
+- minor: [TUI] Extract a helper method for the table/scrollbar area splitter
[#1730](https://github.com/apache/datafusion-ballista/pull/1730) (martin-g)
+- minor: [TUI] Sort the metrics before rendering them
[#1731](https://github.com/apache/datafusion-ballista/pull/1731) (martin-g)
+- chore: Do not run -cli tests twice
[#1732](https://github.com/apache/datafusion-ballista/pull/1732) (martin-g)
+- fix: no executor warning, correct prometheus feature name in TUI
[#1698](https://github.com/apache/datafusion-ballista/pull/1698) (killzoner)
+
## [52.0.0](https://github.com/apache/datafusion-ballista/tree/52.0.0)
(2026-03-02)
**Performance related:**
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
