andygrove commented on code in PR #188:
URL: https://github.com/apache/datafusion-site/pull/188#discussion_r3298486903
##########
content/blog/2026-05-24-datafusion-ballista-53.0.0.md:
##########
@@ -0,0 +1,289 @@
+---
+layout: post
+title: Apache DataFusion Ballista 53.0.0 Released
+date: 2026-05-24
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+We are pleased to announce version [53.0.0] of [Apache DataFusion Ballista].
Ballista is a distributed query
+execution engine that enhances [Apache DataFusion] by enabling parallel
execution of workloads across multiple
+nodes.
+
+[53.0.0]:
https://github.com/apache/datafusion-ballista/blob/main/CHANGELOG.md#5300-2026-05-19
+[Apache DataFusion Ballista]: https://datafusion.apache.org/ballista/
+[Apache DataFusion]: https://datafusion.apache.org
+
+The last Ballista blog post covered [43.0.0], released in January 2025. In the
year and a bit since, the
+project has quietly shipped a release for every DataFusion release: 44, 45,
46, 47, 48, 49, 50, 51, 52, and
+now 53. This post catches up on what changed across that span, what landed
specifically in 53.0.0, and where
+the project is heading.
+
+[43.0.0]: /blog/2025/02/02/datafusion-ballista-43.0.0/
+
+## How Ballista has changed since 43.0.0
+
+The story of 43.0.0 was one of simplification: experimental features were
removed, the `BallistaContext` was
+deprecated in favor of the standard DataFusion `SessionContext`, and the
project's release cadence was
+aligned with DataFusion's. The story of the year that followed has been one of
putting things back, but
+under a more deliberate design.
+
+### Production deployment
+
+A lot of the work over this period has been about running Ballista in real
clusters rather than just on a
+developer's laptop:
+
+- **S3 object store support** has been added to both the executor and
scheduler binaries, including
+ credentials derived from the standard AWS environment, instance metadata,
and explicit configuration.
+- **Docker images** for the scheduler and executor are now published on each
release, making Docker Compose
+ and Kubernetes deployments straightforward.
+- **Cluster RPC** can be configured with TLS and custom headers, enabling
deployments that need encrypted
+ inter-component traffic or pass-through authentication.
+- **Push-based task scheduling** is now the default, replacing pull-staged
scheduling. Push scheduling
+ generally results in lower latency for short queries. Both modes remain
available.
+- **Configurable gRPC timeouts**, retry policies, and message size limits make
it easier to operate clusters
+ under varying network conditions.
+- **Memory bounds for executors** can now be set with `--memory-pool-size`, so
executors no longer rely on
+ unbounded growth.
+
+### Shuffle subsystem
+
+The shuffle subsystem received the largest single rework over this period.
+
+- A new **sort-based shuffle writer** was added in 52.0.0 and made the default
in 53.0.0. The hash-based
+ writer remains available behind a configuration flag.
+- **Buffered I/O** in the shuffle writer significantly reduces the number of
small writes, and disk I/O
+ has been moved off the Tokio worker threads so that I/O latency does not
block scheduling.
+- **Per-task spill thresholds** bound writer memory in the sort-based path,
and a deferred materialization
+ step using `interleave_record_batch` reduces allocator pressure during
shuffle write.
+- **Remote shuffle reads** now use Arrow Flight directly, with a client cache
on the executor side, giving
+ better throughput and resource utilization for shuffle-heavy queries.
+- **Shuffle reader cleanup** removes job-local data once a job completes.
+
+### REST API and observability
+
+The scheduler's REST API has grown from a small status surface to the primary
control plane for inspecting
+running and completed jobs:
+
+- The REST API is now enabled by default.
+- `/api/jobs` and `/api/jobs/<job_id>` expose job status, start/end times,
logical and physical plans,
+ per-stage task information, and metrics.
+- Plans can be rendered as a tree directly from the REST API.
+- Per-executor system and process metrics are reported, and Prometheus metrics
integration is available
+ behind a feature flag.
+
+### A new Python interface
+
+A redesigned Python client has replaced `BallistaBuilder`. The new entry point
is `BallistaSessionContext`,
+which mirrors the DataFusion Python `SessionContext` API:
+
+```python
+from ballista import BallistaSessionContext
+
+ctx = BallistaSessionContext(
+ "df://localhost:50050",
+ cluster_config={
+ "datafusion.execution.target_partitions": "32",
+ },
+)
+
+ctx.register_parquet("trips", "/mnt/bigdata/trips")
+df = ctx.sql("SELECT vendor_id, COUNT(*) FROM trips GROUP BY vendor_id")
+df.show()
+```
+
+A number of fixes since 43.0.0 made this client much more usable in
distributed environments: session
+configuration is now propagated from the Python client to the cluster;
`collect`, `show`, and `to_pandas`
+go through the cluster instead of falling back to a local execution path; and
S3 access works without
+requiring explicit credentials. Jupyter notebook integration is documented in
the [Python user guide].
+
+[Python user guide]:
https://datafusion.apache.org/ballista/user-guide/python/quickstart.html
+
+The release process has also been extended so that future Ballista releases
will publish Python wheels to
+[PyPI] as `ballista`. Note that the Python bindings included in **53.0.0 still
report version 52.0.0**
+because the version bump landed shortly after the 53.0.0 release candidate was
tagged. Wheels matching
+the 53 line will be published with **53.1.0**, which is expected to follow
shortly.
+
+[PyPI]: https://pypi.org/project/ballista/
+
+### Spark compatibility and Substrait
+
+Ballista now supports the `spark-compat` Cargo feature, which auto-registers
the
+[`datafusion-spark`] function library in the executor session context. This
makes it possible to evaluate
+Spark-compatible SQL semantics on a Ballista cluster.
+
+The scheduler also has a Substrait surface: `SubstraitSchedulerClient` accepts
Substrait logical plans, and
+the deprecated SQL-string submission path has been removed. This is an
important step toward decoupling
+Ballista from any one client language.
+
+[`datafusion-spark`]: https://docs.rs/datafusion-spark/latest/datafusion_spark/
+
+## Highlights of 53.0.0
+
+53.0.0 is a feature-heavy release, with significant work in observability, the
planner, and the executor.
+
+### Terminal User Interface
+
+The Ballista CLI now ships with an integrated Terminal User Interface for
monitoring a running cluster.
+The TUI is enabled with `ballista-cli --tui`, or by typing `\tui` inside the
CLI. It provides views for
+executors, jobs, stages, tasks, plan trees, and metrics, all backed by the
scheduler's REST API.
+
+<img
+src="/blog/images/datafusion-ballista-53.0.0/tui-jobs-table.png"
+width="100%"
+class="img-fluid"
+alt="Ballista TUI jobs view"
+/>
+
+Plan rendering, including a graph view, is available directly from the TUI:
+
+<img
+src="/blog/images/datafusion-ballista-53.0.0/tui-job-plan-graph-popup.png"
Review Comment:
Keeping the graph image in this revision — happy to add a text-plan
screenshot in a follow-up if you have one you'd like to use.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
