This is an automated email from the ASF dual-hosted git repository.
andygrove pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-ballista.git
The following commit(s) were added to refs/heads/main by this push:
new acede2739 docs: add user personas as a review contract for additive
changes (#2002)
acede2739 is described below
commit acede27396fa73fc690c5aa77fb40f82ce838871
Author: Andy Grove <[email protected]>
AuthorDate: Sun Jul 12 10:51:13 2026 -0600
docs: add user personas as a review contract for additive changes (#2002)
* docs: add user personas as a review contract for additive changes
Ballista serves several distinct audiences, each depending on a different
set of guarantees: DataFusion users going multi-node who want transparent
distribution with identical results, Spark users who want the stage/task
execution model and AQE exposed and tunable, and library users embedding
Ballista as the foundation for a specialized engine who depend on stable,
composable APIs and extension points.
Add a User Personas page to the Contributors Guide that names these
audiences and, for each, records the capabilities it relies on and the
red flags that would regress it. The registry is a review contract:
changes must be additive, so reviewers can check pull requests against it
to avoid silently taking functionality away from an existing audience.
* docs: add "Who is Ballista for" section to the README
Summarize the three audiences Ballista serves — DataFusion users going
multi-node, Spark users wanting the same execution model, and library
users building a specialized engine — and link to the full User Personas
guide for the guarantees each relies on.
---
README.md | 10 ++
docs/source/contributors-guide/user-personas.md | 148 ++++++++++++++++++++++++
docs/source/index.rst | 1 +
3 files changed, 159 insertions(+)
diff --git a/README.md b/README.md
index fe0c20485..99f3a3033 100644
--- a/README.md
+++ b/README.md
@@ -89,6 +89,16 @@ async fn main() -> datafusion::error::Result<()> {
For documentation or more examples, please refer to the [Ballista User
Guide][user-guide].
+## Who is Ballista for
+
+Ballista serves several distinct audiences:
+
+- **DataFusion users going multi-node** — you already use [Apache
DataFusion](https://github.com/apache/datafusion) on a single machine and have
outgrown it. Ballista runs the same SQL and DataFrame workloads across a
cluster with minimal code changes and the same results.
+- **Spark users wanting the same execution model** — you run Spark SQL or
batch jobs and want a lighter, Rust-native alternative without relearning a new
paradigm. Ballista keeps the familiar model: plans split into stages at shuffle
boundaries, one task per partition, executors with task slots, and adaptive
query execution (AQE).
+- **Library users building a specialized engine** — you are building a bespoke
distributed query engine and want reusable scheduler, executor, and
plan-serialization building blocks with extension points, instead of writing
distributed execution from scratch.
+
+These audiences are documented in more detail, along with the guarantees each
relies on, in the [User
Personas](docs/source/contributors-guide/user-personas.md) guide.
+
## Architecture
A Ballista cluster consists of one or more scheduler processes and one or more
executor processes. These processes
diff --git a/docs/source/contributors-guide/user-personas.md
b/docs/source/contributors-guide/user-personas.md
new file mode 100644
index 000000000..a54d03854
--- /dev/null
+++ b/docs/source/contributors-guide/user-personas.md
@@ -0,0 +1,148 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# User Personas
+
+Ballista serves several distinct audiences, each of which comes to the project
+for a different reason and depends on a different set of capabilities. This
page
+names those audiences as **personas** and, for each one, records the guarantees
+that audience relies on.
+
+## Why this page exists
+
+This is a review contract, not marketing copy. It exists so that changes to
+Ballista stay **additive**:
+
+- A pull request may **add** a new persona, or **extend** an existing one with
+ new capabilities.
+- A pull request must **not remove or silently regress** a capability that an
+ existing persona depends on.
+
+Reviewers evaluate every non-trivial pull request against this list. Each
+persona lists concrete "red flags" — the kinds of change that would quietly
take
+functionality away from that audience. A red flag does not mean a change is
+forbidden, but it does mean the change needs explicit discussion and maintainer
+sign-off, and usually a migration path (for example an entry in the
+[upgrade guide](../upgrading/index)), rather than landing silently.
+
+These personas are deliberately in tension with one another. One audience wants
+distributed execution to be a transparent implementation detail; another wants
+the execution model exposed and tunable; another wants Ballista's internals to
+be a stable, composable library. The point of keeping them all written down is
+to make sure a change that delights one audience does not quietly break
another.
+
+## Persona 1: The DataFusion user going multi-node
+
+- **Who**: A data engineer or analyst already using
+ [DataFusion](https://datafusion.apache.org/) single-process (often through
the
+ Python bindings) who has outgrown a single machine.
+- **Coming from**: DataFusion, single process.
+- **Why Ballista**: Run the same SQL and DataFrame workloads across a cluster
+ with minimal code change and _identical results_.
+- **Depends on** (must not regress):
+ - Query results and semantics match single-process DataFusion for the same
+ input.
+ - A client surface that mirrors DataFusion's `SessionContext` / DataFrame /
SQL
+ API.
+ - `datafusion.*` session configuration overrides are honored end-to-end, so
+ users tune DataFusion the way they already do.
+ - Standard object stores and file formats work and produce correct results
+ across partitions.
+ - Distributed execution is _transparent_: getting a correct answer never
+ requires the user to reason about stages, tasks, or partitioning.
+- **Red flags in a pull request**:
+ - Results diverge from single-process DataFusion for the same input.
+ - `datafusion.*` overrides are dropped or ignored during scheduling/planning.
+ - A DataFusion feature that is correct single-node becomes silently wrong
when
+ the plan is split across stages (for example dynamic filter pushdown,
+ uncorrelated scalar subqueries, or a scan reading more than its assigned
+ partition).
+ - Client API changes that break the DataFusion-mirroring surface.
+
+## Persona 2: The Spark user wanting the same execution model
+
+- **Who**: A data engineer or platform owner running Spark SQL / batch jobs who
+ wants a lighter, Rust-native alternative without relearning a new execution
+ paradigm.
+- **Coming from**: Apache Spark.
+- **Why Ballista**: Keep the Spark mental model — plans split into **stages**
at
+ shuffle boundaries, one **task** per partition, shuffle files, executors with
+ task slots, and adaptive query execution (**AQE**) for runtime adaptivity —
on
+ top of DataFusion and Arrow.
+- **Depends on** (must not regress):
+ - The stage/task execution model at shuffle boundaries.
+ - The AQE / adaptive planner path (partition coalescing, dynamic join
+ selection, runtime broadcast), available and gated behind configuration.
+ - Broadcast joins for small build sides, to avoid shuffles and skew (the
+ equivalent of Spark's broadcast hash join).
+ - The operational surface Spark users expect: a scheduler plus executors,
+ task-slot concurrency, a tunable `target_partitions`, and stage/task
+ observability (task timings, a history server / UI) for debugging skew.
+ - Behavior driven by configuration and `SET`, not hard-coded planner choices.
+- **Red flags in a pull request**:
+ - Removing or short-circuiting the stage/task boundary model, or collapsing
+ toward single-process execution.
+ - Regressing or removing the AQE / adaptive planner path — or making it the
+ only path in a way that breaks Persona 1's transparent static path.
+ - Hard-coding a join or planner strategy instead of gating it behind
+ configuration.
+ - Removing the stage/task observability that Spark users rely on to diagnose
+ skew.
+
+## Persona 3: The library user building a specialized engine
+
+- **Who**: An engineer using Ballista as a _framework_ — the foundation for a
+ specialized or bespoke distributed query engine — rather than as a turnkey
+ cluster.
+- **Coming from**: Building on DataFusion as a library, now needing
distribution
+ (or replacing a hand-rolled distributed layer).
+- **Why Ballista**: Reusable scheduler, executor, and plan-serialization
+ building blocks with extension points, instead of writing distributed
+ execution from scratch.
+- **Depends on** (must not regress):
+ - Stable, composable public APIs across the `scheduler`, `executor`,
`client`,
+ and `core` crates (the execution graph and plan serialization included).
+ - Extension points that do not require forking: custom object stores,
+ physical/logical extension codecs, custom operators and functions, a
+ pluggable runtime producer, custom session configuration, and a custom
query
+ planner.
+ - Behavior that stays _configurable rather than hard-coded_, so an embedder
can
+ swap the planner, join strategy, partitioning, or runtime environment.
+ - Backward-compatible public APIs, with breaking changes signaled in the
+ [upgrade guide](../upgrading/index).
+ - No assumptions baked into the core that only fit the built-in clients or
the
+ TPC-H benchmark.
+- **Red flags in a pull request**:
+ - Breaking a public API signature without an upgrade-guide entry.
+ - Hard-coding behavior that used to be pluggable, or removing an extension
+ point (an extension-codec argument, the runtime-producer override,
+ session-config injection).
+ - Coupling core logic to a specific client, planner, or benchmark instead of
+ keeping it behind a trait or configuration.
+ - Wire-format or serialization changes that break externally-produced plans
+ with no compatibility path.
+
+## Adding a persona
+
+This registry is append-only. As Ballista attracts new kinds of users, add a
+persona for each one via a pull request, using the same schema (who they are,
+what they are coming from, why Ballista, what they depend on, and the red flags
+that would regress them). Do not remove existing personas: an audience that
+relied on Ballista does not stop existing because our priorities shifted, and
+the whole value of this page is that the guarantees only accumulate.
diff --git a/docs/source/index.rst b/docs/source/index.rst
index e66e23752..c208c29ec 100644
--- a/docs/source/index.rst
+++ b/docs/source/index.rst
@@ -78,6 +78,7 @@ Table of content
contributors-guide/architecture
contributors-guide/code-organization
+ contributors-guide/user-personas
contributors-guide/development
Source code <https://github.com/apache/datafusion-ballista/>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
