This is an automated email from the ASF dual-hosted git repository.
andygrove pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-ballista.git
The following commit(s) were added to refs/heads/main by this push:
new b55d1e99e Generate the config reference documentation from the config
registry (#2146)
b55d1e99e is described below
commit b55d1e99e6ddc1a30fd6672b7d23986a7060c7ab
Author: Andy Grove <[email protected]>
AuthorDate: Wed Jul 22 12:12:50 2026 -0600
Generate the config reference documentation from the config registry (#2146)
* feat(core): expose ConfigEntry metadata for doc generation
* feat(core): render config registry as prettier-compatible tables
* refactor: use expect instead of allow for dead_code lint
Replace crate-level #![allow(dead_code)] with #![expect(dead_code)] to
ensure the attribute becomes a hard error once main() is wired up and items
become reachable, forcing cleanup rather than silently suppressing findings.
* feat(core): splice generated config tables into marked doc regions
* Remove dead_code expectation from update_config_docs
The file's test module exercises every function, making the
expect(dead_code) attribute unfulfilled under --all-targets. Dead code
warnings in non-test builds are expected until main is wired up.
* fix: scope dead_code expectation to non-test builds
cargo clippy --all-targets compiles both the normal bin target, where the
stub main leaves every item unreachable, and the test target, where the
tests exercise all of them. An unconditional attribute fails one config or
the other. Guarding on not(test) satisfies both, and expect keeps the
attribute self-cleaning once main is wired up.
* fix(config-docs): don't treat markers shown as examples as live
splice() matched BEGIN/END markers on the trimmed line with no regard for
fenced code blocks or indentation, so a marker documented as a syntax
example (e.g. showing "how to add a config key" inside a fenced block, or
indented in a list item) was indistinguishable from a real one. The
generator would then silently overwrite or prematurely close a documented
example as if it were a live region, corrupting the docs without any
error.
Require markers to start at column 0 (trailing whitespace still tolerated)
and track fenced-code-block state while scanning, ignoring BEGIN/END while
inside a fence. The region-body-consuming inner loop tracks fence state
too, so an END shown inside a fenced example within a live region no
longer closes that region early.
* feat(core): add update_config_docs CLI with drift and coverage checks
* docs: generate config reference tables from the config registry
* docs: clean up two config entry descriptions
Fix grammar and redundant wording in the descriptions for
BALLISTA_PROPAGATE_EMPTY_ENABLED and
BALLISTA_COALESCE_TARGET_PARTITION_BYTES, now that the config docs
generator publishes them to the user-facing reference table.
* docs: drop redundant join reference in propagate-empty description
* ci: check generated config docs stay in sync with the registry
* docs: warn readers not to hand-edit generated config tables
* docs: use plain prose for the generated-tables notice
GitHub alert syntax is not used anywhere else in docs/source and these pages
render through myst-parser, where it may show the literal marker instead of
an
admonition.
* ci: install protoc before config-docs-check job
config-docs-check runs cargo run -p ballista-core, which triggers
build.rs and requires protoc on PATH. The job was modelled on
datafusion-proto-sync-check, which never compiles Rust and doesn't
need protoc, so that template omitted the install step. Add it,
matching the approach already used in docker.yml.
* build: run update_config_docs with the ci profile
Every other Rust-compiling CI job uses --profile ci to strip
debuginfo and keep the rust-cache payload small. Bring
update_config_docs.sh in line; the profile only changes build
output, not behaviour.
* fix: report written files even when the orphan-key check fails
run() wrote changed files inside its main loop but only reported
them afterwards, in a branch skipped by the early return from the
orphan-key check. A contributor who rewrites a description and adds
an undocumented key in the same commit would see only the orphan
error, with no indication that configs.md had already been rewritten
on disk. Print each "updated {path}" line as soon as that file is
written so it always surfaces before any later failure.
* fix: reject unrecognised arguments instead of defaulting to write mode
main() only ever checked for the presence of --check, so any other
argument (a typo like --chek, or --help) silently fell through to
write mode. In CI that would turn the config-docs-check gate into a
permanent no-op pass. Parse arguments explicitly: accept --check,
print usage and exit 0 for --help/-h, and exit non-zero with usage
for anything else.
* fix: collapse newlines in escape_cell
All descriptions are single-line today, but nothing stopped a future
multi-line description from emitting a literal newline mid-row,
silently corrupting the table while still leaving the generator
idempotent. Collapse newlines and carriage returns to a single space
alongside the existing pipe escaping, and add a test mirroring the
existing pipe-escaping invariant test.
* fix: point orphan-key error at the right section, not prefix widening
The message's primary suggestion was to widen an existing region's
prefix=, which a contributor could satisfy by widening all the way
to prefix=ballista., dumping every key into one table and
duplicating the others. Reword so the primary suggestion is adding
the key to the marker region for its own section, and mention
prefix widening only as the narrower fallback it is.
* docs: document the config docs generator in CONTRIBUTING
* docs: regenerate config tables after merging main
PR #2124 changed hash_join_max_build_partition_bytes from 0 to 64 MiB and
reworded its description. The generated table now reflects that.
---
.github/workflows/rust.yml | 19 +
CONTRIBUTING.md | 16 +
ballista/core/src/bin/update_config_docs.rs | 835 ++++++++++++++++++++++++++++
ballista/core/src/config.rs | 97 +++-
ci/scripts/rust_config_docs_check.sh | 23 +
dev/update_config_docs.sh | 23 +
docs/source/user-guide/configs.md | 87 ++-
7 files changed, 1085 insertions(+), 15 deletions(-)
diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.yml
index d6136b6bf..e8c49aaa6 100644
--- a/.github/workflows/rust.yml
+++ b/.github/workflows/rust.yml
@@ -270,6 +270,25 @@ jobs:
- name: Check vendored DataFusion proto matches pinned crate
run: ci/scripts/rust_proto_check.sh
+ config-docs-check:
+ name: Check generated config docs are in sync
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/[email protected]
+ with:
+ fetch-depth: 1
+ - name: Install protoc
+ run: |
+ sudo apt-get update
+ sudo apt-get install -y protobuf-compiler
+ - name: Setup Rust toolchain
+ run: |
+ rustup toolchain install stable
+ rustup default stable
+ - uses: swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4
#2.9.1
+ - name: Check generated config docs match the config registry
+ run: ci/scripts/rust_config_docs_check.sh
+
# Coverage job was failing.
https://github.com/apache/arrow-datafusion/issues/590 tracks re-instating it
# coverage:
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index f5aa7f720..13ca1e51a 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -65,6 +65,22 @@ or run them all at once:
- [dev/rust_lint.sh](dev/rust_lint.sh)
+### Configuration Documentation
+
+The configuration tables in
+[docs/source/user-guide/configs.md](docs/source/user-guide/configs.md) are
+generated from the `CONFIG_ENTRIES` registry in `ballista/core/src/config.rs`.
+After adding or changing a configuration key, regenerate them with:
+
+```bash
+./dev/update_config_docs.sh
+```
+
+and commit the result. Every registered key must be covered by one of the
+generated regions in that page, so a new key whose prefix matches none of them
+will fail the check until it is given a home. CI enforces this with
+[ci/scripts/rust_config_docs_check.sh](ci/scripts/rust_config_docs_check.sh).
+
### Python Environment
Refer to the instructions in the Python Bindings [README](./python/README.md)
diff --git a/ballista/core/src/bin/update_config_docs.rs
b/ballista/core/src/bin/update_config_docs.rs
new file mode 100644
index 000000000..b78b0a9f0
--- /dev/null
+++ b/ballista/core/src/bin/update_config_docs.rs
@@ -0,0 +1,835 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Regenerates the marker-delimited configuration tables in
+//! `docs/source/**/*.md` from the `ballista-core` config registry.
+//!
+//! Run via `./dev/update_config_docs.sh`. Pass `--check` to report drift
+//! without writing, which is what CI does.
+
+use ballista_core::config::{BallistaConfig, ConfigEntry};
+use std::collections::BTreeSet;
+use std::fs;
+use std::path::{Path, PathBuf};
+use std::process::ExitCode;
+
+/// Column headers for every generated table.
+const HEADER: [&str; 4] = ["key", "type", "default", "description"];
+
+/// A configuration key flattened into the cells of a docs table row.
+struct Setting {
+ key: String,
+ cells: [String; 4],
+}
+
+/// Escapes text for use inside a markdown table cell. A raw pipe would
+/// terminate the cell early, and a raw newline (or carriage return) would
+/// split the cell across rows; both are collapsed so the table can't be
+/// corrupted by a description that happens to contain either.
+fn escape_cell(text: &str) -> String {
+ text.replace(['\r', '\n'], " ").replace('|', r"\|")
+}
+
+/// Flattens a registry entry into the four cells of a docs table row.
+fn setting_cells(entry: &ConfigEntry) -> [String; 4] {
+ [
+ escape_cell(entry.name()),
+ format!("{:?}", entry.data_type()),
+ escape_cell(entry.doc_default().unwrap_or("(none)")),
+ escape_cell(entry.description()),
+ ]
+}
+
+/// Reads every registered configuration key into renderable form.
+fn settings_from_registry() -> Vec<Setting> {
+ BallistaConfig::valid_entries()
+ .values()
+ .map(|entry| Setting {
+ key: entry.name().to_string(),
+ cells: setting_cells(entry),
+ })
+ .collect()
+}
+
+/// Renders `rows` as a markdown table, padding each cell to the widest cell in
+/// its column so the raw markdown stays readable.
+///
+/// Callers emit a `<!-- prettier-ignore -->` directive ahead of the table, so
+/// this formatting does not need to match prettier's own table algorithm.
+///
+/// Rows are sorted by their first cell. The registry is a `HashMap`, so
+/// without this the output would differ between runs.
+fn render_table(rows: &[[String; 4]]) -> String {
+ let mut rows = rows.to_vec();
+ rows.sort_by(|a, b| a[0].cmp(&b[0]));
+
+ let widths: [usize; 4] = std::array::from_fn(|i| {
+ rows.iter()
+ .map(|row| row[i].chars().count())
+ .chain(std::iter::once(HEADER[i].chars().count()))
+ .max()
+ .unwrap_or(0)
+ });
+
+ let mut out = render_row(&HEADER.map(str::to_string), &widths);
+ out.push_str(&render_separator(&widths));
+ for row in &rows {
+ out.push_str(&render_row(row, &widths));
+ }
+ out
+}
+
+fn render_row(cells: &[String; 4], widths: &[usize; 4]) -> String {
+ let mut line = String::from("|");
+ for (cell, width) in cells.iter().zip(widths) {
+ line.push(' ');
+ line.push_str(cell);
+ line.extend(std::iter::repeat_n(' ', width - cell.chars().count()));
+ line.push_str(" |");
+ }
+ line.push('\n');
+ line
+}
+
+fn render_separator(widths: &[usize; 4]) -> String {
+ let mut line = String::from("|");
+ for width in widths {
+ line.push(' ');
+ line.extend(std::iter::repeat_n('-', *width));
+ line.push_str(" |");
+ }
+ line.push('\n');
+ line
+}
+
+/// Opening marker. The rest of the line may carry a `prefix=` attribute.
+const BEGIN: &str = "<!-- BEGIN GENERATED CONFIG REFERENCE";
+/// Closing marker, matched exactly.
+const END: &str = "<!-- END GENERATED CONFIG REFERENCE -->";
+/// Stops prettier reformatting the generated table. Without it the prettier CI
+/// job and the config-docs CI job would disagree about the table's layout and
+/// neither could pass.
+const PRETTIER_IGNORE: &str = "<!-- prettier-ignore -->";
+
+/// Returns whether `line` opens or closes a fenced code block, i.e. its
+/// trimmed content starts with a ``` or ~~~ fence delimiter. Fences are
+/// recognised regardless of indentation, since a fenced block nested inside
+/// a list item is still a fence as far as markdown rendering is concerned.
+fn is_fence_delimiter(line: &str) -> bool {
+ let trimmed = line.trim();
+ trimmed.starts_with("```") || trimmed.starts_with("~~~")
+}
+
+/// Extracts the key prefixes selected by a `BEGIN` marker line. An empty
+/// result means the region selects every key.
+fn parse_prefixes(line: &str) -> Result<Vec<String>, String> {
+ let attributes = line
+ .strip_prefix(BEGIN)
+ .and_then(|rest| rest.strip_suffix("-->"))
+ .ok_or_else(|| format!("malformed marker: `{line}`"))?
+ .trim();
+
+ if attributes.is_empty() {
+ return Ok(Vec::new());
+ }
+
+ let value = attributes.strip_prefix("prefix=").ok_or_else(|| {
+ format!("unrecognised marker attribute in `{line}`; only `prefix=` is
supported")
+ })?;
+
+ Ok(value
+ .split(',')
+ .map(|prefix| prefix.trim().to_string())
+ .filter(|prefix| !prefix.is_empty())
+ .collect())
+}
+
+/// Rewrites the body of every generated region in `content`.
+///
+/// Returns the new content and the keys rendered into it. Callers union those
+/// key lists across files to check that no configuration key is undocumented.
+fn splice(content: &str, settings: &[Setting]) -> Result<(String,
Vec<String>), String> {
+ let mut out = String::with_capacity(content.len());
+ let mut rendered: Vec<String> = Vec::new();
+ let mut lines = content.lines();
+ // Whether the line currently being scanned is inside a fenced code
+ // block. Markers only trimmed of trailing whitespace (never leading) so
+ // that a marker must start at column 0 to be live; combined with the
+ // fence tracking, this keeps a marker shown as a syntax example -
+ // fenced, or indented inside a list item - from being mistaken for a
+ // real one and silently overwritten or closed early.
+ let mut in_fence = false;
+
+ while let Some(line) = lines.next() {
+ let trimmed = line.trim_end();
+
+ if is_fence_delimiter(line) {
+ in_fence = !in_fence;
+ }
+
+ if !in_fence && trimmed == END {
+ return Err(format!("found `{END}` with no matching BEGIN marker"));
+ }
+
+ out.push_str(line);
+ out.push('\n');
+
+ if in_fence || !trimmed.starts_with(BEGIN) {
+ continue;
+ }
+
+ let prefixes = parse_prefixes(trimmed)?;
+
+ // Discard the old body.
+ let mut closed = false;
+ for inner in lines.by_ref() {
+ let inner_trimmed = inner.trim_end();
+
+ if is_fence_delimiter(inner) {
+ in_fence = !in_fence;
+ }
+
+ if in_fence {
+ continue;
+ }
+
+ if inner_trimmed.starts_with(BEGIN) {
+ return Err(format!(
+ "nested BEGIN marker inside the region opened by
`{trimmed}`"
+ ));
+ }
+ if inner_trimmed == END {
+ closed = true;
+ break;
+ }
+ }
+ if !closed {
+ return Err(format!("unterminated marker: `{trimmed}`"));
+ }
+
+ let selected: Vec<&Setting> = settings
+ .iter()
+ .filter(|setting| {
+ prefixes.is_empty()
+ || prefixes
+ .iter()
+ .any(|prefix| setting.key.starts_with(prefix.as_str()))
+ })
+ .collect();
+
+ if selected.is_empty() {
+ return Err(format!("marker `{trimmed}` matched no configuration
keys"));
+ }
+
+ rendered.extend(selected.iter().map(|setting| setting.key.clone()));
+
+ let rows: Vec<[String; 4]> = selected
+ .iter()
+ .map(|setting| setting.cells.clone())
+ .collect();
+
+ // The prettier-ignore directive keeps prettier from reformatting the
+ // table. The blank line before END is required even so: prettier
+ // inserts one between a table and a following HTML comment.
+ out.push('\n');
+ out.push_str(PRETTIER_IGNORE);
+ out.push('\n');
+ out.push_str(&render_table(&rows));
+ out.push('\n');
+ out.push_str(END);
+ out.push('\n');
+ }
+
+ Ok((out, rendered))
+}
+
+/// Configuration keys that no generated region rendered.
+///
+/// A non-empty result is a hard error: it means a key exists in the registry
+/// but appears in no documentation, which is the drift this tool exists to
+/// prevent.
+fn orphan_keys<'a>(settings: &'a [Setting], covered: &BTreeSet<String>) ->
Vec<&'a str> {
+ let mut orphans: Vec<&str> = settings
+ .iter()
+ .map(|setting| setting.key.as_str())
+ .filter(|key| !covered.contains(*key))
+ .collect();
+ orphans.sort_unstable();
+ orphans
+}
+
+/// Collects every `.md` file under `dir`, recursively.
+fn markdown_files(dir: &Path, out: &mut Vec<PathBuf>) -> std::io::Result<()> {
+ for entry in fs::read_dir(dir)? {
+ let path = entry?.path();
+ if path.is_dir() {
+ markdown_files(&path, out)?;
+ } else if path.extension().is_some_and(|ext| ext == "md") {
+ out.push(path);
+ }
+ }
+ Ok(())
+}
+
+fn run(check: bool) -> Result<(), String> {
+ // Resolved at compile time so the tool works from any working directory.
+ let docs = Path::new(env!("CARGO_MANIFEST_DIR")).join("../../docs/source");
+
+ let mut files = Vec::new();
+ markdown_files(&docs, &mut files)
+ .map_err(|e| format!("failed to walk {}: {e}", docs.display()))?;
+ files.sort();
+
+ let settings = settings_from_registry();
+ let mut covered: BTreeSet<String> = BTreeSet::new();
+ let mut stale: Vec<PathBuf> = Vec::new();
+
+ for file in &files {
+ let content = fs::read_to_string(file)
+ .map_err(|e| format!("failed to read {}: {e}", file.display()))?;
+ if !content.contains(BEGIN) {
+ continue;
+ }
+
+ let (updated, keys) = splice(&content, &settings)
+ .map_err(|e| format!("{}: {e}", file.display()))?;
+ covered.extend(keys);
+
+ if updated != content {
+ stale.push(file.clone());
+ if !check {
+ fs::write(file, updated)
+ .map_err(|e| format!("failed to write {}: {e}",
file.display()))?;
+ // Reported immediately, rather than batched after the loop, so
+ // that a later orphan-key error (which returns before that
+ // batched report) can never leave the caller unaware that
+ // files were already rewritten on disk.
+ println!("updated {}", file.display());
+ }
+ }
+ }
+
+ let orphans = orphan_keys(&settings, &covered);
+ if !orphans.is_empty() {
+ let list = orphans
+ .iter()
+ .map(|key| format!(" - {key}"))
+ .collect::<Vec<_>>()
+ .join("\n");
+ return Err(format!(
+ "these configuration keys are rendered by no generated
region:\n{list}\n\
+ Add them to the marker region for the section they belong in.
Only \
+ widen an existing region's `prefix=` list if that's actually the \
+ right section for them; widening all the way to
`prefix=ballista.` \
+ would dump every key into one table and duplicate the others."
+ ));
+ }
+
+ if check && !stale.is_empty() {
+ let list = stale
+ .iter()
+ .map(|file| format!(" - {}", file.display()))
+ .collect::<Vec<_>>()
+ .join("\n");
+ return Err(format!(
+ "the generated configuration tables are out of date:\n{list}\n\
+ Run ./dev/update_config_docs.sh and commit the result."
+ ));
+ }
+
+ if check {
+ println!("configuration tables are up to date");
+ } else if stale.is_empty() {
+ println!("configuration tables already up to date");
+ }
+ // Else: the "updated {path}" lines were already printed inline above, as
+ // each file was written.
+
+ Ok(())
+}
+
+/// Printed by `--help`/`-h`, and ahead of the error for any unrecognised
+/// argument so a typo (e.g. `--chek`) fails loudly instead of silently
+/// falling back to write mode.
+const USAGE: &str = "\
+usage: update_config_docs [--check]
+
+Regenerates the marker-delimited configuration tables in docs/source
+from the CONFIG_ENTRIES registry in ballista/core/src/config.rs.
+
+options:
+ --check report drift without writing (used by CI)
+ -h, --help print this message and exit
+";
+
+fn main() -> ExitCode {
+ let mut check = false;
+ for arg in std::env::args().skip(1) {
+ match arg.as_str() {
+ "--check" => check = true,
+ "-h" | "--help" => {
+ print!("{USAGE}");
+ return ExitCode::SUCCESS;
+ }
+ other => {
+ eprintln!("error: unrecognised argument `{other}`\n\n{USAGE}");
+ return ExitCode::FAILURE;
+ }
+ }
+ }
+
+ match run(check) {
+ Ok(()) => ExitCode::SUCCESS,
+ Err(message) => {
+ eprintln!("error: {message}");
+ ExitCode::FAILURE
+ }
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ fn row(key: &str, ty: &str, default: &str, description: &str) -> [String;
4] {
+ [
+ key.to_string(),
+ ty.to_string(),
+ default.to_string(),
+ description.to_string(),
+ ]
+ }
+
+ #[test]
+ fn render_table_aligns_columns() {
+ // Every cell is padded to the widest cell in its column, and the
+ // separator cell is exactly that many dashes. This is for readability
+ // of the raw markdown; correctness does not depend on matching any
+ // particular formatter. The first row's key is wider than the "key"
+ // header, which forces the column past its header width.
+ let rows = vec![
+ row("ballista.a.very.long.key.name", "Boolean", "false", "short"),
+ row(
+ "b",
+ "UInt64",
+ "268435456",
+ "a much longer description here that goes on",
+ ),
+ ];
+
+ // Sorted order, not insertion order: "b" is a proper prefix of
+ // "ballista.a.very.long.key.name", and standard lexicographic string
+ // ordering sorts the shorter prefix first (`"b" < "ballista..."`).
+ let expected = "\
+| key | type | default | description
|
+| ----------------------------- | ------- | --------- |
------------------------------------------- |
+| b | UInt64 | 268435456 | a much longer
description here that goes on |
+| ballista.a.very.long.key.name | Boolean | false | short
|
+";
+
+ assert_eq!(render_table(&rows), expected);
+ }
+
+ #[test]
+ fn render_table_sorts_by_key() {
+ // The registry is a HashMap, so unsorted output would never converge.
+ let rows = vec![
+ row("ballista.z", "Utf8", "-", "last"),
+ row("ballista.a", "Utf8", "-", "first"),
+ ];
+
+ let rendered = render_table(&rows);
+ let a = rendered.find("ballista.a").expect("row present");
+ let z = rendered.find("ballista.z").expect("row present");
+ assert!(a < z, "rows should be sorted by key:\n{rendered}");
+ }
+
+ #[test]
+ fn settings_from_registry_covers_every_key() {
+ let settings = settings_from_registry();
+ let keys: Vec<&str> = settings.iter().map(|s|
s.key.as_str()).collect();
+
+ assert_eq!(settings.len(), BallistaConfig::valid_entries().len());
+ assert!(keys.contains(&"ballista.shuffle.sort_based.batch_size"));
+ assert!(keys.contains(&"ballista.testing.chaos_execution.seed"));
+ }
+
+ #[test]
+ fn setting_cells_prefers_doc_default() {
+ let entries = BallistaConfig::valid_entries();
+
+ // The real default is the empty string, which would render as a blank
+ // cell; doc_default supplies something readable instead.
+ let seed = entries
+ .get("ballista.testing.chaos_execution.seed")
+ .expect("entry is registered");
+ assert_eq!(setting_cells(seed)[2], "(empty)");
+
+ // Entries with no override show their real default.
+ let batch_size = entries
+ .get("ballista.shuffle.sort_based.batch_size")
+ .expect("entry is registered");
+ assert_eq!(setting_cells(batch_size)[2], "8192");
+ }
+
+ #[test]
+ fn escape_cell_escapes_pipes() {
+ // A raw pipe would terminate the markdown table cell early.
+ assert_eq!(escape_cell("a | b"), r"a \| b");
+ assert_eq!(escape_cell("no pipes here"), "no pipes here");
+ }
+
+ #[test]
+ fn escape_cell_collapses_newlines() {
+ // A raw newline would split the cell across markdown table rows.
+ assert_eq!(escape_cell("a\nb"), "a b");
+ assert_eq!(escape_cell("a\r\nb"), "a b");
+ assert_eq!(escape_cell("a\rb"), "a b");
+ }
+
+ #[test]
+ fn setting_cells_escapes_pipes_in_descriptions() {
+ // No registered description contains a pipe today. Assert that
+ // invariant directly, so this fails loudly if one is ever added
+ // without going through escape_cell.
+ for entry in BallistaConfig::valid_entries().values() {
+ let description = setting_cells(entry)[3].clone();
+ assert!(
+ !description.contains('|') || description.contains(r"\|"),
+ "unescaped pipe in description of {}: {description}",
+ entry.name()
+ );
+ }
+ }
+
+ #[test]
+ fn setting_cells_never_contain_a_newline() {
+ // All descriptions are single-line today (they use Rust `\` line
+ // continuations), but nothing in the registry's types prevents a
+ // future multi-line description from emitting a literal newline
+ // mid-row, which would silently corrupt the table. Assert the
+ // invariant directly, so this fails loudly the day one is added
+ // without going through escape_cell.
+ for entry in BallistaConfig::valid_entries().values() {
+ for cell in setting_cells(entry) {
+ assert!(
+ !cell.contains('\n') && !cell.contains('\r'),
+ "raw newline in a cell of {}: {cell:?}",
+ entry.name()
+ );
+ }
+ }
+ }
+
+ fn settings() -> Vec<Setting> {
+ vec![
+ Setting {
+ key: "ballista.shuffle.one".to_string(),
+ cells: row("ballista.shuffle.one", "Utf8", "-", "shuffle one"),
+ },
+ Setting {
+ key: "ballista.shuffle.two".to_string(),
+ cells: row("ballista.shuffle.two", "Utf8", "-", "shuffle two"),
+ },
+ Setting {
+ key: "ballista.testing.one".to_string(),
+ cells: row("ballista.testing.one", "Utf8", "-", "testing one"),
+ },
+ ]
+ }
+
+ fn doc(body: &str) -> String {
+ format!("# Title\n\n{body}\ntail\n")
+ }
+
+ #[test]
+ fn splice_fills_an_empty_region() {
+ let content = doc(
+ "<!-- BEGIN GENERATED CONFIG REFERENCE prefix=ballista.testing.
-->\n\
+ <!-- END GENERATED CONFIG REFERENCE -->\n",
+ );
+
+ let (out, keys) = splice(&content, &settings()).expect("splice
succeeds");
+
+ assert_eq!(keys, vec!["ballista.testing.one".to_string()]);
+ assert!(out.contains("| ballista.testing.one |"), "got:\n{out}");
+ assert!(!out.contains("ballista.shuffle.one"), "got:\n{out}");
+ // Prose outside the region survives.
+ assert!(out.starts_with("# Title\n"), "got:\n{out}");
+ assert!(out.ends_with("tail\n"), "got:\n{out}");
+ }
+
+ #[test]
+ fn splice_emits_prettier_ignore_and_blank_lines() {
+ // Without the directive the prettier CI job and the config-docs CI job
+ // would fight over the table's formatting and neither could pass. The
+ // blank line before END is required even with the directive: prettier
+ // inserts one there regardless.
+ let content = doc(
+ "<!-- BEGIN GENERATED CONFIG REFERENCE prefix=ballista.testing.
-->\n\
+ <!-- END GENERATED CONFIG REFERENCE -->\n",
+ );
+
+ let (out, _) = splice(&content, &settings()).expect("splice succeeds");
+
+ assert!(
+ out.contains(
+ "<!-- BEGIN GENERATED CONFIG REFERENCE
prefix=ballista.testing. -->\n\n<!-- prettier-ignore -->\n| key"
+ ),
+ "missing blank line or prettier-ignore after BEGIN:\n{out}"
+ );
+ assert!(
+ out.contains(" |\n\n<!-- END GENERATED CONFIG REFERENCE -->"),
+ "missing blank line before END:\n{out}"
+ );
+ }
+
+ #[test]
+ fn splice_is_idempotent() {
+ let content = doc(
+ "<!-- BEGIN GENERATED CONFIG REFERENCE prefix=ballista.shuffle.
-->\n\
+ <!-- END GENERATED CONFIG REFERENCE -->\n",
+ );
+
+ let (once, _) = splice(&content, &settings()).expect("first splice");
+ let (twice, _) = splice(&once, &settings()).expect("second splice");
+
+ assert_eq!(once, twice, "splice must converge");
+ }
+
+ #[test]
+ fn splice_replaces_stale_content() {
+ let content = doc(
+ "<!-- BEGIN GENERATED CONFIG REFERENCE prefix=ballista.testing.
-->\n\n\
+ | key | type |\n| --- | ---- |\n| ballista.gone | Utf8 |\n\n\
+ <!-- END GENERATED CONFIG REFERENCE -->\n",
+ );
+
+ let (out, _) = splice(&content, &settings()).expect("splice succeeds");
+
+ assert!(!out.contains("ballista.gone"), "stale row survived:\n{out}");
+ assert!(out.contains("ballista.testing.one"), "got:\n{out}");
+ }
+
+ #[test]
+ fn splice_handles_multiple_regions() {
+ let content = doc(
+ "<!-- BEGIN GENERATED CONFIG REFERENCE prefix=ballista.shuffle.
-->\n\
+ <!-- END GENERATED CONFIG REFERENCE -->\n\n\
+ ## Testing\n\n\
+ <!-- BEGIN GENERATED CONFIG REFERENCE prefix=ballista.testing.
-->\n\
+ <!-- END GENERATED CONFIG REFERENCE -->\n",
+ );
+
+ let (out, mut keys) = splice(&content, &settings()).expect("splice
succeeds");
+ keys.sort();
+
+ assert_eq!(
+ keys,
+ vec![
+ "ballista.shuffle.one".to_string(),
+ "ballista.shuffle.two".to_string(),
+ "ballista.testing.one".to_string(),
+ ]
+ );
+ assert!(
+ out.contains("## Testing"),
+ "prose between regions lost:\n{out}"
+ );
+ }
+
+ #[test]
+ fn splice_ignores_marker_pair_inside_fenced_code_block() {
+ // A marker pair shown as a syntax example inside a fenced code block
+ // must not be mistaken for a live region: there is nothing to
+ // render into, so the fenced block passes through untouched and no
+ // keys are reported.
+ let content = doc("```\n\
+ <!-- BEGIN GENERATED CONFIG REFERENCE prefix=ballista.testing.
-->\n\
+ <!-- END GENERATED CONFIG REFERENCE -->\n\
+ ```\n");
+
+ let (out, keys) = splice(&content, &settings()).expect("splice
succeeds");
+
+ assert!(
+ keys.is_empty(),
+ "fenced markers must not be treated as live: {keys:?}"
+ );
+ assert_eq!(out, content, "fenced block must be left untouched");
+ }
+
+ #[test]
+ fn splice_ignores_indented_begin_marker() {
+ // A BEGIN marker that does not start at column 0 - e.g. indented
+ // inside a list item as a syntax example - is not live. With no
+ // live BEGIN to match it, the following unindented END is an
+ // orphan.
+ let content = doc(
+ " <!-- BEGIN GENERATED CONFIG REFERENCE prefix=ballista.testing.
-->\n\
+ <!-- END GENERATED CONFIG REFERENCE -->\n",
+ );
+
+ let err = splice(&content, &settings()).expect_err("must fail");
+
+ assert!(err.contains("no matching BEGIN"), "got: {err}");
+ }
+
+ #[test]
+ fn splice_handles_live_region_alongside_fenced_marker_example() {
+ // A fenced block documenting the marker syntax elsewhere in the
+ // same file must not interfere with a genuine, unindented live
+ // region.
+ let content = doc("Example syntax:\n\n\
+ ```\n\
+ <!-- BEGIN GENERATED CONFIG REFERENCE prefix=... -->\n\
+ <!-- END GENERATED CONFIG REFERENCE -->\n\
+ ```\n\n\
+ <!-- BEGIN GENERATED CONFIG REFERENCE prefix=ballista.testing.
-->\n\
+ <!-- END GENERATED CONFIG REFERENCE -->\n");
+
+ let (out, keys) = splice(&content, &settings()).expect("splice
succeeds");
+
+ assert_eq!(keys, vec!["ballista.testing.one".to_string()]);
+ assert!(out.contains("| ballista.testing.one |"), "got:\n{out}");
+ assert!(
+ out.contains("<!-- BEGIN GENERATED CONFIG REFERENCE prefix=...
-->"),
+ "fenced example must survive untouched:\n{out}"
+ );
+ }
+
+ #[test]
+ fn splice_fenced_end_does_not_close_a_live_region_early() {
+ // The region-body-consuming inner loop must also track fence state:
+ // an END shown inside a fenced example within a live region must
+ // not close that region early.
+ let content = doc(
+ "<!-- BEGIN GENERATED CONFIG REFERENCE prefix=ballista.testing.
-->\n\
+ Example of the closing marker:\n\n\
+ ```\n\
+ <!-- END GENERATED CONFIG REFERENCE -->\n\
+ ```\n\n\
+ <!-- END GENERATED CONFIG REFERENCE -->\n",
+ );
+
+ let (out, keys) = splice(&content, &settings()).expect("splice
succeeds");
+
+ assert_eq!(keys, vec!["ballista.testing.one".to_string()]);
+ assert!(out.contains("| ballista.testing.one |"), "got:\n{out}");
+ // The old body, including the fenced example, was discarded along
+ // with the rest of the stale region.
+ assert!(
+ !out.contains("Example of the closing marker"),
+ "got:\n{out}"
+ );
+ }
+
+ #[test]
+ fn splice_without_prefix_selects_every_key() {
+ let content = doc("<!-- BEGIN GENERATED CONFIG REFERENCE -->\n\
+ <!-- END GENERATED CONFIG REFERENCE -->\n");
+
+ let (_, keys) = splice(&content, &settings()).expect("splice
succeeds");
+
+ assert_eq!(keys.len(), 3);
+ }
+
+ #[test]
+ fn splice_rejects_unterminated_region() {
+ let content = doc("<!-- BEGIN GENERATED CONFIG REFERENCE -->\n");
+
+ let err = splice(&content, &settings()).expect_err("must fail");
+
+ assert!(err.contains("unterminated"), "got: {err}");
+ }
+
+ #[test]
+ fn splice_rejects_nested_begin() {
+ let content = doc("<!-- BEGIN GENERATED CONFIG REFERENCE -->\n\
+ <!-- BEGIN GENERATED CONFIG REFERENCE -->\n\
+ <!-- END GENERATED CONFIG REFERENCE -->\n");
+
+ let err = splice(&content, &settings()).expect_err("must fail");
+
+ assert!(err.contains("nested"), "got: {err}");
+ }
+
+ #[test]
+ fn splice_rejects_orphan_end() {
+ let content = doc("<!-- END GENERATED CONFIG REFERENCE -->\n");
+
+ let err = splice(&content, &settings()).expect_err("must fail");
+
+ assert!(err.contains("no matching BEGIN"), "got: {err}");
+ }
+
+ #[test]
+ fn splice_rejects_prefix_matching_nothing() {
+ // Catches typos in a prefix attribute.
+ let content = doc(
+ "<!-- BEGIN GENERATED CONFIG REFERENCE prefix=ballista.typo. -->\n\
+ <!-- END GENERATED CONFIG REFERENCE -->\n",
+ );
+
+ let err = splice(&content, &settings()).expect_err("must fail");
+
+ assert!(err.contains("matched no configuration keys"), "got: {err}");
+ }
+
+ #[test]
+ fn splice_rejects_unknown_attribute() {
+ let content = doc(
+ "<!-- BEGIN GENERATED CONFIG REFERENCE suffix=ballista. -->\n\
+ <!-- END GENERATED CONFIG REFERENCE -->\n",
+ );
+
+ let err = splice(&content, &settings()).expect_err("must fail");
+
+ assert!(err.contains("unrecognised marker attribute"), "got: {err}");
+ }
+
+ #[test]
+ fn parse_prefixes_reads_a_comma_separated_list() {
+ let prefixes = parse_prefixes(
+ "<!-- BEGIN GENERATED CONFIG REFERENCE
prefix=ballista.a.,ballista.b. -->",
+ )
+ .expect("parses");
+
+ assert_eq!(prefixes, vec!["ballista.a.", "ballista.b."]);
+ }
+
+ #[test]
+ fn orphan_keys_reports_undocumented_settings() {
+ let settings = settings();
+ let mut covered = std::collections::BTreeSet::new();
+ covered.insert("ballista.shuffle.one".to_string());
+ covered.insert("ballista.shuffle.two".to_string());
+
+ assert_eq!(
+ orphan_keys(&settings, &covered),
+ vec!["ballista.testing.one"]
+ );
+ }
+
+ #[test]
+ fn orphan_keys_is_empty_when_all_covered() {
+ let settings = settings();
+ let covered: std::collections::BTreeSet<String> =
+ settings.iter().map(|s| s.key.clone()).collect();
+
+ assert!(orphan_keys(&settings, &covered).is_empty());
+ }
+}
diff --git a/ballista/core/src/config.rs b/ballista/core/src/config.rs
index 63196a95b..fd3060297 100644
--- a/ballista/core/src/config.rs
+++ b/ballista/core/src/config.rs
@@ -170,8 +170,9 @@ static CONFIG_ENTRIES: LazyLock<HashMap<String,
ConfigEntry>> = LazyLock::new(||
"Sets the job name that will appear in the web user
interface for any submitted jobs".to_string(),
DataType::Utf8, None),
ConfigEntry::new(BALLISTA_STANDALONE_PARALLELISM.to_string(),
- "Standalone processing parallelism ".to_string(),
- DataType::UInt16,
Some(std::thread::available_parallelism().map(|v|
v.get()).unwrap_or(1).to_string())),
+ "Number of concurrent tasks a standalone in-process
executor will run.".to_string(),
+ DataType::UInt16,
Some(std::thread::available_parallelism().map(|v|
v.get()).unwrap_or(1).to_string()))
+ .with_doc_default("number of available CPU cores"),
ConfigEntry::new(BALLISTA_CACHE_NOOP.to_string(),
"Disable default cache node extension".to_string(),
DataType::Boolean,
@@ -296,13 +297,14 @@ static CONFIG_ENTRIES: LazyLock<HashMap<String,
ConfigEntry>> = LazyLock::new(||
DataType::Boolean,
Some(false.to_string())),
ConfigEntry::new(BALLISTA_PROPAGATE_EMPTY_ENABLED.to_string(),
- "Configuration key to enable AQE propagate empty exec
rule. \
- This could benefit the workload by injecting EmptyExec
in the plan (i.e during joins)".to_string(),
+ "Enables the AQE propagate-empty-relation rule.
Injects EmptyExec \
+ into the plan where an input is known to be empty,
such as one side \
+ of a join, allowing downstream work to be
skipped.".to_string(),
DataType::Boolean,
Some(true.to_string())),
ConfigEntry::new(
BALLISTA_COALESCE_TARGET_PARTITION_BYTES.to_string(),
- "Target post-coalesce partition byte size in bytes. Mirrors
Spark's \
+ "Target post-coalesce partition size in bytes. Mirrors Spark's \
advisoryPartitionSizeInBytes."
.to_string(),
DataType::UInt64,
@@ -310,13 +312,19 @@ static CONFIG_ENTRIES: LazyLock<HashMap<String,
ConfigEntry>> = LazyLock::new(||
),
ConfigEntry::new(
BALLISTA_COALESCE_SMALL_PARTITION_FACTOR.to_string(),
- "Small-partition merge factor (Spark legacy).".to_string(),
+ "A coalesced partition smaller than target_partition_bytes times
this \
+ factor counts as small and is merged into its neighbour. Mirrors \
+ Spark's legacy coalesce semantics."
+ .to_string(),
DataType::Float64,
Some("0.2".to_string()),
),
ConfigEntry::new(
BALLISTA_COALESCE_MERGED_PARTITION_FACTOR.to_string(),
- "Merged-partition early-flush factor (Spark legacy).".to_string(),
+ "Two adjacent partitions are merged when their combined size is
below \
+ target_partition_bytes times this factor. Mirrors Spark's legacy \
+ coalesce semantics."
+ .to_string(),
DataType::Float64,
Some("1.2".to_string()),
),
@@ -360,7 +368,8 @@ static CONFIG_ENTRIES: LazyLock<HashMap<String,
ConfigEntry>> = LazyLock::new(||
to get reproducible fault injection across runs.".to_string(),
DataType::Utf8,
Some("".to_string()),
- ),
+ )
+ .with_doc_default("(empty)"),
ConfigEntry::new(
BALLISTA_SHUFFLE_COMPRESSION_CODEC.to_string(),
"Compression codec specification \
@@ -396,6 +405,7 @@ pub struct ConfigEntry {
description: String,
data_type: DataType,
default_value: Option<String>,
+ doc_default: Option<String>,
}
impl ConfigEntry {
@@ -410,8 +420,45 @@ impl ConfigEntry {
description,
data_type,
default_value,
+ doc_default: None,
}
}
+
+ /// Overrides the value shown as this setting's default in generated
+ /// documentation. Use it when the real default is machine-dependent or
+ /// otherwise unreadable, so that generated docs stay reproducible.
+ fn with_doc_default(mut self, doc_default: impl Into<String>) -> Self {
+ self.doc_default = Some(doc_default.into());
+ self
+ }
+
+ /// The configuration key, for example `ballista.job.name`.
+ pub fn name(&self) -> &str {
+ &self.name
+ }
+
+ /// Human-readable description of what this setting controls.
+ pub fn description(&self) -> &str {
+ &self.description
+ }
+
+ /// The type that a value for this setting must parse as.
+ pub fn data_type(&self) -> &DataType {
+ &self.data_type
+ }
+
+ /// The default value as a string, or `None` when the setting is optional
+ /// with no default.
+ pub fn default_value(&self) -> Option<&str> {
+ self.default_value.as_deref()
+ }
+
+ /// The default value as it should appear in generated documentation.
+ /// Falls back to [`ConfigEntry::default_value`] when no documentation
+ /// override was set.
+ pub fn doc_default(&self) -> Option<&str> {
+ self.doc_default.as_deref().or(self.default_value())
+ }
}
/// Ballista configuration
@@ -902,4 +949,38 @@ mod tests {
64 * 1024 * 1024
);
}
+
+ #[test]
+ fn config_entry_exposes_metadata() {
+ let entries = BallistaConfig::valid_entries();
+ let entry = entries
+ .get(BALLISTA_SHUFFLE_SORT_BASED_BATCH_SIZE)
+ .expect("entry is registered");
+
+ assert_eq!(entry.name(), "ballista.shuffle.sort_based.batch_size");
+ assert_eq!(entry.data_type(), &DataType::UInt64);
+ assert_eq!(entry.default_value(), Some("8192"));
+ assert!(entry.description().contains("batch size"));
+ }
+
+ #[test]
+ fn doc_default_overrides_machine_dependent_default() {
+ let entries = BallistaConfig::valid_entries();
+
+ // This entry's real default is available_parallelism(), which varies
by
+ // machine. Generated docs must show a stable string instead.
+ let parallelism = entries
+ .get(BALLISTA_STANDALONE_PARALLELISM)
+ .expect("entry is registered");
+ assert_eq!(
+ parallelism.doc_default(),
+ Some("number of available CPU cores")
+ );
+
+ // Entries without an override fall back to the real default.
+ let batch_size = entries
+ .get(BALLISTA_SHUFFLE_SORT_BASED_BATCH_SIZE)
+ .expect("entry is registered");
+ assert_eq!(batch_size.doc_default(), Some("8192"));
+ }
}
diff --git a/ci/scripts/rust_config_docs_check.sh
b/ci/scripts/rust_config_docs_check.sh
new file mode 100755
index 000000000..abdd8a3ee
--- /dev/null
+++ b/ci/scripts/rust_config_docs_check.sh
@@ -0,0 +1,23 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Fail if the generated configuration tables in docs/source drift from the
+# ballista-core config registry, or if a registered key is documented nowhere.
+set -ex
+dev/update_config_docs.sh --check
diff --git a/dev/update_config_docs.sh b/dev/update_config_docs.sh
new file mode 100755
index 000000000..e709b6c4e
--- /dev/null
+++ b/dev/update_config_docs.sh
@@ -0,0 +1,23 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Regenerate the marker-delimited configuration tables in docs/source from the
+# ballista-core config registry. Pass --check to report drift without writing.
+set -e
+cargo run --profile ci -p ballista-core --bin update_config_docs -- "$@"
diff --git a/docs/source/user-guide/configs.md
b/docs/source/user-guide/configs.md
index 8bc7276f4..c40f1a582 100644
--- a/docs/source/user-guide/configs.md
+++ b/docs/source/user-guide/configs.md
@@ -19,6 +19,14 @@
# Configuration
+**The configuration tables on this page are generated** from the `ConfigEntry`
+definitions in `ballista/core/src/config.rs`. Anything between a pair of
+generated config reference markers is overwritten the next time the docs are
+regenerated, so do not hand-edit it. To change a setting's description, type,
or
+default, edit its `ConfigEntry` in `ballista/core/src/config.rs`, then run
+`./dev/update_config_docs.sh` and commit the regenerated file. Prose outside
the
+generated sections is hand-written and safe to edit normally.
+
## Ballista Configuration Settings
Configuring Ballista is quite similar to configuring DataFusion. Most settings
are identical, with only a few configurations specific to Ballista.
@@ -88,19 +96,84 @@ let expected = [
];
```
+### Session Settings
+
+The following keys can be set on a `SessionConfig`. Ballista also accepts all
+standard DataFusion settings.
+
+<!-- BEGIN GENERATED CONFIG REFERENCE
prefix=ballista.job.,ballista.standalone.,ballista.cache.,ballista.client.,ballista.optimizer.,ballista.planner.,ballista.scheduler.
-->
+
+<!-- prettier-ignore -->
+| key | type | default
| description
[...]
+| ------------------------------------------------------ | ------- |
----------------------------- |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[...]
+| ballista.cache.noop | Boolean | true
| Disable default cache node extension
[...]
+| ballista.client.grpc_connect_timeout_seconds | UInt64 | 20
| Connection timeout for gRPC client in seconds
[...]
+| ballista.client.grpc_http2_keepalive_interval_seconds | UInt64 | 300
| HTTP/2 keep-alive interval for gRPC client in seconds
[...]
+| ballista.client.grpc_max_message_size | UInt64 | 16777216
| Configuration for max message size in gRPC clients
[...]
+| ballista.client.grpc_tcp_keepalive_seconds | UInt64 | 3600
| TCP keep-alive interval for gRPC client in seconds
[...]
+| ballista.client.grpc_timeout_seconds | UInt64 | 20
| Request timeout for gRPC client in seconds
[...]
+| ballista.client.initial_connection_window_size | UInt64 | 67108864
| HTTP/2 initial connection-level flow-control window for
gRPC data-plane clients, in bytes. Should be >= the shuffle governor byte
budget so the governor, not the transport window, is the binding backpressure.
0 leaves the tonic default.
[...]
+| ballista.client.initial_stream_window_size | UInt64 | 16777216
| HTTP/2 initial stream-level flow-control window for gRPC
data-plane clients, in bytes. 0 leaves the tonic default.
[...]
+| ballista.client.io_retries_times | UInt16 | 3
| Number of retries for IO operations in the Ballista
client.
[...]
+| ballista.client.io_retry_wait_time_ms | UInt64 | 3000
| Wait time in milliseconds between IO retries in the
Ballista client.
[...]
+| ballista.client.pull | Boolean | false
| Should client employ pull or push job tracking. In pull
mode client will make a request to server in the loop, until job finishes. Pull
mode is kept for legacy clients.
[...]
+| ballista.client.use_tls | Boolean | false
| Should connection between client, scheduler, and
executors use TLS.
[...]
+| ballista.job.name | Utf8 | (none)
| Sets the job name that will appear in the web user
interface for any submitted jobs
[...]
+| ballista.optimizer.broadcast_join_threshold_bytes | UInt64 | 10485760
| Byte-size threshold below which a hash join's smaller
side is promoted to CollectLeft and lowered via the broadcast pattern. Governs
broadcast selection under both the static distributed planner and adaptive
query planning (AQE). Set to 0 to disable promotion.
[...]
+| ballista.optimizer.broadcast_join_threshold_rows | UInt64 | 1000000
| Row-count threshold below which a hash join's smaller
side is promoted to CollectLeft and lowered via the broadcast pattern, used as
a fallback when byte-size statistics are unavailable. Applies to adaptive query
planning (AQE). Set to 0 to disable promotion via the row-count path.
[...]
+| ballista.optimizer.hash_join_max_build_partition_bytes | UInt64 | 67108864
| Maximum per-partition hash-join build-side bytes for a
Partitioned hash join under AQE. A build partition larger than this falls back
to SortMergeJoin (spillable). Defaults to 64 MiB; 0 disables the check, which
makes AQE use a hash join regardless of build size.
[...]
+| ballista.planner.adaptive.enabled | Boolean | false
| Enables Adaptive Query Planning (EXPERIMENTAL)
[...]
+| ballista.planner.adaptive_join.enabled | Boolean | true
| Enables the AQE dynamic join-selection rule
(SelectJoinRule). When true (default), DynamicJoinSelectionExec nodes are
resolved to concrete HashJoin or CollectLeft join implementations at runtime.
Disable only for debugging.
[...]
+| ballista.planner.coalesce.enabled | Boolean | false
| Enables the AQE coalesce-shuffle-partitions rule.
Disabled by default — opt in when fewer/larger downstream tasks matter more
than parallelism.
[...]
+| ballista.planner.coalesce.merged_partition_factor | Float64 | 1.2
| Two adjacent partitions are merged when their combined
size is below target_partition_bytes times this factor. Mirrors Spark's legacy
coalesce semantics.
[...]
+| ballista.planner.coalesce.small_partition_factor | Float64 | 0.2
| A coalesced partition smaller than target_partition_bytes
times this factor counts as small and is merged into its neighbour. Mirrors
Spark's legacy coalesce semantics.
[...]
+| ballista.planner.coalesce.target_partition_bytes | UInt64 | 67108864
| Target post-coalesce partition size in bytes. Mirrors
Spark's advisoryPartitionSizeInBytes.
[...]
+| ballista.planner.propagate_empty.enabled | Boolean | true
| Enables the AQE propagate-empty-relation rule. Injects
EmptyExec into the plan where an input is known to be empty, such as one side
of a join, allowing downstream work to be skipped.
[...]
+| ballista.scheduler.max_partitions_per_task | UInt64 | 1
| Upper bound on the number of input partitions packed into
a single task's `partition_slice`. `1` (default) means one task per input
partition. Raise to enable multi-partition tasks (fewer tasks, parallel-sort /
parallel-join wins); `0` means unbounded — the scheduler fills each task up to
the executor's free vcore count. Does not apply to collapse stages, which must
pack their full pendin [...]
+| ballista.standalone.parallelism | UInt16 | number of
available CPU cores | Number of concurrent tasks a standalone in-process
executor will run.
[...]
+
+<!-- END GENERATED CONFIG REFERENCE -->
+
## Shuffle Settings
The following session-level keys control Ballista's shuffle behavior. See
the [tuning guide](tuning-guide.md#shuffle-implementation) for an
explanation of the sort-based shuffle writer.
-| key | type | default
| description
|
-| ------------------------------------------------------- | ------- |
--------- |
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|
-| ballista.shuffle.max_concurrent_read_requests | UInt64 | 64
| Maximum number of concurrent fetch requests the shuffle reader will issue.
|
-| ballista.shuffle.force_remote_read | Boolean | false
| Forces the shuffle reader to fetch every partition through Arrow Flight,
even when the data is local. Intended for testing.
|
-| ballista.shuffle.remote_read_prefer_flight | Boolean | false
| For remote reads, prefer the Arrow Flight reader over the block reader. The
block reader is generally faster.
|
-| ballista.shuffle.sort_based.batch_size | UInt64 | 8192
| Target row count when coalescing buffered batches before they are written or
spilled.
|
-| ballista.shuffle.sort_based.memory_limit_per_task_bytes | UInt64 |
268435456 | Per-task buffered-bytes budget at which the sort-based writer
spills to disk (256 MiB default), counted independently of the runtime memory
pool. Set to `0` to disable the per-task budget and spill only under
memory-pool pressure — safe only with a bounded memory pool, otherwise the
writer never spills and may run out of memory. |
+<!-- BEGIN GENERATED CONFIG REFERENCE prefix=ballista.shuffle. -->
+
+<!-- prettier-ignore -->
+| key | type | default
| description
[...]
+| -------------------------------------------------------- | ------- |
--------- |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[...]
+| ballista.shuffle.compression.codec | Utf8 | lz4
| Compression codec specification used in the shuffle process. Possible
values: none, lz4, zstd. Defaults to lz4 to preserve current behaviour
[...]
+| ballista.shuffle.force_remote_read | Boolean | false
| Forces the shuffle reader to always read partitions via the Arrow Flight
client, even when partitions are local to the node.
[...]
+| ballista.shuffle.max_concurrent_read_requests | UInt64 | 64
| Maximum concurrent requests shuffle reader can process
[...]
+| ballista.shuffle.reader.default_block_size_bytes | UInt64 | 1048576
| Assumed per-partition byte size charged to the shuffle governor when
partition stats carry no byte count.
[...]
+| ballista.shuffle.reader.max_blocks_in_flight_per_address | UInt64 | 128
| Reduce-side shuffle governor: maximum concurrent in-flight partition
fetches to a single executor address.
[...]
+| ballista.shuffle.reader.max_bytes_in_flight | UInt64 |
50331648 | Reduce-side shuffle governor: maximum total in-flight bytes across
concurrent remote partition fetches. Mirrors Spark's
spark.reducer.maxSizeInFlight. Values above 4 GiB are clamped to 4 GiB (u32
semaphore limit).
[...]
+| ballista.shuffle.remote_read_prefer_flight | Boolean | false
| Forces the shuffle reader to use flight reader instead of block reader for
remote read. Block reader usually has better performance and resource
utilization
[...]
+| ballista.shuffle.sort_based.batch_size | UInt64 | 8192
| Target batch size in rows for coalescing small batches in sort shuffle
[...]
+| ballista.shuffle.sort_based.memory_limit_per_task_bytes | UInt64 |
268435456 | Per-task buffered-bytes budget at which the sort shuffle writer
spills its in-memory batches to disk. Counted independently of the runtime
memory pool, so spilling kicks in even when the pool is unbounded. Total
worst-case sort shuffle memory per executor is approximately vcores * this
value. Set to 0 to disable the per-task budget and rely solely on runtime
memory-pool pressure to trigger spilling; this i [...]
+| ballista.shuffle.writer_channel_capacity | UInt32 | 8
| Bounded channel capacity for async-to-blocking I/O bridge in shuffle writer
[...]
+
+<!-- END GENERATED CONFIG REFERENCE -->
+
+## Testing and Fault Injection
+
+These settings drive chaos-monkey fault injection and exist for robustness
+testing. They are not intended for production use.
+
+<!-- BEGIN GENERATED CONFIG REFERENCE prefix=ballista.testing. -->
+
+<!-- prettier-ignore -->
+| key | type | default |
description
|
+| -------------------------------------------- | ------- | --------- |
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|
+| ballista.testing.chaos_execution.enabled | Boolean | false | Enables
chaos-monkey execution injection for robustness testing. When true, ChaosExec
is inserted at a random point in the plan once per optimize call.
|
+| ballista.testing.chaos_execution.fault_type | Utf8 | transient | Fault
type injected by chaos-monkey execution. "transient": IoError (retryable);
"fatal": Execution error (non-retryable); "panic": panics the task thread;
"delay" or "delay:N": sleeps N ms per batch with no error (default N=1). |
+| ballista.testing.chaos_execution.probability | Float64 | 0.25 | Failure
probability (0.0–1.0) passed to ChaosExec when chaos execution is enabled.
|
+| ballista.testing.chaos_execution.seed | Utf8 | (empty) |
Optional u64 seed for the chaos RNG. Empty string (default) means
non-deterministic; set to a numeric value to get reproducible fault injection
across runs.
|
+
+<!-- END GENERATED CONFIG REFERENCE -->
## Ballista Scheduler Configuration Settings
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
