This is an automated email from the ASF dual-hosted git repository.
andygrove pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-ballista.git
The following commit(s) were added to refs/heads/main by this push:
new 1fe9e6f70 test(chaos): add a Kubernetes (kind) backend for the chaos
harness (#2244)
1fe9e6f70 is described below
commit 1fe9e6f707653ff4e45f64371cf6b9e014f16a5b
Author: Akshay Chitneni <[email protected]>
AuthorDate: Sat Aug 15 08:32:12 2026 -0700
test(chaos): add a Kubernetes (kind) backend for the chaos harness (#2244)
* test(chaos): add a Kubernetes (kind) backend for the chaos harness
Adds an opt-in Kubernetes backend so the chaos scenarios can run on a real
kind cluster, alongside the existing multi-process harness. The scheduler
runs
as a Deployment behind a ClusterIP Service and executors as a labelled
Deployment; the harness reaches the scheduler over a kubectl port-forward
and
fetches results through the scheduler's embedded flight proxy, so it never
contacts executor pod IPs.
Gated behind the 'k8s' cargo feature and CHAOS_BACKEND=kind, so the default
cargo test and the process harness are unchanged.
Includes:
- src/k8s.rs: K8sCluster backend (kubectl-driven) with scale/kill verbs
wired
for future kill scenarios, diagnostics-on-timeout, and CHAOS_KEEP_NS.
- tests/k8s.rs: baseline scenario asserting the cluster result matches local
DataFusion.
- chaos bins: host/bind/flight-proxy env wiring for running in pods.
- dev/docker/chaos.Dockerfile (+ build script) compiled in-image so the
binary
matches the container architecture.
- dev/chaos-kind.sh runbook script and a gated k8s-chaos CI workflow.
* test(chaos): address PR #2244 review feedback
- CI: install kind+kubectl via pinned curl (drop non-allowlisted actions);
permissions: contents: read; keep namespaces on failure for log
collection;
fix Swatinem casing.
- Backend: unique namespace per start(); refuse non-kind kube-context; clear
shared-dir contents on start; scheduler /healthz probes; imagePullPolicy
Never.
- Image: strip symbols + no debuginfo (fixes ld OOM at link, shrinks image).
- Scripts: /usr/bin/env bash shebangs.
---------
Co-authored-by: Akshay Chitneni <[email protected]>
Co-authored-by: Martin Grigorov <[email protected]>
---
.github/workflows/k8s-chaos.yml | 116 ++++++
chaos-testing/Cargo.toml | 6 +
chaos-testing/README.md | 54 +++
chaos-testing/k8s/kind-config.yaml | 39 ++
chaos-testing/src/bin/chaos-executor.rs | 42 ++-
chaos-testing/src/bin/chaos-scheduler.rs | 15 +-
chaos-testing/src/k8s.rs | 629 +++++++++++++++++++++++++++++++
chaos-testing/src/lib.rs | 2 +
chaos-testing/tests/k8s.rs | 113 ++++++
dev/build-chaos-docker.sh | 31 ++
dev/chaos-kind.sh | 171 +++++++++
dev/docker/chaos.Dockerfile | 58 +++
12 files changed, 1272 insertions(+), 4 deletions(-)
diff --git a/.github/workflows/k8s-chaos.yml b/.github/workflows/k8s-chaos.yml
new file mode 100644
index 000000000..92147aefe
--- /dev/null
+++ b/.github/workflows/k8s-chaos.yml
@@ -0,0 +1,116 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Kubernetes (kind) chaos scenarios. Gated: runs only when the chaos harness,
+# the engine, or the docker artifacts change, plus a nightly cron schedule and
+# on demand — it is deliberately off the per-PR critical path, since the
+# process-based chaos harness already covers HA on every PR.
+name: k8s-chaos
+
+on:
+ pull_request:
+ paths:
+ - "chaos-testing/**"
+ - "ballista/**"
+ - "Cargo.toml"
+ - "Cargo.lock"
+ - "rust-toolchain.toml"
+ - "dev/docker/chaos.Dockerfile"
+ - "dev/build-chaos-docker.sh"
+ - ".github/workflows/k8s-chaos.yml"
+ # docs-only changes cannot affect this job; exclusions are applied last
+ - "!**/*.md"
+ schedule:
+ # Nightly cron (not the Rust nightly toolchain — this job builds with
stable).
+ - cron: "0 6 * * *"
+ workflow_dispatch:
+
+concurrency:
+ group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{
github.workflow }}
+ cancel-in-progress: true
+
+# Minimal token: this job only reads the repository.
+permissions:
+ contents: read
+
+env:
+ # Pinned tool versions, installed via curl to avoid marketplace actions that
+ # must be on the ASF allowlist.
+ KIND_VERSION: v0.30.0
+ KUBECTL_VERSION: v1.31.4
+ CLUSTER_NAME: ballista-chaos
+ CHAOS_FIXTURE_DIR: /tmp/ballista-chaos-fixtures
+
+jobs:
+ k8s-chaos:
+ name: kind chaos scenarios
+ runs-on: ubuntu-latest
+ timeout-minutes: 45
+ steps:
+ - name: Checkout
+ uses: actions/[email protected]
+ with:
+ fetch-depth: 1
+
+ - name: Install protoc
+ run: |
+ sudo apt-get update
+ sudo apt-get install -y protobuf-compiler
+
+ - name: Setup Rust toolchain
+ run: |
+ rustup update stable
+ rustup toolchain install stable
+ rustup default stable
+
+ - uses: swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4
#2.9.1
+
+ - name: Install kind and kubectl
+ run: |
+ curl -fsSLo ./kind
"https://kind.sigs.k8s.io/dl/${KIND_VERSION}/kind-linux-amd64";
+ curl -fsSLo ./kubectl
"https://dl.k8s.io/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl";
+ chmod +x ./kind ./kubectl
+ sudo mv ./kind ./kubectl /usr/local/bin/
+
+ - name: Create kind cluster
+ run: |
+ mkdir -p "${CHAOS_FIXTURE_DIR}"
+ kind create cluster --name "${CLUSTER_NAME}" --config
chaos-testing/k8s/kind-config.yaml
+
+ - name: Build the chaos image
+ run: ./dev/build-chaos-docker.sh
+
+ - name: Load the chaos image into kind
+ run: kind load docker-image ballista-chaos:test --name
"${CLUSTER_NAME}"
+
+ - name: Run the kind chaos scenarios
+ env:
+ CHAOS_BACKEND: kind
+ # Keep namespaces on failure so the diagnostics step below can
collect
+ # logs — teardown would otherwise delete them. The runner is
ephemeral.
+ CHAOS_KEEP_NS: "1"
+ run: |
+ cargo test -p ballista-chaos --features k8s --test k8s --
--test-threads=1 --nocapture
+
+ - name: Dump cluster state on failure
+ if: failure()
+ run: |
+ kubectl get pods -A -o wide || true
+ for ns in $(kubectl get ns -o name | grep chaos- || true); do
+ kubectl logs -n "${ns#namespace/}" -l app=ballista-scheduler
--tail=-1 || true
+ kubectl logs -n "${ns#namespace/}" -l app=ballista-executor
--all-containers --tail=-1 || true
+ done
diff --git a/chaos-testing/Cargo.toml b/chaos-testing/Cargo.toml
index 7b03cb0e2..e1e7e3f06 100644
--- a/chaos-testing/Cargo.toml
+++ b/chaos-testing/Cargo.toml
@@ -24,6 +24,12 @@ edition = { workspace = true }
rust-version = { workspace = true }
publish = false
+[features]
+# Enables the Kubernetes (kind) backend and its integration test. Off by
default
+# so the process-based harness and default `cargo test` stay dependency-light
and
+# require no cluster.
+k8s = []
+
[dependencies]
arrow = { workspace = true }
ballista = { path = "../ballista/client" }
diff --git a/chaos-testing/README.md b/chaos-testing/README.md
index b4afbfc63..020d54b29 100644
--- a/chaos-testing/README.md
+++ b/chaos-testing/README.md
@@ -173,6 +173,60 @@ written under that cluster's own temp directory, in a
`logs/` subdirectory
(`TestCluster::log_dir()`). When a scenario fails, those logs are the first
place to look for what the scheduler and executors were actually doing.
+## Running on Kubernetes (kind)
+
+The harness also has an opt-in Kubernetes backend (`K8sCluster`, in
+`src/k8s.rs`) that runs the scheduler and executors as pods in a local
+[kind](https://kind.sigs.k8s.io) cluster rather than as local processes. It is
+gated behind the `k8s` feature _and_ `CHAOS_BACKEND=kind`, so a plain `cargo
+test` never touches a cluster. Today it runs a single baseline scenario — a
real
+query whose result must match plain local DataFusion — as a walking skeleton
for
+the executor-kill scenarios the
+[#2029](https://github.com/apache/datafusion-ballista/issues/2029) follow-ups
+will add. It needs [Docker](https://docs.docker.com/get-docker/),
+[kind](https://kind.sigs.k8s.io/docs/user/quick-start/#installation), and
+`kubectl`.
+
+`dev/chaos-kind.sh` wraps the whole loop — build the image, create the cluster,
+load the image, run the tests:
+
+```sh
+dev/chaos-kind.sh # build image, create cluster, load,
test
+dev/chaos-kind.sh test -- --nocapture # re-run against an existing cluster
+dev/chaos-kind.sh down # delete the cluster
+```
+
+or run the steps by hand: `dev/build-chaos-docker.sh`, then `kind create
cluster
+--config chaos-testing/k8s/kind-config.yaml`, `kind load docker-image
+ballista-chaos:test`, and `CHAOS_BACKEND=kind cargo test -p ballista-chaos
+--features k8s --test k8s -- --test-threads=1`. Behavior is tuned through
+`CHAOS_FIXTURE_DIR` (host dir shared into the pods, default
+`$HOME/.ballista-chaos-fixtures`), `CHAOS_KEEP_NS` (keep the namespace for
+`kubectl` inspection), `CLUSTER_NAME`, `KEEP_CLUSTER`, and `KIND_NODE_IMAGE`.
+
+Because the harness runs outside the cluster, a few pieces bridge the gap:
+
+- **Fixture sharing.** The harness writes the deterministic parquet fixture to
+ `CHAOS_FIXTURE_DIR` on the host; kind's `extraMounts` bind that path into the
+ node and both pods `hostPath`-mount it, so the path is identical on host,
node,
+ and pod and the schema-inferring `CREATE EXTERNAL TABLE ... LOCATION`
resolves
+ the same everywhere, with no object store. The directory lives under `$HOME`
+ because Docker Desktop shares the home dir into its VM but not `/tmp` (the
+ static `k8s/kind-config.yaml` uses `/tmp` for Linux CI; `dev/chaos-kind.sh`
+ generates a `$HOME` config for local use).
+- **Reaching the cluster.** The client talks to the scheduler's gRPC + REST
+ (both on one port) through a `kubectl port-forward`, and fetches query
results
+ through the scheduler's embedded Flight proxy, so it never contacts executor
+ pod IPs directly.
+- **Pods.** Both expose `/healthz` + `/readyz` with liveness/readiness probes
+ (the scheduler's readiness uses `/healthz`, not `/readyz`, so its Service
+ routes before executors register), and use `imagePullPolicy: Never` since the
+ image is only ever `kind load`ed. The backend refuses any `kubectl` context
+ without a `kind-` prefix, since it creates and deletes namespaces.
+
+CI runs this on the gated `k8s-chaos` workflow (path-filtered plus nightly),
not
+per-PR — the process harness above already covers HA on every PR.
+
## Scenarios
Every scenario runs under both `ballista.planner.adaptive.enabled=false` (AQE
diff --git a/chaos-testing/k8s/kind-config.yaml
b/chaos-testing/k8s/kind-config.yaml
new file mode 100644
index 000000000..71e2c7db0
--- /dev/null
+++ b/chaos-testing/k8s/kind-config.yaml
@@ -0,0 +1,39 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# kind cluster for the chaos harness's Kubernetes backend.
+#
+# The single control-plane node bind-mounts the fixture directory from the host
+# so the harness (running outside the cluster) and every pod see the identical
+# path. This static config uses /tmp and is intended for Linux CI, where the
+# kind node binds host paths directly.
+#
+# On macOS, use `dev/chaos-kind.sh`, which generates its own config with the
+# fixture dir under $HOME (`CHAOS_FIXTURE_DIR`): Docker Desktop reliably shares
+# the home directory into its VM, whereas the VM's /tmp is not the host's, so a
+# /tmp extraMount would not propagate host files into pods.
+#
+# The path must match CHAOS_FIXTURE_DIR used by the test (see k8s-chaos.yml).
+# The host directory must exist before `kind create cluster`:
+# mkdir -p /tmp/ballista-chaos-fixtures
+kind: Cluster
+apiVersion: kind.x-k8s.io/v1alpha4
+nodes:
+ - role: control-plane
+ extraMounts:
+ - hostPath: /tmp/ballista-chaos-fixtures
+ containerPath: /tmp/ballista-chaos-fixtures
diff --git a/chaos-testing/src/bin/chaos-executor.rs
b/chaos-testing/src/bin/chaos-executor.rs
index 830d295ae..820eb48bc 100644
--- a/chaos-testing/src/bin/chaos-executor.rs
+++ b/chaos-testing/src/bin/chaos-executor.rs
@@ -23,7 +23,9 @@
use ballista_executor::executor_process::{
ExecutorProcessConfig, start_executor_process,
};
+use ballista_executor::health::spawn_health_server;
use chaos_testing::registry::chaos_function_registry;
+use std::net::SocketAddr;
use std::sync::Arc;
fn env_u16(key: &str) -> u16 {
@@ -38,10 +40,16 @@ async fn main() -> ballista_core::error::Result<()> {
env_logger::init();
let config = ExecutorProcessConfig {
- bind_host: "127.0.0.1".to_string(),
+ // Loopback in the process harness; under Kubernetes the executor binds
+ // all interfaces, finds the scheduler by Service name, and advertises
its
+ // own pod IP so the scheduler and peers can reach it for Arrow Flight.
+ bind_host: std::env::var("CHAOS_BIND_HOST")
+ .unwrap_or_else(|_| "127.0.0.1".into()),
+ external_host: std::env::var("CHAOS_EXECUTOR_EXTERNAL_HOST").ok(),
port: env_u16("CHAOS_EXECUTOR_PORT"),
grpc_port: env_u16("CHAOS_EXECUTOR_GRPC_PORT"),
- scheduler_host: "127.0.0.1".to_string(),
+ scheduler_host: std::env::var("CHAOS_SCHEDULER_HOST")
+ .unwrap_or_else(|_| "127.0.0.1".into()),
scheduler_port: env_u16("CHAOS_SCHEDULER_PORT"),
scheduler_connect_timeout_seconds: 10,
vcores: std::env::var("CHAOS_CONCURRENT_TASKS")
@@ -59,5 +67,33 @@ async fn main() -> ballista_core::error::Result<()> {
..Default::default()
};
- start_executor_process(Arc::new(config)).await
+ // Kubernetes health probes. `start_executor_process` (the library entry
+ // point) does not serve them — only the standalone binary does — so wire
the
+ // HTTP probe server here on the config's shared `ExecutorHealth` handle,
+ // exactly as `ballista-executor`'s `bin/main.rs` does. `/healthz` is
process
+ // liveness; `/readyz` reflects heartbeat state. Only started when
+ // `CHAOS_EXECUTOR_HEALTH_PORT` is set (the k8s manifest sets it), so the
+ // process-based `TestCluster` harness spawns no extra server.
+ let health_server =
std::env::var("CHAOS_EXECUTOR_HEALTH_PORT").ok().map(|port| {
+ let port: u16 = port
+ .parse()
+ .unwrap_or_else(|e| panic!("CHAOS_EXECUTOR_HEALTH_PORT must be a
u16: {e}"));
+ let addr: SocketAddr = format!("{}:{}", config.bind_host, port)
+ .parse()
+ .expect("health server address must parse");
+ let (shutdown_tx, shutdown_rx) = tokio::sync::oneshot::channel();
+ let handle = spawn_health_server(addr, config.health.clone(),
shutdown_rx);
+ (shutdown_tx, handle)
+ });
+
+ let result = start_executor_process(Arc::new(config)).await;
+
+ // Ask the health server to stop so the process can exit cleanly.
+ if let Some((shutdown_tx, handle)) = health_server {
+ let _ = shutdown_tx.send(());
+ if let Err(e) = handle.await {
+ log::warn!("health server task join error: {e}");
+ }
+ }
+ result
}
diff --git a/chaos-testing/src/bin/chaos-scheduler.rs
b/chaos-testing/src/bin/chaos-scheduler.rs
index 460582060..6a65f111d 100644
--- a/chaos-testing/src/bin/chaos-scheduler.rs
+++ b/chaos-testing/src/bin/chaos-scheduler.rs
@@ -35,12 +35,19 @@ fn env_parsed<T: std::str::FromStr>(key: &str, default: T)
-> T {
.unwrap_or(default)
}
+fn env_or(key: &str, default: &str) -> String {
+ std::env::var(key).unwrap_or_else(|_| default.to_string())
+}
+
#[tokio::main]
async fn main() -> ballista_core::error::Result<()> {
env_logger::init();
let config = SchedulerConfig {
- bind_host: "127.0.0.1".to_string(),
+ // In the process harness these default to loopback; under Kubernetes
the
+ // scheduler must bind all interfaces and advertise its Service name.
+ bind_host: env_or("CHAOS_BIND_HOST", "127.0.0.1"),
+ external_host: env_or("CHAOS_EXTERNAL_HOST", "localhost"),
bind_port: std::env::var("CHAOS_SCHEDULER_PORT")
.expect("CHAOS_SCHEDULER_PORT must be set")
.parse()
@@ -60,6 +67,12 @@ async fn main() -> ballista_core::error::Result<()> {
"CHAOS_NO_EXECUTORS_GRACE_SECONDS",
1,
),
+ // Under Kubernetes the client is outside the cluster and cannot reach
+ // executor pod IPs to fetch results. Setting this (to an empty string
in
+ // the k8s manifest) starts an embedded flight proxy on the scheduler
and
+ // makes clients fetch results through the scheduler instead of
directly
+ // from executors. Unset in the process harness -> None -> unchanged.
+ advertise_flight_sql_endpoint:
std::env::var("CHAOS_ADVERTISE_FLIGHT_PROXY").ok(),
override_session_builder: Some(Arc::new(chaos_session_state)),
..Default::default()
};
diff --git a/chaos-testing/src/k8s.rs b/chaos-testing/src/k8s.rs
new file mode 100644
index 000000000..06cd7ba18
--- /dev/null
+++ b/chaos-testing/src/k8s.rs
@@ -0,0 +1,629 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! A Kubernetes (kind) backend for the chaos harness.
+//!
+//! Where [`crate::cluster::TestCluster`] spawns the scheduler and executors as
+//! local OS processes, [`K8sCluster`] runs them as pods in a `kind` cluster:
the
+//! scheduler as a Deployment behind a `ClusterIP` Service, and the executors
as
+//! a labelled Deployment (so a later scenario can `scale`/`delete pod` them).
+//!
+//! The fixture is shared through a `hostPath` volume mounted into both pods.
+//! The harness writes the parquet under [`fixture_dir`] on the host; kind's
+//! `extraMounts` bind that path into the node, and each pod `hostPath`-mounts
+//! it, so the path string is identical on host, node, and pod and the
+//! schema-inferring `CREATE EXTERNAL TABLE ... LOCATION` that
+//! `Fixture::register_sql` emits resolves the same everywhere — no object
store.
+//! The directory lives under `$HOME` rather than `/tmp`: Docker Desktop
reliably
+//! shares the home directory into its VM (and thus the kind node), whereas the
+//! VM's `/tmp` is not the host's. Because kind is single-node, a rescheduled
pod
+//! re-mounts the same directory, so the fixture survives executor kills.
+//!
+//! The harness process runs outside the cluster, so it reaches the scheduler's
+//! gRPC + REST (both on one port) through a `kubectl port-forward`. Results
are
+//! fetched through the scheduler's embedded flight proxy
+//! (`advertise_flight_sql_endpoint`), so the client never contacts executor
pod
+//! IPs directly.
+//!
+//! This backend shells out to `kubectl`; it assumes a `kind` cluster already
+//! exists, `kubectl` is on `PATH` pointed at it, and the chaos image has been
+//! `kind load`ed. See `chaos-testing/k8s/` and the crate README for the
runbook.
+
+use std::net::TcpListener;
+use std::path::{Path, PathBuf};
+use std::process::{Child, Command, Stdio};
+use std::sync::atomic::{AtomicU32, Ordering};
+use std::time::{Duration, Instant};
+
+/// Directory shared between the harness and the pods, holding the fixture.
+///
+/// Defaults to `$HOME/.ballista-chaos-fixtures` and can be overridden with
+/// `CHAOS_FIXTURE_DIR` (the run script sets it so the kind `extraMounts` and
+/// this backend agree). A path under `$HOME` is used rather than `/tmp`
because
+/// Docker Desktop reliably shares the home directory into its VM (and thus
into
+/// the kind node), whereas the VM has its own `/tmp` that is not the host's.
+/// The path is identical on host, node, and pod, so the schema-inferring
+/// `CREATE EXTERNAL TABLE ... LOCATION` that `Fixture::register_sql` emits
+/// resolves the same everywhere.
+fn fixture_dir() -> String {
+ std::env::var("CHAOS_FIXTURE_DIR").unwrap_or_else(|_| {
+ let home = std::env::var("HOME").unwrap_or_else(|_|
"/tmp".to_string());
+ format!("{home}/.ballista-chaos-fixtures")
+ })
+}
+
+const CHAOS_IMAGE: &str = "ballista-chaos:test";
+const SCHEDULER_PORT: u16 = 50050;
+/// Port the executor pods serve `/healthz` + `/readyz` on for the k8s probes.
+const EXECUTOR_HEALTH_PORT: u16 = 50053;
+const EXECUTOR_DEPLOYMENT: &str = "ballista-executor";
+
+/// Per-process counter so each `K8sCluster::start` gets a distinct namespace,
+/// even if several run in one process (`--test-threads=1` serialises them
today,
+/// but this keeps namespaces unique if that ever changes or a start is
retried).
+static NS_SEQ: AtomicU32 = AtomicU32::new(0);
+
+/// How an executor pod is removed.
+#[derive(Clone, Copy, Debug)]
+pub enum KillMode {
+ /// `kubectl delete pod` — SIGTERM plus the termination grace period, so
the
+ /// executor's graceful-shutdown path runs (the path a raw process
`SIGKILL`
+ /// can never reach).
+ Graceful,
+ /// `kubectl delete pod --grace-period=0 --force` — an abrupt loss, the
+ /// closest k8s analogue of the process harness's `SIGKILL`.
+ Forced,
+}
+
+/// A Ballista cluster running as pods in a kind cluster.
+pub struct K8sCluster {
+ namespace: String,
+ scheduler_local_port: u16,
+ port_forward: Child,
+ shared_dir: PathBuf,
+}
+
+impl K8sCluster {
+ /// Deploy a scheduler + `executors` executor pods, wait until all
executors
+ /// have registered, and open a port-forward to the scheduler.
+ pub async fn start(executors: usize) -> Result<Self, String> {
+ require_kubectl()?;
+
+ // One cluster per process; --test-threads=1 keeps it to one at a time.
+ // The counter guards against collisions if that ever changes.
+ let namespace = format!(
+ "chaos-{}-{}",
+ std::process::id(),
+ NS_SEQ.fetch_add(1, Ordering::Relaxed)
+ );
+ let shared_dir = PathBuf::from(fixture_dir());
+
+ // Ensure the shared dir exists, then clear its *contents* so a
previous
+ // run's fixture (e.g. one written by an older build with a different
+ // schema) cannot leak into this run. This matters for local,
+ // non-ephemeral use; on CI the runner is fresh. We clear the contents
+ // rather than the directory itself: it is the bind-mount root, and
+ // removing it can sever the mount so pod writes no longer reach the
node.
+ std::fs::create_dir_all(&shared_dir)
+ .map_err(|e| format!("create shared dir {}: {e}",
shared_dir.display()))?;
+ clear_dir_contents(&shared_dir)?;
+
+ let manifests = render_manifests(&namespace, executors, &shared_dir);
+ kubectl_apply(&manifests).await?;
+
+ // Guard so the namespace is torn down even if a later step fails.
+ let guard = NamespaceGuard {
+ namespace: namespace.clone(),
+ };
+
+ kubectl(&[
+ "-n",
+ &namespace,
+ "rollout",
+ "status",
+ "deploy/ballista-scheduler",
+ "--timeout=120s",
+ ])
+ .await?;
+
+ let scheduler_local_port = free_port()?;
+ let port_forward = spawn_port_forward(&namespace,
scheduler_local_port)?;
+
+ let cluster = Self {
+ namespace,
+ scheduler_local_port,
+ port_forward,
+ shared_dir,
+ };
+
+ cluster.await_executors(executors).await?;
+
+ // Everything is up; keep the namespace (transfer ownership to
`cluster`).
+ std::mem::forget(guard);
+ Ok(cluster)
+ }
+
+ /// `df://…` endpoint for the `ballista` client, via the port-forward.
+ pub fn scheduler_url(&self) -> String {
+ format!("df://127.0.0.1:{}", self.scheduler_local_port)
+ }
+
+ /// `http://…` endpoint for the scheduler REST API, via the port-forward.
+ pub fn rest_url(&self) -> String {
+ format!("http://127.0.0.1:{}";, self.scheduler_local_port)
+ }
+
+ /// The host directory shared into every pod; write the fixture here.
+ pub fn shared_dir(&self) -> &Path {
+ &self.shared_dir
+ }
+
+ /// Block until `n` executors have registered with the scheduler.
+ pub async fn await_executors(&self, n: usize) -> Result<(), String> {
+ let deadline = Instant::now() + Duration::from_secs(120);
+ loop {
+ if let Ok(count) = self.registered_executors().await
+ && count == n
+ {
+ return Ok(());
+ }
+ if Instant::now() > deadline {
+ self.dump_diagnostics().await;
+ return Err(format!(
+ "timed out waiting for {n} executors to register with the
scheduler"
+ ));
+ }
+ tokio::time::sleep(Duration::from_millis(500)).await;
+ }
+ }
+
+ /// Print pod status and scheduler/executor logs to stderr — invoked when a
+ /// wait times out, so a failed run is diagnosable even though the
namespace
+ /// is torn down afterwards. Set `CHAOS_KEEP_NS=1` to keep the namespace
for
+ /// manual `kubectl` inspection.
+ pub async fn dump_diagnostics(&self) {
+ eprintln!("==> chaos k8s diagnostics for namespace {}",
self.namespace);
+ for args in [
+ vec!["-n", &self.namespace, "get", "pods", "-o", "wide"],
+ vec![
+ "-n",
+ &self.namespace,
+ "logs",
+ "-l",
+ "app=ballista-scheduler",
+ "--tail=40",
+ ],
+ vec![
+ "-n",
+ &self.namespace,
+ "logs",
+ "-l",
+ "app=ballista-executor",
+ "--tail=40",
+ "--prefix",
+ ],
+ ] {
+ match kubectl(&args).await {
+ Ok(out) => eprintln!("$ kubectl {}\n{out}", args.join(" ")),
+ Err(e) => eprintln!("$ kubectl {} -> {e}", args.join(" ")),
+ }
+ }
+ }
+
+ /// How many executors the scheduler currently considers registered.
+ pub async fn registered_executors(&self) -> Result<usize, String> {
+ // A short timeout so a stalled port-forward surfaces as a retryable
+ // error in the polling loop rather than hanging the whole wait.
+ let client = reqwest::Client::builder()
+ .timeout(Duration::from_secs(5))
+ .build()
+ .map_err(|e| e.to_string())?;
+ let body: serde_json::Value = client
+ .get(format!("{}/api/executors", self.rest_url()))
+ .send()
+ .await
+ .map_err(|e| e.to_string())?
+ .json()
+ .await
+ .map_err(|e| e.to_string())?;
+ Ok(body.as_array().map(|a| a.len()).unwrap_or(0))
+ }
+
+ /// Scale the executor Deployment. `0` is a total loss that stays lost (the
+ /// controller does not recreate the pods); scaling back up recovers.
+ ///
+ /// Not yet exercised by a scenario — this is the k8s primitive the
executor
+ /// kill/loss scenarios (the #2029 follow-ups) will drive; the baseline
test
+ /// only needs a healthy cluster. Kept here so the backend is complete.
+ pub async fn scale_executors(&self, replicas: usize) -> Result<(), String>
{
+ kubectl(&[
+ "-n",
+ &self.namespace,
+ "scale",
+ &format!("deploy/{EXECUTOR_DEPLOYMENT}"),
+ &format!("--replicas={replicas}"),
+ ])
+ .await
+ .map(|_| ())
+ }
+
+ /// Delete one executor pod. The Deployment reschedules a replacement (a
fresh
+ /// executor with a new id), exercising k8s rescheduling plus Ballista's
+ /// executor-loss recovery.
+ pub async fn kill_one_executor(&self, mode: KillMode) -> Result<(),
String> {
+ let pods = self.pods_by_label("app=ballista-executor").await?;
+ let name = pods
+ .into_iter()
+ .next()
+ .ok_or_else(|| "no executor pods found".to_string())?;
+ let mut args = vec!["-n", &self.namespace, "delete", "pod", &name];
+ if matches!(mode, KillMode::Forced) {
+ args.extend_from_slice(&["--grace-period=0", "--force"]);
+ }
+ kubectl(&args).await.map(|_| ())
+ }
+
+ async fn pods_by_label(&self, label: &str) -> Result<Vec<String>, String> {
+ let out = kubectl(&[
+ "-n",
+ &self.namespace,
+ "get",
+ "pods",
+ "-l",
+ label,
+ "-o",
+ "jsonpath={.items[*].metadata.name}",
+ ])
+ .await?;
+ Ok(out.split_whitespace().map(|s| s.to_string()).collect())
+ }
+}
+
+impl Drop for K8sCluster {
+ fn drop(&mut self) {
+ let _ = self.port_forward.kill();
+ let _ = self.port_forward.wait();
+ delete_namespace(&self.namespace);
+ }
+}
+
+/// Deletes a namespace on drop; `mem::forget`ten once startup fully succeeds.
+struct NamespaceGuard {
+ namespace: String,
+}
+
+impl Drop for NamespaceGuard {
+ fn drop(&mut self) {
+ delete_namespace(&self.namespace);
+ }
+}
+
+/// Best-effort namespace teardown, skipped when `CHAOS_KEEP_NS` is set so a
+/// failed run can be inspected with `kubectl`.
+fn delete_namespace(namespace: &str) {
+ if std::env::var_os("CHAOS_KEEP_NS").is_some() {
+ eprintln!(
+ "CHAOS_KEEP_NS set: leaving namespace {namespace} in place; \
+ delete it with `kubectl delete namespace {namespace}`"
+ );
+ return;
+ }
+ let _ = Command::new("kubectl")
+ .args([
+ "delete",
+ "namespace",
+ namespace,
+ "--wait=false",
+ "--ignore-not-found",
+ ])
+ .stdout(Stdio::null())
+ .stderr(Stdio::null())
+ .status();
+}
+
+/// Remove everything *inside* `dir` without removing `dir` itself. `dir` is a
+/// bind-mount root, so deleting it can sever the mount; deleting only its
+/// entries is safe and leaves the mount intact.
+fn clear_dir_contents(dir: &Path) -> Result<(), String> {
+ for entry in std::fs::read_dir(dir)
+ .map_err(|e| format!("read shared dir {}: {e}", dir.display()))?
+ {
+ let entry = entry.map_err(|e| format!("read dir entry: {e}"))?;
+ let path = entry.path();
+ let is_dir = entry.file_type().map_err(|e| e.to_string())?.is_dir();
+ let result = if is_dir {
+ std::fs::remove_dir_all(&path)
+ } else {
+ std::fs::remove_file(&path)
+ };
+ result.map_err(|e| format!("remove {}: {e}", path.display()))?;
+ }
+ Ok(())
+}
+
+fn require_kubectl() -> Result<(), String> {
+ Command::new("kubectl")
+ .arg("version")
+ .arg("--client")
+ .stdout(Stdio::null())
+ .stderr(Stdio::null())
+ .status()
+ .map_err(|e| format!("kubectl not found on PATH: {e}"))
+ .and_then(|s| {
+ s.success()
+ .then_some(())
+ .ok_or_else(|| "`kubectl version --client` failed".to_string())
+ })?;
+
+ // This backend creates and deletes namespaces, so refuse to run unless the
+ // current context is a kind cluster — a guard against pointing it at a
real
+ // cluster by accident. kind names its context `kind-<cluster>`.
+ let output = Command::new("kubectl")
+ .args(["config", "current-context"])
+ .output()
+ .map_err(|e| format!("read kubectl current-context: {e}"))?;
+ let context = String::from_utf8_lossy(&output.stdout);
+ let context = context.trim();
+ if !context.starts_with("kind-") {
+ return Err(format!(
+ "refusing to run: current kubectl context is {context:?}, not a
kind \
+ cluster (expected a `kind-` prefix). Point kubectl at a kind
cluster, \
+ e.g. `kubectl config use-context kind-ballista-chaos`."
+ ));
+ }
+ Ok(())
+}
+
+/// Reserve a free local TCP port for the port-forward.
+fn free_port() -> Result<u16, String> {
+ let listener = TcpListener::bind("127.0.0.1:0")
+ .map_err(|e| format!("bind ephemeral port: {e}"))?;
+ listener
+ .local_addr()
+ .map(|a| a.port())
+ .map_err(|e| e.to_string())
+}
+
+fn spawn_port_forward(namespace: &str, local_port: u16) -> Result<Child,
String> {
+ Command::new("kubectl")
+ .args([
+ "-n",
+ namespace,
+ "port-forward",
+ "svc/ballista-scheduler",
+ &format!("{local_port}:{SCHEDULER_PORT}"),
+ ])
+ .stdout(Stdio::null())
+ .stderr(Stdio::null())
+ .spawn()
+ .map_err(|e| format!("spawn kubectl port-forward: {e}"))
+}
+
+/// Run `kubectl` with the given args, returning stdout on success.
+async fn kubectl(args: &[&str]) -> Result<String, String> {
+ let output = tokio::process::Command::new("kubectl")
+ .args(args)
+ .output()
+ .await
+ .map_err(|e| format!("run kubectl {args:?}: {e}"))?;
+ if output.status.success() {
+ Ok(String::from_utf8_lossy(&output.stdout).into_owned())
+ } else {
+ Err(format!(
+ "kubectl {args:?} failed: {}",
+ String::from_utf8_lossy(&output.stderr)
+ ))
+ }
+}
+
+/// `kubectl apply` a rendered manifest by piping it to stdin.
+async fn kubectl_apply(manifests: &str) -> Result<(), String> {
+ use tokio::io::AsyncWriteExt;
+
+ let mut child = tokio::process::Command::new("kubectl")
+ .args(["apply", "-f", "-"])
+ .stdin(Stdio::piped())
+ .stdout(Stdio::null())
+ .stderr(Stdio::piped())
+ .spawn()
+ .map_err(|e| format!("spawn kubectl apply: {e}"))?;
+
+ child
+ .stdin
+ .take()
+ .expect("stdin piped")
+ .write_all(manifests.as_bytes())
+ .await
+ .map_err(|e| format!("write manifests to kubectl: {e}"))?;
+
+ let output = child
+ .wait_with_output()
+ .await
+ .map_err(|e| format!("wait for kubectl apply: {e}"))?;
+ if output.status.success() {
+ Ok(())
+ } else {
+ Err(format!(
+ "kubectl apply failed: {}",
+ String::from_utf8_lossy(&output.stderr)
+ ))
+ }
+}
+
+/// Render the namespace + scheduler (Deployment + Service) + executor
Deployment.
+/// `mount` is the fixture directory, bind-mounted into both pods (see
+/// [`fixture_dir`]); it must match the kind `extraMounts` path.
+fn render_manifests(
+ namespace: &str,
+ executors: usize,
+ mount: &std::path::Path,
+) -> String {
+ let mount = mount.display();
+ format!(
+ r#"
+apiVersion: v1
+kind: Namespace
+metadata:
+ name: {namespace}
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: ballista-scheduler
+ namespace: {namespace}
+spec:
+ replicas: 1
+ selector:
+ matchLabels:
+ app: ballista-scheduler
+ template:
+ metadata:
+ labels:
+ app: ballista-scheduler
+ spec:
+ containers:
+ - name: scheduler
+ image: {CHAOS_IMAGE}
+ imagePullPolicy: Never
+ command: ["/root/chaos-scheduler"]
+ env:
+ - name: CHAOS_SCHEDULER_PORT
+ value: "{SCHEDULER_PORT}"
+ - name: CHAOS_BIND_HOST
+ value: "0.0.0.0"
+ - name: CHAOS_EXTERNAL_HOST
+ value: "ballista-scheduler"
+ - name: CHAOS_ADVERTISE_FLIGHT_PROXY
+ value: ""
+ - name: CHAOS_EXECUTOR_TIMEOUT_SECONDS
+ value: "5"
+ - name: CHAOS_EXPIRE_INTERVAL_SECONDS
+ value: "1"
+ - name: RUST_LOG
+ value: "info"
+ ports:
+ - containerPort: {SCHEDULER_PORT}
+ # Both probes use /healthz, not /readyz: executors reach the
scheduler
+ # through the Service below, and a Service only routes to Ready pods.
+ # The scheduler's /readyz gates on registered executors, so a /readyz
+ # readiness probe would deadlock (no endpoints -> executors can't
+ # register -> never ready). /healthz reports process liveness, which
is
+ # all the Service needs to start routing.
+ readinessProbe:
+ httpGet:
+ path: /healthz
+ port: {SCHEDULER_PORT}
+ periodSeconds: 2
+ failureThreshold: 3
+ livenessProbe:
+ httpGet:
+ path: /healthz
+ port: {SCHEDULER_PORT}
+ periodSeconds: 10
+ failureThreshold: 3
+ volumeMounts:
+ - name: fixtures
+ mountPath: {mount}
+ volumes:
+ - name: fixtures
+ hostPath:
+ path: {mount}
+ type: DirectoryOrCreate
+---
+apiVersion: v1
+kind: Service
+metadata:
+ name: ballista-scheduler
+ namespace: {namespace}
+spec:
+ selector:
+ app: ballista-scheduler
+ ports:
+ - port: {SCHEDULER_PORT}
+ targetPort: {SCHEDULER_PORT}
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: {EXECUTOR_DEPLOYMENT}
+ namespace: {namespace}
+spec:
+ replicas: {executors}
+ selector:
+ matchLabels:
+ app: ballista-executor
+ template:
+ metadata:
+ labels:
+ app: ballista-executor
+ spec:
+ terminationGracePeriodSeconds: 30
+ containers:
+ - name: executor
+ image: {CHAOS_IMAGE}
+ imagePullPolicy: Never
+ command: ["/root/chaos-executor"]
+ env:
+ - name: CHAOS_SCHEDULER_HOST
+ value: "ballista-scheduler"
+ - name: CHAOS_SCHEDULER_PORT
+ value: "{SCHEDULER_PORT}"
+ - name: CHAOS_EXECUTOR_PORT
+ value: "50051"
+ - name: CHAOS_EXECUTOR_GRPC_PORT
+ value: "50052"
+ - name: CHAOS_EXECUTOR_HEALTH_PORT
+ value: "{EXECUTOR_HEALTH_PORT}"
+ - name: CHAOS_BIND_HOST
+ value: "0.0.0.0"
+ - name: CHAOS_EXECUTOR_EXTERNAL_HOST
+ valueFrom:
+ fieldRef:
+ fieldPath: status.podIP
+ - name: CHAOS_HEARTBEAT_SECONDS
+ value: "1"
+ - name: RUST_LOG
+ value: "info"
+ ports:
+ - containerPort: {EXECUTOR_HEALTH_PORT}
+ # The executor is reached by pod IP (not a readiness-gated Service),
so
+ # /readyz here is safe and meaningful: it reports SERVICE_UNAVAILABLE
+ # until the first heartbeat lands, then 200. Liveness stays on
/healthz
+ # (process-alive) so a slow/again-disconnected executor is not
killed.
+ readinessProbe:
+ httpGet:
+ path: /readyz
+ port: {EXECUTOR_HEALTH_PORT}
+ periodSeconds: 2
+ failureThreshold: 3
+ livenessProbe:
+ httpGet:
+ path: /healthz
+ port: {EXECUTOR_HEALTH_PORT}
+ periodSeconds: 10
+ failureThreshold: 3
+ volumeMounts:
+ - name: fixtures
+ mountPath: {mount}
+ volumes:
+ - name: fixtures
+ hostPath:
+ path: {mount}
+ type: DirectoryOrCreate
+"#
+ )
+}
diff --git a/chaos-testing/src/lib.rs b/chaos-testing/src/lib.rs
index 5c1c52764..29cb897c5 100644
--- a/chaos-testing/src/lib.rs
+++ b/chaos-testing/src/lib.rs
@@ -23,5 +23,7 @@
pub mod budget;
pub mod cluster;
pub mod fixture;
+#[cfg(feature = "k8s")]
+pub mod k8s;
pub mod registry;
pub mod udf;
diff --git a/chaos-testing/tests/k8s.rs b/chaos-testing/tests/k8s.rs
new file mode 100644
index 000000000..af5d9420a
--- /dev/null
+++ b/chaos-testing/tests/k8s.rs
@@ -0,0 +1,113 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Kubernetes (kind) backend scenarios.
+//!
+//! Gated behind the `k8s` feature and `CHAOS_BACKEND=kind`, so the default
+//! `cargo test` never touches a cluster. Run with the kind runbook in the
crate
+//! README:
+//!
+//! ```sh
+//! dev/build-chaos-docker.sh
+//! kind create cluster --config chaos-testing/k8s/kind-config.yaml
+//! kind load docker-image ballista-chaos:test
+//! CHAOS_BACKEND=kind cargo test -p ballista-chaos --features k8s --test k8s
-- --test-threads=1
+//! ```
+#![cfg(feature = "k8s")]
+
+use ballista::prelude::{SessionConfigExt, SessionContextExt};
+use chaos_testing::fixture::Fixture;
+use chaos_testing::k8s::K8sCluster;
+use datafusion::arrow::util::pretty::pretty_format_batches;
+use datafusion::execution::session_state::SessionStateBuilder;
+use datafusion::prelude::{SessionConfig, SessionContext};
+
+/// The k8s scenarios need a running kind cluster with the chaos images loaded;
+/// they are opt-in via `CHAOS_BACKEND=kind` so a plain `cargo test` skips
them.
+fn kind_backend_selected() -> bool {
+ if std::env::var("CHAOS_BACKEND").as_deref() == Ok("kind") {
+ true
+ } else {
+ eprintln!(
+ "skipping k8s scenario: set CHAOS_BACKEND=kind and provide a kind
cluster \
+ with the chaos images loaded (see the crate README runbook)"
+ );
+ false
+ }
+}
+
+/// The chaos-free baseline query, run on a fresh local DataFusion context.
This
+/// is the reference the cluster must reproduce exactly.
+async fn local_baseline(fixture: &Fixture) -> String {
+ let ctx = SessionContext::new();
+ for stmt in fixture.register_sql() {
+ ctx.sql(&stmt).await.unwrap().collect().await.unwrap();
+ }
+ let batches = ctx
+ .sql(Fixture::baseline_query())
+ .await
+ .unwrap()
+ .collect()
+ .await
+ .unwrap();
+ pretty_format_batches(&batches).unwrap().to_string()
+}
+
+/// Smoke test: a real query on a real kind cluster returns the same result as
+/// plain local DataFusion. Exercises the whole path — client → scheduler →
pods
+/// → shuffle → result — with the fixture shared through the `hostPath` mount.
+#[tokio::test]
+async fn baseline_matches_local_datafusion_on_k8s() {
+ if !kind_backend_selected() {
+ return;
+ }
+
+ let cluster = K8sCluster::start(2).await.expect("kind cluster must start");
+
+ // Written into the shared mount, so the scheduler and executor pods see
it.
+ let fixture = Fixture::write(cluster.shared_dir())
+ .await
+ .expect("fixture must be written to the shared mount");
+
+ let expected = local_baseline(&fixture).await;
+
+ let config = SessionConfig::new_with_ballista();
+ let state = SessionStateBuilder::new()
+ .with_config(config)
+ .with_default_features()
+ .build();
+ let ctx = SessionContext::remote_with_state(&cluster.scheduler_url(),
state)
+ .await
+ .expect("client must connect to the scheduler");
+
+ for stmt in fixture.register_sql() {
+ ctx.sql(&stmt).await.unwrap().collect().await.unwrap();
+ }
+ let batches = ctx
+ .sql(Fixture::baseline_query())
+ .await
+ .unwrap()
+ .collect()
+ .await
+ .unwrap();
+ let actual = pretty_format_batches(&batches).unwrap().to_string();
+
+ assert_eq!(
+ actual, expected,
+ "cluster result must match plain local DataFusion"
+ );
+}
diff --git a/dev/build-chaos-docker.sh b/dev/build-chaos-docker.sh
new file mode 100755
index 000000000..9b19b35e1
--- /dev/null
+++ b/dev/build-chaos-docker.sh
@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Build the `ballista-chaos:test` image (both chaos binaries) for the kind
+# backend. The binaries are compiled inside the image so it always matches the
+# container architecture — no host cross-toolchain needed. Load it with:
+#
+# kind load docker-image ballista-chaos:test
+#
+# Requires BuildKit (default in modern Docker).
+
+set -euo pipefail
+
+DOCKER_BUILDKIT=1 docker build \
+ -t "ballista-chaos:test" -f dev/docker/chaos.Dockerfile .
diff --git a/dev/chaos-kind.sh b/dev/chaos-kind.sh
new file mode 100755
index 000000000..5aa0a7357
--- /dev/null
+++ b/dev/chaos-kind.sh
@@ -0,0 +1,171 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Run the chaos harness's Kubernetes (kind) scenarios locally.
+#
+# dev/chaos-kind.sh [command] [-- <extra args passed to cargo test>]
+#
+# Commands:
+# up Create the kind cluster (if missing), build + load the chaos images.
+# test Run the kind chaos scenarios against the current cluster.
+# all up + test (default)
+# down Delete the kind cluster.
+#
+# Env:
+# CLUSTER_NAME kind cluster name (default: ballista-chaos)
+# KEEP_CLUSTER with the `all` command: "0" tears the cluster down at the
end,
+# any other value (default "1") leaves it running
+#
+# Examples:
+# dev/chaos-kind.sh # build, (re)create, load, run
everything
+# dev/chaos-kind.sh up # just stand the cluster up
+# dev/chaos-kind.sh test -- --nocapture # re-run tests on an existing
cluster
+# dev/chaos-kind.sh down # tear it down
+
+set -euo pipefail
+
+CLUSTER_NAME=${CLUSTER_NAME:-ballista-chaos}
+KEEP_CLUSTER=${KEEP_CLUSTER:-1}
+# Fixture dir shared host<->pods. Under $HOME (not /tmp) because Docker Desktop
+# reliably shares the home directory into its VM/kind node, whereas the VM's
+# /tmp is not the host's. Exported so the test's k8s backend uses the same
path.
+export CHAOS_FIXTURE_DIR=${CHAOS_FIXTURE_DIR:-$HOME/.ballista-chaos-fixtures}
+CHAOS_IMAGE=ballista-chaos:test
+
+# Run from the repository root so the relative paths below resolve.
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+cd "$REPO_ROOT"
+
+# kind config is generated so its extraMounts match CHAOS_FIXTURE_DIR exactly.
+KIND_CONFIG="$(mktemp -t kind-config.XXXXXX.yaml)"
+gen_kind_config() {
+ cat > "$KIND_CONFIG" <<EOF
+kind: Cluster
+apiVersion: kind.x-k8s.io/v1alpha4
+nodes:
+ - role: control-plane
+ extraMounts:
+ - hostPath: ${CHAOS_FIXTURE_DIR}
+ containerPath: ${CHAOS_FIXTURE_DIR}
+EOF
+}
+
+need_cmd() {
+ command -v "$1" >/dev/null 2>&1 || {
+ echo "error: '$1' is required but not found on PATH" >&2
+ exit 1
+ }
+}
+
+cluster_exists() {
+ kind get clusters 2>/dev/null | grep -qx "$CLUSTER_NAME"
+}
+
+up() {
+ need_cmd docker
+ need_cmd kind
+ need_cmd kubectl
+
+ mkdir -p "$CHAOS_FIXTURE_DIR"
+
+ if cluster_exists; then
+ echo "==> kind cluster '$CLUSTER_NAME' already exists"
+ else
+ echo "==> creating kind cluster '$CLUSTER_NAME' (fixture mount:
$CHAOS_FIXTURE_DIR)"
+ gen_kind_config
+ # KIND_NODE_IMAGE lets you pin a node image compatible with your kind
+ # binary (e.g. kindest/node:v1.31.4). A mismatch between the kind version
+ # and the node image is the usual cause of the "could not find a log line
+ # that matches ... Multi-User System" boot failure.
+ local create_args=(--name "$CLUSTER_NAME" --config "$KIND_CONFIG")
+ if [ -n "${KIND_NODE_IMAGE:-}" ]; then
+ create_args+=(--image "$KIND_NODE_IMAGE")
+ fi
+ if ! kind create cluster "${create_args[@]}"; then
+ echo "" >&2
+ echo "error: kind failed to create the cluster. This is an environment
issue" >&2
+ echo " (Docker/kind), not the Ballista harness. Diagnostics:" >&2
+ echo "--- versions ---" >&2
+ kind version >&2 || true
+ docker version --format '{{.Server.Version}}' >&2 || true
+ echo "--- node container logs (if retained) ---" >&2
+ docker ps -a --filter "name=${CLUSTER_NAME}-control-plane" >&2 || true
+ docker logs "${CLUSTER_NAME}-control-plane" 2>&1 | tail -30 >&2 || true
+ echo "" >&2
+ echo "Try: upgrade kind (brew upgrade kind), (re)start/enlarge Docker,
or pin a" >&2
+ echo " node image: KIND_NODE_IMAGE=kindest/node:v1.31.4 $0 up" >&2
+ exit 1
+ fi
+ fi
+
+ echo "==> building chaos image (compiled inside Docker)"
+ ./dev/build-chaos-docker.sh
+
+ echo "==> loading image into kind"
+ kind load docker-image "$CHAOS_IMAGE" --name "$CLUSTER_NAME"
+}
+
+run_tests() {
+ need_cmd kubectl
+ cluster_exists || {
+ echo "error: kind cluster '$CLUSTER_NAME' does not exist; run '$0 up'
first" >&2
+ exit 1
+ }
+ # Point kubectl at this cluster for the duration of the run.
+ kubectl config use-context "kind-${CLUSTER_NAME}" >/dev/null
+
+ echo "==> running kind chaos scenarios"
+ CHAOS_BACKEND=kind cargo test -p ballista-chaos --features k8s --test k8s \
+ -- --test-threads=1 "$@"
+}
+
+down() {
+ need_cmd kind
+ echo "==> deleting kind cluster '$CLUSTER_NAME'"
+ kind delete cluster --name "$CLUSTER_NAME"
+}
+
+command="all"
+if [ $# -gt 0 ] && [[ "$1" != "--" ]]; then
+ command="$1"
+ shift
+fi
+# Drop a leading "--" so callers can write: `test -- --nocapture`.
+if [ $# -gt 0 ] && [[ "$1" == "--" ]]; then
+ shift
+fi
+
+case "$command" in
+ up) up ;;
+ test) run_tests "$@" ;;
+ down) down ;;
+ all)
+ up
+ run_tests "$@"
+ if [ "$KEEP_CLUSTER" = "0" ]; then
+ down
+ else
+ echo "==> leaving cluster '$CLUSTER_NAME' up (set KEEP_CLUSTER=0 to
delete, or run '$0 down')"
+ fi
+ ;;
+ *)
+ echo "usage: $0 [up|test|down|all] [-- <extra cargo test args>]" >&2
+ exit 1
+ ;;
+esac
diff --git a/dev/docker/chaos.Dockerfile b/dev/docker/chaos.Dockerfile
new file mode 100644
index 000000000..0709f948f
--- /dev/null
+++ b/dev/docker/chaos.Dockerfile
@@ -0,0 +1,58 @@
+# syntax=docker/dockerfile:1
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# One image containing both chaos binaries, used by the Kubernetes backend in
+# `chaos-testing`. The pod manifest selects the role via `command`.
+#
+# The binaries are compiled *inside* this build, so the image is always built
+# for the container's architecture. A host `cargo build` would embed the host's
+# binary (e.g. a macOS Mach-O on Apple Silicon), which fails in a Linux pod
with
+# "exec format error". BuildKit cache mounts keep the cargo registry and target
+# dir across builds, so an incremental rebuild after a chaos-only change is
fast.
+#
+# Debug info is disabled (CARGO_PROFILE_DEV_DEBUG=0) and symbols are stripped:
a
+# full debug binary statically links all of DataFusion + aws-lc/ring and can
OOM
+# `ld` at link time, and the ~250MB result is slow (and sometimes fails) to
+# `kind load`. Without DWARF the link is light and each binary is tens of MB.
+
+FROM rust:1-bookworm AS builder
+RUN apt-get update \
+ && apt-get install -y --no-install-recommends protobuf-compiler \
+ && rm -rf /var/lib/apt/lists/*
+WORKDIR /src
+COPY . .
+ENV CARGO_PROFILE_DEV_DEBUG=0
+ENV RUSTFLAGS="-C strip=symbols"
+RUN --mount=type=cache,target=/usr/local/cargo/registry \
+ --mount=type=cache,target=/src/target \
+ cargo build -p ballista-chaos --bin chaos-scheduler --bin chaos-executor \
+ && mkdir -p /out \
+ && cp target/debug/chaos-scheduler target/debug/chaos-executor /out/
+
+FROM debian:bookworm-slim
+ENV RUST_LOG=info
+ENV RUST_BACKTRACE=full
+COPY --from=builder /out/chaos-scheduler /root/chaos-scheduler
+COPY --from=builder /out/chaos-executor /root/chaos-executor
+
+# scheduler gRPC/REST (50050); executor Arrow Flight (50051), gRPC (50052), and
+# HTTP health probes (50053).
+EXPOSE 50050 50051 50052 50053
+
+# No ENTRYPOINT: the pod manifest sets `command` to the desired binary.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
