akshaychitneni opened a new pull request, #2244:
URL: https://github.com/apache/datafusion-ballista/pull/2244
Rationale
The chaos harness (#2026) exercises Ballista's HA paths on a real
multi-process cluster, but only as OS processes on one host. Several failure
modes are k8s-specific — graceful SIGTERM/decommission, pod rescheduling,
/readyz→Service gating, autoscaling — and can't be reached that way. This adds
a Kubernetes backend so the same scenarios can run on a real kind cluster.
What's included
- K8sCluster backend (kubectl-driven): scheduler as a Deployment behind a
ClusterIP Service, executors as a labelled Deployment; reaches the scheduler
via kubectl port-forward.
- One baseline scenario (tests/k8s.rs) asserting the cluster result
matches local DataFusion — a full client → scheduler → executors → shuffle →
result smoke test.
Are there user-facing changes?
No. Gated behind the k8s cargo feature + CHAOS_BACKEND=kind; the default
cargo test and the process harness are unchanged.
Key decisions
- Executors as a Deployment, not bare pods — Ballista executors
self-register with a fresh UUID and are fungible, so the Deployment model fits
and enables scale/delete pod for kill scenarios.
- Results via the scheduler's embedded flight proxy
(advertise_flight_sql_endpoint) — the out-of-cluster client can't reach
executor pod IPs; the proxy relays through the port-forwarded scheduler.
Exposing executors directly was rejected: their advertised address is shared
with in-cluster peers, so it can't be both pod-internal and
host-reachable.
- Gated + opt-in CI — separate job (paths + nightly), off the per-PR
critical path, since the process harness already covers HA on every PR.
Testing
Baseline scenario passes end-to-end on a local kind cluster via
dev/chaos-kind.sh; process harness and default build/clippy unaffected.
Follow-ups
Kill / graceful-decommission / total-loss (#2029) scenarios
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
