andygrove opened a new pull request, #2160:
URL: https://github.com/apache/datafusion-ballista/pull/2160
# Which issue does this PR close?
<!-- No tracked issue yet; can be linked if one is filed. -->
# Rationale for this change
By default (`--memory-pool-size` unset), an executor runs with DataFusion's
unbounded memory pool, so spillable operators (sort, spillable aggregate/join,
the sort-shuffle writer) never spill. On heavy plans — for example TPC-H q9
stage 1, with several concurrent tasks per executor each scanning `lineitem`
and writing shuffle output — memory grows until the OS (or the container
cgroup) OOM-kills the process. The executor is then restarted, the scheduler
re-dispatches the same tasks, and it OOMs again: a reset → re-dispatch → reset
livelock that surfaces to the user as executors "heartbeat timing out" and the
query never completing.
Installing a bounded `FairSpillPool` avoids this by letting DataFusion spill
to disk instead of exhausting memory. Making that the default means a fresh
cluster does not OOM on heavy plans without manual tuning.
# What changes are included in this PR?
Executor-side only (`ballista/executor/src/executor_process.rs`,
`ballista/executor/src/config.rs`), plus a doc update.
- `--memory-pool-size` now has three-way semantics:
- **unset (default):** a bounded `FairSpillPool` sized at a fraction
(default 70%) of the detected host/cgroup memory limit, split evenly across
vcores.
- **`0`:** unbounded (DataFusion's default, no spilling) — the explicit
opt-out that preserves the previous behavior.
- **`N`:** exactly `N` bytes (unchanged).
- New `--memory-pool-fraction` flag (range `(0.0, 1.0]`, default `0.70`)
tunes the auto path; ignored when `--memory-pool-size` is set explicitly.
- Cgroup-aware detection: reads cgroup v2 `memory.max` / v1
`memory.limit_in_bytes` and uses `min(host, cgroup)`, so container memory
limits are respected. Falls back to host memory (via `sysinfo`) when no cgroup
limit applies, and to unbounded-with-warning if no limit can be detected.
- A startup log line reporting the resolved pool size and its source, e.g.
`Executor memory pool: 22.4 GiB (auto: 70% of host memory), split across 8
vcores (2.8 GiB/vcore)`.
- Updated `docs/source/user-guide/tuning-guide.md` to describe the new
default, the escape hatch, and the new flag.
The pure resolution logic (classification, `min`/fraction, cgroup/host
detection) is unit-tested across the full matrix, including cgroup-v2 `max`,
the cgroup-v1 "unlimited" sentinel, host-only, cgroup-only, and undetected
cases.
# Are there any user-facing changes?
Yes — this changes default executor behavior:
- **Default is now a bounded, spilling memory pool.** Executors with
`--memory-pool-size` unset will now spill to disk under memory pressure instead
of running unbounded. Ensure the executor's `--work-dir` has adequate scratch
space.
- **`--memory-pool-size 0` now means "unbounded."** Previously an explicit
`0` would have failed at startup (per-vcore share of zero); it is now the
supported way to opt back into the old unbounded pool.
- **New `--memory-pool-fraction` flag** (default `0.70`).
- **`ExecutorProcessConfig::default()` changed:** library embedders and
standalone/example executors constructed via `..Default::default()` now get the
auto-sized bounded pool instead of the unbounded one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
