andygrove opened a new issue, #2031: URL: https://github.com/apache/datafusion-ballista/issues/2031
Ballista's executor memory accounting relies entirely on voluntary DataFusion `MemoryPool` reservations. `executor_process.rs` splits `--memory-pool-size` across `--concurrent-tasks` and gives each task a `FairSpillPool`, but that pool only sees bytes an operator explicitly reserves through a `MemoryReservation`. It does not see Arrow buffer allocations made outside a reservation, join scratch space, expression-kernel temporaries, or shuffle read/write and gRPC framing buffers. Real RSS therefore runs well above what the pool believes is reserved. Two problems follow. **1. The executor process gets OOM-killed.** This is much worse in Ballista than a single failed task: the dead executor takes *all of its shuffle output files* with it, so every downstream stage that was going to read from it raises `FetchPartitionError`, which cascades into stage rollbacks and re-runs of the map stages — potentially across other concurrent jobs on the same cluster. **2. The pool spills too late, or not at all,** because it is measuring the wrong quantity. The only remedy today is to hand-tune `--memory-pool-size` down to some workload-specific fraction of the container limit. Separately, and independent of the above: a task that *does* exhaust its budget raises `DataFusionError::ResourcesExhausted`, which `From<BallistaError> for FailedTask` currently maps to `FailedReason::ExecutionError`. The scheduler fails the whole stage on that reason with no retry, so a transient memory spike kills the job outright. ### Proposed approach Track the bytes the global allocator actually hands out, and use that signal to (a) reject memory-pool growth before real usage exceeds the budget, so DataFusion spills instead of OOMing, and (b) as a last resort, fail a single task rather than let the process die. Classify `ResourcesExhausted` as a retriable task failure so the scheduler reschedules the task. Prior art: apache/datafusion-comet#4582 explores the same idea for Comet. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
