andygrove opened a new issue, #2658:
URL: https://github.com/apache/datafusion-ballista/issues/2658
**Describe the bug**
Executors cannot raise DataFusion's spill disk limit, so any stage whose
spills exceed 100 GB on one executor fails, even with terabytes of free disk:
```
Task failed due to runtime execution error:
DataFusionError(Shared(ArrowError(IoError("The used disk space during the
spilling process has exceeded the allowable limit of 100.0 GB. Please try
increasing the config: `datafusion.runtime.max_temp_directory_size`." ...
```
The error points at `datafusion.runtime.max_temp_directory_size`, but there
is no way to apply it on an executor:
- `datafusion.runtime.*` settings are not part of `SessionConfig`.
DataFusion applies them only to the local `RuntimeEnv` via `SessionContext`'s
`SET` handling, so a client-side setting never reaches executors.
- The executor builds its runtime with `RuntimeEnvBuilder::new()` (the
default producer in `executor_process.rs`, and `runtime_env_with_s3_support` in
the standalone binary), which always gets DataFusion's 100 GB default.
- There is no executor CLI flag or Ballista config key for it.
**To Reproduce**
TPC-H q5 at SF1000 on 2 executors (`--memory-pool-size=48GB`), with the
shuffle-reader memory fix for #2656 applied so the stage spills instead of
OOMing. Stage 4 sorts ~6B `lineitem` rows; each executor's sort spills more
than 100 GB and the job fails with the error above.
**Expected behavior**
Operators can size an executor's spill space to its disk, e.g. an executor
flag `--max-temp-directory-size=1TB` applied to every task runtime the executor
creates, including runtimes from a custom `override_runtime_producer`.
**Additional context**
Found while validating the fix for #2656, which removes the OOM that
previously killed this stage before it reached the spill limit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
