andygrove opened a new issue, #2660:
URL: https://github.com/apache/datafusion-ballista/issues/2660
**Describe the bug**
The standalone `ballista-executor` binary writes DataFusion spill files to
the OS temp directory instead of `--work-dir`.
`ExecutorProcessConfig`'s default runtime producer roots the disk manager at
the work dir:
```rust
RuntimeEnvBuilder::new()
.with_temp_file_path(wd.clone())
.build()?
```
but `ballista/executor/src/bin/main.rs` replaces it with
`runtime_env_with_s3_support` (`ballista/core/src/object_store.rs`), which
builds its runtime with `RuntimeEnvBuilder::new()` and never sets a temp file
path. DataFusion's disk manager then falls back to the OS temp directory
(`/tmp` in the container image).
The tuning guide says the opposite:
> Because operators spill to disk rather than fail, make sure the executor's
`--work-dir` has enough scratch space to absorb spills under memory pressure.
**To Reproduce**
Run `ballista-executor --work-dir /work ...` and execute a query that spills
(e.g. TPC-H q5 at SF1000 with a bounded memory pool). Spill files appear under
the OS temp dir, not `/work`.
**Expected behavior**
Spill files go to `--work-dir`, as with the default runtime producer and as
the docs describe, regardless of whether the S3-enabled runtime is used.
**Additional context**
This matters in Kubernetes deployments, where `--work-dir` is typically a
mounted volume sized for shuffle and spill data while `/tmp` is the container's
writable layer: large spills fill the node's root filesystem (or hit
ephemeral-storage limits and get the pod evicted) instead of using the volume
provisioned for them. At SF1000 a single sort-merge-join stage spills well over
100 GB per executor (see #2658).
A fix could have `runtime_env_with_s3_support` take the work dir, or apply
the temp path on top of any runtime producer the way #2659 applies
`--max-temp-directory-size`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
