andygrove opened a new issue, #1981:
URL: https://github.com/apache/datafusion-ballista/issues/1981
**Describe the bug**
The `TPC-H SF10 (all queries, AQE off + on)` CI job
(`.github/workflows/tpch.yml`) intermittently fails with `No space left on
device (os error 28)` while writing/spilling shuffle files. It is
non-deterministic — the same commit and even `main` pass on some runs and fail
on others — so it is a flaky, infra-level failure rather than a code
regression. It shows up as red X's on unrelated PRs and erodes trust in CI
signal.
The failure surfaces two ways depending on where the disk fills:
```
Task 27 failed: DataFusion error: Arrow error: Io error:
No space left on device (os error 28)
```
```
called `Result::unwrap()` on an `Err` value:
External(Os { code: 28, kind: StorageFull, message: "No space left on
device" })
```
**To Reproduce**
Not deterministically reproducible (that's the problem), but recent examples
on 2026-07-09:
- `main` @ `fd30f37` (PR #1973 merge commit) — SF10 failed, `StorageFull`.
- PR #1954 branch `reuse-exchange` @ `ebfba6a` — SF10 failed, `No space left
on device`; the *same branch* passed SF10 on several earlier commits the same
day.
Job configuration for context:
- `runs-on: ubuntu-latest` (standard GitHub-hosted runner, ~14 GB free disk)
inside a container.
- Generates TPC-H **scale factor 10** data into `$RUNNER_TEMP/tpch-data`,
then runs a scheduler + executor cluster and executes all 22 queries with AQE
off and on.
- Shuffle files are written (LZ4-compressed) to the executor work dir on the
same volume, on top of the SF10 source data and build artifacts.
So the disk holds: build artifacts + SF10 source parquet + shuffle spill for
all 22 queries × 2 AQE modes — and occasionally that exceeds the runner's free
space.
**Expected behavior**
SF10 CI should be a reliable signal — either it always has enough headroom
to complete, or a genuine failure is distinguishable from "the runner ran out
of disk."
**Additional context**
Filing this so solutions can be discussed rather than proposing one. Some
options that have come up / are worth weighing:
- **Free up runner disk** before the job (e.g. remove preinstalled
toolchains/SDKs, `docker image prune`) — cheap, common GitHub-Actions
mitigation, buys several GB.
- **Reduce on-disk footprint** — clean up per-query shuffle output between
queries, and/or point shuffle work dir and TPC-H data at different volumes if
available.
- **Denser shuffle compression** — the shuffle codec is currently hard-coded
to LZ4; a ZSTD option would shrink the shuffle footprint. Tracked separately in
#1980.
- **Larger runner** — a runner/volume with more disk (if the project has
access to larger hosted or self-hosted runners).
- **Lower the scale factor or split the matrix** — smaller SF, or splitting
AQE-off / AQE-on into separate jobs so peak concurrent disk use is lower.
- **Surface disk usage** — a `df -h` step before/after to quantify actual
headroom and confirm which artifact dominates.
Related: #1980 (configurable/denser shuffle compression codec).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
