andygrove opened a new issue, #1980:
URL: https://github.com/apache/datafusion-ballista/issues/1980
**Is your feature request related to a problem or challenge? Please describe
what you are trying to do.**
Ballista compresses shuffle files with a hard-coded `LZ4_FRAME` codec. There
is no way to change the codec, and no way to disable compression. The codec is
fixed in two places:
- `ballista/core/src/execution_plans/shuffle_writer.rs` (the
`ShuffleWriterExec` partition-file writer)
- `ballista/core/src/utils.rs` (`write_stream_to_disk`)
Both do:
```rust
let options = IpcWriteOptions::default()
.try_with_compression(Some(CompressionType::LZ4_FRAME))?;
```
LZ4 favors throughput over ratio. On disk-constrained environments this
matters — for example, the `TPC-H SF10` CI job intermittently fails with `No
space left on device (os error 28)` while spilling/writing shuffle files on
GitHub runners. A denser codec such as ZSTD would trade some CPU for a smaller
shuffle footprint, which would help both CI and real deployments that are I/O-
or storage-bound rather than CPU-bound. Different workloads want different
points on that trade-off, so it should be configurable rather than one
hard-coded choice.
**Describe the solution you'd like**
Make the shuffle-write compression codec configurable via `BallistaConfig`
(`ballista/core/src/config.rs`), plumbed through to both write sites above.
Concretely:
- A config key such as `ballista.shuffle.compression.codec` accepting `none`
| `lz4` | `zstd`, defaulting to `lz4` to preserve current behavior.
- Apply the resulting `Option<CompressionType>` to
`IpcWriteOptions::try_with_compression` in `shuffle_writer.rs` and `utils.rs`
(a single shared helper would keep the two paths in sync).
**Describe alternatives you've considered**
- Keeping LZ4 hard-coded and just bumping CI runner disk / lowering scale
factor — treats the symptom, not the general need for storage/CPU trade-off
control.
- A configurable compression *level* as well: worth noting Arrow's
`IpcWriteOptions::try_with_compression` only accepts a `CompressionType`
(codec), not a level — ZSTD level is fixed inside `arrow-ipc`. So level tuning
can't be exposed at the IPC layer today without upstream Arrow support; this
issue therefore focuses on codec selection (incl. disabling). Level
configurability can be a follow-up gated on Arrow exposing it.
**Additional context**
Arrow IPC currently supports `LZ4_FRAME` and `ZSTD` (`arrow-ipc`
`CompressionType`), so ZSTD requires no new dependency. The Flight serve path
(`ballista/executor/src/flight_service.rs`) also advertises `LZ4_FRAME`;
readers negotiate the codec from the IPC stream metadata, so changing the write
codec is transparent to `ShuffleReaderExec` without a format-version bump.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
