andygrove opened a new issue, #2089:
URL: https://github.com/apache/datafusion-ballista/issues/2089
## Describe the bug
The sort-based shuffle writer has a per-task buffered-bytes budget,
`ballista.shuffle.sort_based.memory_limit_per_task_bytes`
(`SortShuffleConfig::memory_limit_per_task_bytes`), that controls the point
at
which the writer spills its in-memory batches to disk.
The scheduler builds `SortShuffleWriterExec` with the configured value in
`ballista/scheduler/src/planner.rs`:
```rust
SortShuffleConfig::new(...)
.with_memory_limit_per_task_bytes(
ballista_config.shuffle_sort_based_memory_limit_per_task_bytes(),
)
```
However, the value is not carried in the `SortShuffleWriterExecNode` protobuf
message (`ballista/core/proto/ballista.proto`), and the executor-side decode
in
`ballista/core/src/serde/mod.rs` reconstructs the config with:
```rust
let config = SortShuffleConfig::new(true, batch_size);
```
which ignores the configured value. Only `batch_size` survives the round
trip.
## To Reproduce
Set `ballista.shuffle.sort_based.memory_limit_per_task_bytes` to a
non-default
value in the session config and run a query whose plan uses the sort-based
shuffle writer. The executors that run the writer use the built-in default,
not
the configured value.
## Impact
The setting has no effect on executors — where the shuffle writer actually
runs
— so the knob is effectively inert in distributed execution. Executors always
use the built-in value regardless of the session configuration.
## Expected behavior
The configured `memory_limit_per_task_bytes` should propagate to executors
so the
override takes effect.
## Suggested fix
Add `memory_limit_per_task_bytes` to the `SortShuffleWriterExecNode` message
and
apply it on decode (via `with_memory_limit_per_task_bytes`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
