andygrove opened a new pull request, #2090:
URL: https://github.com/apache/datafusion-ballista/pull/2090
# Which issue does this PR close?
Closes #2089.
# Rationale for this change
`ballista.shuffle.sort_based.memory_limit_per_task_bytes`
(`SortShuffleConfig::memory_limit_per_task_bytes`) controls the point at
which
the sort shuffle writer spills its in-memory batches to disk. The scheduler
built `SortShuffleWriterExec` with the configured value, but the value was
not
carried in the `SortShuffleWriterExecNode` protobuf message. On decode, the
executor reconstructed the config with `SortShuffleConfig::new(true,
batch_size)`,
which discards the override and uses the built-in default. Because the writer
actually runs on the executor, the knob was effectively inert in distributed
execution.
# What changes are included in this PR?
- Add `memory_limit_per_task_bytes` to the `SortShuffleWriterExecNode`
message
as an `optional uint64` and populate it on encode. Presence lets a plan
produced before this field existed decode to the built-in default rather
than
to `0`.
- Apply the value on decode via `with_memory_limit_per_task_bytes`, so the
session override reaches the executor.
- Treat a value of `0` as "disable the per-task budget": the writer then
spills
only under runtime memory-pool pressure. Previously a `0` limit would have
forced a spill on every batch.
- Refresh the tuning guide and configs docs, which still documented removed
keys
(`buffer_size`, `memory_limit`, `spill_threshold`), and describe the
per-task
budget and the `0`-disables behavior.
# Are there any user-facing changes?
The configuration setting now takes effect on executors as documented.
Setting
it to `0` is newly meaningful: it disables the per-task budget and relies on
memory-pool pressure to trigger spilling. No public Rust API changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
