andygrove commented on code in PR #2803:
URL:
https://github.com/apache/datafusion-ballista/pull/2803#discussion_r4579320901
##########
ballista/core/src/execution_plans/sort_shuffle/writer.rs:
##########
@@ -122,10 +130,19 @@ fn encode_buffered_partitions(
// One writer open at a time: each holds its own compression context.
buffered.drain(|partition, batches| {
let (buf, stat) = write_ipc_stream(Vec::new(), schema, opts, batches)?;
- encoded[partition] = buf;
- stats[partition] = stat;
+ // Not shrink_to_fit: under mimalloc it frees nothing but lowers
capacity().
+ if reservation.try_grow(buf.capacity()).is_ok() {
+ encoded[partition] = buf;
+ stats[partition] = stat;
+ } else {
+ let _timer = spill_time.timer();
+ spill_manager
+ .spill_stream(partition, &buf, stat)
+ .map_err(|e| DataFusionError::Execution(format!("{e:?}")))?;
+ }
Ok(())
})?;
+ reservation.resize(encoded.iter().map(Vec::capacity).sum());
Review Comment:
Small one. The doc on `InputPartitionOutput::encoded` still says a buffer is
empty only when this input buffered no rows for that partition. With this
change it's also empty when the pool refused the partition's encoded bytes and
they went to the spill file. Could you add that, so nobody reads an empty
buffer as "no rows"?
##########
ballista/core/src/execution_plans/sort_shuffle/writer.rs:
##########
@@ -122,10 +130,19 @@ fn encode_buffered_partitions(
// One writer open at a time: each holds its own compression context.
buffered.drain(|partition, batches| {
let (buf, stat) = write_ipc_stream(Vec::new(), schema, opts, batches)?;
- encoded[partition] = buf;
- stats[partition] = stat;
+ // Not shrink_to_fit: under mimalloc it frees nothing but lowers
capacity().
+ if reservation.try_grow(buf.capacity()).is_ok() {
Review Comment:
Your reasoning on `capacity()` over `shrink_to_fit` makes sense to me.
Over-counting is the right side to err on. One thing to keep in mind for later:
with `compression_codec = none`, the encoded bytes are about the buffered size,
so capacity doubling can roughly triple what the end of input asks for. If that
ever shows up as extra spills, sizing the `Vec` up front would be an easy
follow-up. No change needed here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]