NoahKusaba commented on code in PR #2803:
URL:
https://github.com/apache/datafusion-ballista/pull/2803#discussion_r4581185388
##########
ballista/core/src/execution_plans/sort_shuffle/writer.rs:
##########
@@ -122,10 +130,19 @@ fn encode_buffered_partitions(
// One writer open at a time: each holds its own compression context.
buffered.drain(|partition, batches| {
let (buf, stat) = write_ipc_stream(Vec::new(), schema, opts, batches)?;
- encoded[partition] = buf;
- stats[partition] = stat;
+ // Not shrink_to_fit: under mimalloc it frees nothing but lowers
capacity().
+ if reservation.try_grow(buf.capacity()).is_ok() {
+ encoded[partition] = buf;
+ stats[partition] = stat;
+ } else {
+ let _timer = spill_time.timer();
+ spill_manager
+ .spill_stream(partition, &buf, stat)
+ .map_err(|e| DataFusionError::Execution(format!("{e:?}")))?;
+ }
Ok(())
})?;
+ reservation.resize(encoded.iter().map(Vec::capacity).sum());
Review Comment:
Done in fae18c88, the doc now covers the spilled case too.
##########
ballista/core/src/execution_plans/sort_shuffle/writer.rs:
##########
@@ -122,10 +130,19 @@ fn encode_buffered_partitions(
// One writer open at a time: each holds its own compression context.
buffered.drain(|partition, batches| {
let (buf, stat) = write_ipc_stream(Vec::new(), schema, opts, batches)?;
- encoded[partition] = buf;
- stats[partition] = stat;
+ // Not shrink_to_fit: under mimalloc it frees nothing but lowers
capacity().
+ if reservation.try_grow(buf.capacity()).is_ok() {
Review Comment:
Thanks, agreed. If it shows up as extra spills, I'll size the `Vec` up front
in a follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]