mbutrovich commented on code in PR #3858:
URL: https://github.com/apache/datafusion-comet/pull/3858#discussion_r3028928569
##########
native/shuffle/src/partitioners/multi_partition.rs:
##########
@@ -523,6 +545,69 @@ impl MultiPartitionShuffleRepartitioner {
})
}
+ /// Fast path for zero-column schemas: distribute total_rows evenly across
partitions
+ /// and emit one RecordBatch per partition carrying the row count.
+ /// Skips interleave, coalesce, and spill entirely.
+ fn shuffle_write_zero_col(
+ &self,
+ total_rows: usize,
+ start_time: Instant,
+ ) -> datafusion::common::Result<()> {
+ let num_output_partitions = self.partition_writers.len();
+ let mut offsets = vec![0u64; num_output_partitions + 1];
+
+ let output_data = OpenOptions::new()
+ .write(true)
+ .create(true)
+ .truncate(true)
+ .open(&self.output_data_file)
+ .map_err(|e| DataFusionError::Execution(format!("shuffle write
error: {e:?}")))?;
+ let mut output_data = BufWriter::with_capacity(self.write_buffer_size,
output_data);
+
+ // Distribute rows evenly: each partition gets total/N, first
(total%N) get one extra
+ let base = total_rows / num_output_partitions;
Review Comment:
I'm still a bit confused why we partition at all in this case? Why not send
all `num_rows` count to partition 0 in one batch, and leave the others empty?
You'd effectively be doing the final aggregation/partition coalescing at this
step, so I have no idea if that's valid for all aggregations that could yield
this shuffle scenario, but it seems we're doing extra work here just to decode
O(partitions) IPC batches with `num_rows` on the other side of the shuffle and
final aggregation.
That's what I was originally thinking when we could catch this early and
just write a single value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
