andygrove opened a new issue, #2674: URL: https://github.com/apache/datafusion-ballista/issues/2674
## Describe the issue Two throughput questions about `PipelinedShuffleWriterExec` (#2670, part of #2652) that need a benchmark before anything changes: 1. **One coalescer per output, shared by every input.** A hash writer keeps a `Mutex<BatchCoalescer>` per output partition, and every input task of the stage locks it to push its slice of rows. The lock is held while the coalescer copies the rows, and the rows are copied twice: once by `BatchPartitioner::partition_iter` to build the slice, once by the coalescer to append it. Per-input coalescers would remove the lock at a memory cost of inputs × outputs × batch size. Feeding the coalescer directly from the partition indices would remove the first copy. 2. **Sends are awaited in partition order.** `drive_hashed` sends each coalesced batch in turn, so one full queue (default `ballista.pipelined.queue_capacity` is 2) stalls every other output of that input until its consumer catches up. Sending through a `FuturesUnordered` or a per-output task would let a full queue stall only itself. Related design question from the same review: `RepartitionExec` already hash-partitions and coalesces. The writer partitions rows itself, as `ShuffleWriterExec` does, to avoid a channel hop and a second set of spawned tasks per input; it would be worth measuring whether wrapping the input in `RepartitionExec` is close enough to drop the writer's own hash path. ## Proposed change Add the pipelined writer to the shuffle benchmark, then try the three changes above against it. ## Context Raised in the review of #2670. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
