andygrove opened a new issue, #7072:
URL: https://github.com/apache/datafusion-comet/issues/7072
### What is the problem the feature request solves?
`MultiPartitionShuffleRepartitioner::buffer_partitioned_batch_may_spill` in
`native/shuffle/src/partitioners/multi_partition.rs` pushes the row indices of
each partition one at a time after `reserve`:
```rust
indices.reserve(row_indices.len());
for row_idx in row_indices {
indices.push((buffered_partition_idx, *row_idx));
}
```
`Vec::push` re-checks capacity on every iteration.
`indices.extend(row_indices.iter().map(|&r| (buffered_partition_idx, r)))` gets
the trusted-length specialization and writes the tuples in one pass.
### Describe the potential solution
Replace the loop with `extend`. Run `cargo bench --bench shuffle_writer`
before and after (save a baseline with `--save-baseline before`); the change is
small, so report the result either way and drop it if it is within noise.
### Additional context
Micro item from #5002, carried into #5905. Good first issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
