andygrove opened a new issue, #6917: URL: https://github.com/apache/datafusion-comet/issues/6917
### What is the problem the feature request solves? For hash partitioning, `MultiPartitionShuffleRepartitioner` records one `(batch, row)` pair per row (`PartitionIndices::Rows`) and assembles each partition's output with `interleave_record_batch`, which walks every column and every nested child for each row. Positional round robin records contiguous runs instead and copies them in bulk, as the comment on `PartitionIndices` explains. In `CometIcebergEndToEndBenchmark` (#6910), the gather is 9.2% of all CPU for nested types, mostly `interleave_list`, `interleave_bytes` and `interleave_map`, and 7.5% for primitives. Every byte it copies is then copied again by IPC encoding and read once more by LZ4. ### Describe the potential solution Reorder each input batch by partition id when it is inserted, with one `take` over the batch using the partition ids already computed, so that each partition's rows from that batch form one contiguous run and the flush can copy runs instead of interleaving rows. The reorder is a copy of its own, so this needs measuring against the current path. Faster `interleave` for list and map types in arrow-rs would also help. ### Additional context `native/shuffle/src/partitioners/multi_partition.rs` and `native/shuffle/src/partitioners/partitioned_batch_iterator.rs`. Measured on the fully native configuration of `CometIcebergEndToEndBenchmark`. Part of #6911. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
