Rich-T-kid opened a new issue, #11141:
URL: https://github.com/apache/arrow-rs/issues/11141
BatchCoalescer already has `push_batch_with_indices(batch, indices: &dyn
Array)` for the single-batch take case. There is no equivalent for the
multi-batch case.
Callers that need to gather rows from multiple source batches must today
call `interleave_record_batch` (copy 1) and then `push_batch` the result (copy
2). The intermediate batch is immediately discarded.
## Proposed API
```rust
/// Semantically equivalent to `interleave_record_batch(batches, indices)`
/// followed by `push_batch`, without the intermediate RecordBatch
allocation.
pub fn push_batch_interleaved(
&mut self,
batches: &[RecordBatch],
indices: &[(usize, usize)], // (batch_idx, row_idx)
) -> Result<(), ArrowError>
```
Known callers in DataFusion that would benefit (tracked in (TODO make
datafusion issue)):
- Top-K; `TopKHeap::emit_with_state` calls interleave_record_batch to
reconstruct k surviving rows from the RecordBatchStore, then immediately pushes
the result to a BatchCoalescer.
- Hash join / nested loop join; `build_batch_from_indices` or
`interleave_record_batch` to materialize matched row pairs (copy 1), then push
to the coalescer (copy 2).
- Sort merge join; join rows go through a materialization step before
reaching the coalescer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
