alamb commented on code in PR #2132:
URL: https://github.com/apache/arrow-datafusion/pull/2132#discussion_r842657428
##########
datafusion/core/src/physical_plan/sorts/sort.rs:
##########
@@ -497,6 +497,49 @@ impl Stream for SortedSizedRecordBatchStream {
}
}
+struct CompositeSlice {
+ batch_idx: u32,
+ start_row_idx: u32,
+ len: usize,
+}
+
+/// Combine adjacent indexes from the same batch to make a slice, for more
efficient `extend` later.
+fn combine_adjacent_indexes(combined: Vec<CompositeIndex>) ->
Vec<CompositeSlice> {
+ let mut last_batch_idx = 0;
+ let mut start_row_idx = 0;
+ let mut len = 0;
+
+ let mut slices = vec![];
+ for ci in combined {
+ if len == 0 {
+ last_batch_idx = ci.batch_idx;
+ start_row_idx = ci.row_idx;
+ len = 1;
+ } else if ci.batch_idx == last_batch_idx {
+ len += 1;
+ // since we have pre-sort each of the incoming batches,
+ // so if we witnessed a wrong order of indexes from the same batch,
+ // it must be of the same key with the row pointed by
start_row_index.
Review Comment:
if this happened, does it mean the sort wasn't stable (as in it rearranged
inputs that sorted to equal keys)?
##########
datafusion/core/src/physical_plan/sorts/sort.rs:
##########
@@ -497,6 +497,49 @@ impl Stream for SortedSizedRecordBatchStream {
}
}
+struct CompositeSlice {
+ batch_idx: u32,
+ start_row_idx: u32,
+ len: usize,
+}
+
+/// Combine adjacent indexes from the same batch to make a slice, for more
efficient `extend` later.
+fn combine_adjacent_indexes(combined: Vec<CompositeIndex>) ->
Vec<CompositeSlice> {
+ let mut last_batch_idx = 0;
+ let mut start_row_idx = 0;
+ let mut len = 0;
+
+ let mut slices = vec![];
+ for ci in combined {
+ if len == 0 {
+ last_batch_idx = ci.batch_idx;
+ start_row_idx = ci.row_idx;
+ len = 1;
+ } else if ci.batch_idx == last_batch_idx {
+ len += 1;
+ // since we have pre-sort each of the incoming batches,
+ // so if we witnessed a wrong order of indexes from the same batch,
+ // it must be of the same key with the row pointed by
start_row_index.
+ start_row_idx = min(start_row_idx, ci.row_idx);
+ } else {
+ slices.push(CompositeSlice {
+ batch_idx: last_batch_idx,
+ start_row_idx,
+ len,
+ });
+ last_batch_idx = ci.batch_idx;
+ start_row_idx = ci.row_idx;
+ len = 1;
+ }
+ }
+ slices.push(CompositeSlice {
Review Comment:
Is it important to check here that `len > 0`? I don't think it is possible
to try and sort a 0 length `RecordBatch` but I wonder what would happen in that
case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
