alamb commented on code in PR #2182:
URL: https://github.com/apache/arrow-datafusion/pull/2182#discussion_r846387172
##########
datafusion/core/tests/sql/order.rs:
##########
@@ -198,3 +199,25 @@ async fn sort_empty() -> Result<()> {
assert_eq!(results.len(), 0);
Ok(())
}
+
+#[tokio::test]
+async fn sort_with_lots_of_repetition_values() -> Result<()> {
Review Comment:
BTW I ran the test locally without the changes in this PR to confirm
coverage:
```shell
cargo test -p datafusion --test sql_integration --
sort_with_lots_of_repetition_values
```
They failed with:
```
---- sql::order::sort_with_lots_of_repetition_values stdout ----
thread 'sql::order::sort_with_lots_of_repetition_values' panicked at
'assertion failed: `(left == right)`
left: `Some(2451809)`,
right: `Some(2451816)`', datafusion/core/tests/sql/order.rs:220:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
```
👍
##########
datafusion/core/src/physical_plan/sorts/sort.rs:
##########
@@ -426,52 +426,78 @@ impl Iterator for SortedIterator {
// Combine adjacent indexes from the same batch to make a slice,
// for more efficient `extend` later.
let mut last_batch_idx = 0;
- let mut start_row_idx = 0;
- let mut len = 0;
+ let mut indices_in_batch = vec![];
let mut slices = vec![];
for i in 0..current_size {
let p = self.pos + i;
let c_index = self.indices.value(p) as usize;
let ci = self.composite[c_index];
- if len == 0 {
+ if indices_in_batch.is_empty() {
last_batch_idx = ci.batch_idx;
- start_row_idx = ci.row_idx;
- len = 1;
+ indices_in_batch.push(ci.row_idx);
} else if ci.batch_idx == last_batch_idx {
- len += 1;
- // since we have pre-sort each of the incoming batches,
- // so if we witnessed a wrong order of indexes from the same
batch,
- // it must be of the same key with the row pointed by
start_row_index.
- start_row_idx = min(start_row_idx, ci.row_idx);
+ indices_in_batch.push(ci.row_idx);
} else {
- slices.push(CompositeSlice {
- batch_idx: last_batch_idx,
- start_row_idx,
- len,
- });
+ let indices = indices_in_batch.drain(..).collect::<Vec<_>>();
+ group_indices(last_batch_idx, indices, &mut slices);
last_batch_idx = ci.batch_idx;
- start_row_idx = ci.row_idx;
- len = 1;
+ indices_in_batch.push(ci.row_idx);
}
}
assert!(
- len > 0,
+ !indices_in_batch.is_empty(),
"There should have at least one record in a sort output slice."
);
- slices.push(CompositeSlice {
- batch_idx: last_batch_idx,
- start_row_idx,
- len,
- });
+ let indices = indices_in_batch.drain(..).collect::<Vec<_>>();
+ group_indices(last_batch_idx, indices, &mut slices);
self.pos += current_size;
Some(slices)
}
}
+/// Group continuous indices into a slice for better `extend` performance
Review Comment:
I don't really understand how positions can be non contiguous (doesn't this
function get called each time `batch_idx` changes to a new batch)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
