ariel-miculas commented on code in PR #23619:
URL: https://github.com/apache/datafusion/pull/23619#discussion_r3594613135
##########
datafusion/physical-plan/src/sorts/cursor.rs:
##########
@@ -113,15 +123,23 @@ impl<T: CursorValues> Cursor<T> {
t
}
- pub fn is_eq_to_prev_one(&self, prev_cursor: Option<&Cursor<T>>) -> bool {
+ pub fn is_eq_to_prev_one(&self, prev_value: Option<&T::SingleRowValue>) ->
bool {
if self.offset > 0 {
self.is_eq_to_prev_row()
- } else if let Some(prev_cursor) = prev_cursor {
- self.is_eq_to_prev_row_in_prev_batch(prev_cursor)
+ } else if let Some(prev_value) = prev_value {
+ T::eq_to_single_row_value(&self.values, self.offset, prev_value)
} else {
false
}
}
+
+ /// Extract an owned copy of the last row in this cursor, decoupled from
+ /// any buffer the cursor's [`CursorValues`] holds. Used to remember a
+ /// partition's last row across a batch boundary without keeping the
+ /// whole exhausted batch's memory alive (see [`Self::is_eq_to_prev_one`]).
+ pub fn last_value(&self) -> T::SingleRowValue {
Review Comment:
values() shouldn't be empty, there are some asserts in the code:
```
assert!(rows.num_rows() > 0);
assert!(array.len() > 0, "Empty array passed to FieldCursor");
```
I don't imagine a Cursor with empty values being useful.
```
impl RowValues {
/// Create a new [`RowValues`] from `rows` and a `reservation`
/// that tracks its memory. There must be at least one row
///
/// Panics if the reservation is not for exactly `rows.size()`
/// bytes or if `rows` is empty.
pub fn new(rows: Arc<Rows>, reservation: MemoryReservation) -> Self {
assert_eq!(
rows.size(),
reservation.size(),
"memory reservation mismatch"
);
assert!(rows.num_rows() > 0);
Self {
rows,
_reservation: reservation,
}
}
}
```
```
pub fn new<A: CursorArray<Values = T>>(
options: SortOptions,
array: &A,
reservation: MemoryReservation,
) -> Self {
assert!(array.len() > 0, "Empty array passed to FieldCursor");
let null_threshold = match options.nulls_first {
true => array.null_count(),
false => array.len() - array.null_count(),
};
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
