alamb commented on code in PR #24048: URL: https://github.com/apache/datafusion/pull/24048#discussion_r3707513745
########## datafusion/physical-plan/src/sorts/partial_sort.rs: ########## @@ -98,15 +98,82 @@ use log::trace; /// +---+---+---+ +---+---+---+ /// | a | b | c | | a | b | c | /// +---+---+---+ +---+---+---+ -/// | 0 | 0 | 3 | -- same group --> | 0 | 0 | 2 | -/// | 0 | 0 | 2 | | 0 | 0 | 3 | -/// | 0 | 1 | 1 | -- single row --> | 0 | 1 | 1 | -/// | 0 | 2 | 4 | -- same group --> | 0 | 2 | 0 | +/// | 0 | 0 | 3 | -- new group --> | 0 | 0 | 1 | +/// | 0 | 0 | 2 | | 0 | 0 | 2 | +/// | 0 | 0 | 1 | | 0 | 0 | 3 | +/// | 0 | 1 | 1 | -- new group --> | 0 | 1 | 1 | +/// | 0 | 2 | 4 | -- new group --> | 0 | 2 | 0 | /// | 0 | 2 | 0 | | 0 | 2 | 4 | -/// | 1 | 0 | 5 | -- single row --> | 1 | 0 | 5 | +/// | 1 | 0 | 5 | -- new group --> | 1 | 0 | 5 | /// +---+---+---+ +---+---+---+ /// ``` /// +/// # Buffering and Emitting Rows +/// +/// [`PartialSortExec`] buffers rows only until it can *prove* a prefix group +/// will never be seen again, then sorts and emits buffered rows. A group is +/// guaranteed to never be seen again once a row with a *different* prefix +/// value arrives. This relies on the input's existing ordering guarantees. +/// +/// Using the example from above, rows accumulate in the in-memory buffer in +/// batches. As long as the `(a, b)` prefix keeps repeating, more rows are +/// buffered. +/// +/// ```text +/// Buffer +/// +---+---+---+ +/// | a | b | c | +/// +---+---+---+ +/// | 0 | 0 | 3 | +/// | 0 | 0 | 2 | +/// | 0 | 0 | 1 | +/// +---+---+---+ +/// ``` +/// +/// Once a batch arrives that contains a new `(a, b)` prefix, e.g. `(0, 2)`: +/// every buffered row fpr previous prefixes may be emitted: Review Comment: ```suggestion /// every buffered row for previous prefixes may be emitted: ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
