2010YOUY01 commented on code in PR #24048: URL: https://github.com/apache/datafusion/pull/24048#discussion_r3697832030
########## datafusion/physical-plan/src/sorts/partial_sort.rs: ########## @@ -98,15 +98,82 @@ use log::trace; /// +---+---+---+ +---+---+---+ /// | a | b | c | | a | b | c | /// +---+---+---+ +---+---+---+ -/// | 0 | 0 | 3 | -- same group --> | 0 | 0 | 2 | -/// | 0 | 0 | 2 | | 0 | 0 | 3 | -/// | 0 | 1 | 1 | -- single row --> | 0 | 1 | 1 | -/// | 0 | 2 | 4 | -- same group --> | 0 | 2 | 0 | +/// | 0 | 0 | 3 | -- new group --> | 0 | 0 | 1 | +/// | 0 | 0 | 2 | | 0 | 0 | 2 | +/// | 0 | 0 | 1 | | 0 | 0 | 3 | +/// | 0 | 1 | 1 | -- new group --> | 0 | 1 | 1 | +/// | 0 | 2 | 4 | -- new group --> | 0 | 2 | 0 | /// | 0 | 2 | 0 | | 0 | 2 | 4 | -/// | 1 | 0 | 5 | -- single row --> | 1 | 0 | 5 | +/// | 1 | 0 | 5 | -- new group --> | 1 | 0 | 5 | /// +---+---+---+ +---+---+---+ /// ``` /// +/// # Buffering and Emitting Rows +/// +/// [`PartialSortExec`] buffers rows only until it can *prove* a prefix group +/// will never be seen again, then sorts and emits buffered rows. A group is +/// guaranteed to never be seen again once a row with a *different* prefix +/// value arrives. This relies on the input's existing ordering guarantees. +/// +/// Using the example from above, rows accumulate in the in-memory buffer in +/// batches. As long as the `(a, b)` prefix keeps repeating, more rows are +/// buffered. +/// +/// ```text +/// Buffer +/// +---+---+---+ +/// | a | b | c | +/// +---+---+---+ +/// | 0 | 0 | 3 | +/// | 0 | 0 | 2 | +/// | 0 | 0 | 1 | +/// +---+---+---+ +/// ``` +/// +/// Once a batch arrives that contains a new `(a, b)` prefix, e.g. `(0, 2)`: +/// every buffered row fpr previous prefixes may be emitted: +/// +/// ```text +/// Buffer +/// +---+---+---+ +/// | a | b | c | +/// +---+---+---+ +/// | 0 | 0 | 3 | +/// | 0 | 0 | 2 | +/// | 0 | 0 | 1 | +/// | 0 | 1 | 1 | <-- first row of new batch, new prefix +/// | 0 | 2 | 4 | <-- new prefix +/// | 0 | 2 | 0 | +/// | 1 | 0 | 5 | <-- last row of new batch, new prefix +/// +---+---+---+ +/// ``` +/// +/// Once known complete, the uffered rows are sorted by the full `(a, b, c)` Review Comment: ```suggestion /// Once known complete, the buffered rows are sorted by the full `(a, b, c)` ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
