alamb commented on code in PR #27211: URL: https://github.com/apache/datafusion/pull/27211#discussion_r4503769337
########## datafusion/physical-optimizer/src/topk_aggregation.rs: ########## @@ -18,12 +18,78 @@ use datafusion_physical_plan::projection::ProjectionExec; use datafusion_physical_plan::sorts::sort::SortExec; use itertools::Itertools; -/// An optimizer rule that passes a `limit` hint to aggregations if the whole result is not needed +/// An optimizer rule that transforms Sort + Limit above a grouped Aggregate to use +/// a more efficient Grouped TopK aggregation implementation when possible. +/// +/// For example, a query like +/// +/// ```sql +/// SELECT trace_id, MAX(timestamp) +/// FROM traces +/// GROUP BY trace_id +/// ORDER BY MAX(timestamp) DESC +/// LIMIT 10 +/// ``` +/// +/// is planned as a `SortExec` with a `fetch` above an `AggregateExec` (left). +/// After this rule, the `AggregateExec` uses a [`GroupedTopKAggregateStream`] +/// that retains only the best `N` groups (right). +/// +/// [`GroupedTopKAggregateStream`]: mod@datafusion_physical_plan::aggregates#4-grouped-topk-aggregation +/// +/// ```text Review Comment: As you probably know I love the ascii art -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
