thinkharderdev commented on code in PR #7192:
URL: https://github.com/apache/arrow-datafusion/pull/7192#discussion_r1284462553
##########
datafusion/core/src/physical_optimizer/limit_aggregation.rs:
##########
@@ -0,0 +1,67 @@
+use std::sync::Arc;
+use datafusion_common::config::ConfigOptions;
+use crate::physical_optimizer::PhysicalOptimizerRule;
+use crate::physical_plan::ExecutionPlan;
+use crate::physical_plan::sorts::sort::SortExec;
+use datafusion_common::{DataFusionError, Result};
+use crate::physical_plan::aggregates::AggregateExec;
+
+pub struct LimitAggregation {
+
+}
+
+impl LimitAggregation {
+ pub fn new() -> Self {
+ Self {}
+ }
+
+ fn recurse(plan: Arc<dyn ExecutionPlan>) -> Result<Arc<dyn ExecutionPlan>>
{
+ if let Some(sort) = plan.as_any().downcast_ref::<SortExec>() {
+ let children = sort.children();
+ let child = match children.as_slice() {
+ [] => Err(DataFusionError::Execution("Sorts should have
children".to_string()))?,
+ [child] => child,
+ _ => Err(DataFusionError::Execution("Sorts should have 1
child".to_string()))?,
+ };
+ let binding = (*child).as_any();
+ if let Some(aggr) = binding.downcast_ref::<AggregateExec>() {
Review Comment:
Yeah, I think for this to work you either need something like min/max where
one partition "wins" when the partial states are merged or you need to have the
total order of groups consistent between partitions. I'm sure you could come up
with an aggregation function which has the latter property (or have the data
partitioned a certain way in which it would be true for standard aggregation
functions) but in practice it's probably not the case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
