zhuqi-lucas commented on PR #21580: URL: https://github.com/apache/datafusion/pull/21580#issuecomment-4267114384
> Another idea (maybe even more impactful/generally applicable than the pure - stats): > > Currently many queries go like Scan -> Filter -> Aggregate(Partial) -> Aggregate(Final) -> TopK > > Needs to first go through the the full aggregation before it reaches Topk (so it's not useful to push down to scan as we usually scan the entire dataset before emitting the partial aggregate) > > We can update bounds from aggregate, once we detect at least K distinct rows in a Partial aggregate. > > In practice it would mean after the first batch / batches arrive we already have e.g. 10 rows, so can already prune / filter with the much tighter bound. Amazing idea! > But let's focus first on getting the ordering in 😆 Sure 😊 , i will let benchmark PR merged soon, and we can see the performance improvement more easily. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
