Dandandan commented on PR #21580: URL: https://github.com/apache/datafusion/pull/21580#issuecomment-4266697066
Another idea (maybe even more impactful/generally applicable than the pure - stats): Currently many queries go like Scan -> Filter -> Aggregate(Partial) -> Aggregate(Final) -> TopK Needs to first go through the the full aggregation before it reaches Topk (so it's not useful to push down to scan) We can update bounds from aggregate, once we detect at least K distinct rows in a Partial aggregate. In practice it would mean after the first batch / batches arrive we already have e.g. 10 rows, so can already prune / filter with the much tighter bound. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
