adriangb commented on PR #15697: URL: https://github.com/apache/datafusion/pull/15697#issuecomment-2803022524
> Thanks @Dandandan and @2010YOUY01 and @adriangb and @geoffreyclaude ! > > > > One thing I was wondering about for this PR is how much will it help once we implement actual topk filter pushdown into the scan (aka https://github.com/apache/datafusion/issues/15037) > > > > I am thinking that the topk filter pushdown will already filter out rows that are known not to be in the topK > > > > Specifically, once we implement topk filter pushdown the rows should already be filtered and so checking again in the TopK itself won't add any benefit, will it? > > > > Yes that's right for Parquet, but not all data sources support filter pushdown, so there's still benefit for those. But yeah, I'm hoping we can structure this in a way that we get an immediate win that justifies the change but also introduces all of the code necessary for filter push down later on. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
