alamb commented on PR #15301: URL: https://github.com/apache/datafusion/pull/15301#issuecomment-2759361675
> My main concern is that regardless of what phase (optimizer or runtime) the pushdown happens if what we push down is a custom PhysicalExpr we're going to have trouble making that play nice with PruningPredicate & co, which is where a lot of the performance win is coming from. The pruning predicate is a good point 🤔 it seems like the limitation is that PruningPredicate has a bunch of hard coded rules that needs to be rewritten. We would have to teach it somehow that this TopK column could be treated the same as a `Column` > A secondary concern is that feel like writing an optimizer rule for this will be harder, but maybe that's just because I'm not good at doing it, my first attempt was precisely this and I failed and found this approach much more tractable. I think we can help with that as we get closer to merge -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
