adriangb commented on PR #15301: URL: https://github.com/apache/datafusion/pull/15301#issuecomment-2758981967
Thank you all for the feedback. @alamb I think the main thing to figure out is if this should happen during an optimizer pass and what gets pushed down is an `Arc<dyn PhysicalExpr>` or if we continue to do it at runtime via the `DynamicFilterSource` trait I introduced (discussed in https://github.com/apache/datafusion/pull/15301#discussion_r2015081219 and https://github.com/apache/datafusion/pull/15301#discussion_r2006853236). My main concern is that regardless of what phase (optimizer or runtime) the pushdown happens if what we push down is a custom `PhysicalExpr` we're going to have trouble making that play nice with `PruningPredicate` & co, which is where a lot of the performance win is coming from. A secondary concern is that feel like writing an optimizer rule for this will be harder, but maybe that's just because I'm not good at doing it, my first attempt was precisely this and I failed and found this approach much more tractable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
