jayshrivastava commented on PR #24601: URL: https://github.com/apache/datafusion/pull/24601#issuecomment-5397343882
> If a new rule wants to add a new consumer in a way that doesn't just transfer/copy the filters from the existing consumer then yes they'd need to run the filter pushdown optimizer again. I think that's a reasonable thing to do. But this is also the case today. > If the rule wants to replace the scan with a new operator and is happy to copy the filters (as @jayshrivastava / datafusion-distributed want, or at least I understand they want) then there is no need to re-run filter pushdown. The tests you added cover what I want to do in datafusion-distributed. Imo, it makes sense that all filter-related decisions are finalized during the `FilterPushdown` rule and not revisited after. One can implement their own filter pushdown rule that runs after if they want, using `handle_child_pushdown_result` and `gather_filters_for_pushdown`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
