jayshrivastava commented on issue #23814: URL: https://github.com/apache/datafusion/issues/23814#issuecomment-5069532595
> I think a "consumer" is a consumer simply by nature of evaluating an expression tree that happens to have [DynamicFilterPhysicalExpr](https://github.com/apache/datafusion/blob/c1180022e63f48a3af175d8d73dc21cca9379c65/datafusion/physical-expr/src/expressions/dynamic_filters/mod.rs#L69) somewhere. > 1. A generic API to walk and modify expressions on ExecutionPlan similar to LogicalPlan::map_expressions and LogicalPlan::apply_expressions (basically what @LiaCastaneda added in https://github.com/apache/datafusion/pull/20337 ) > 2. Add an API to ExecutionPlan to return what dynamic filters it produced (basically promote existing methods like [SortExec::dynamic_filters](https://github.com/apache/datafusion/blob/c1180022e63f48a3af175d8d73dc21cca9379c65/datafusion/physical-plan/src/sorts/sort.rs#L1102) and [HashJoinExec::dynamic_filters](https://github.com/apache/datafusion/blob/c1180022e63f48a3af175d8d73dc21cca9379c65/datafusion/physical-plan/src/joins/hash_join/exec.rs#L957) to the ExecutionPlan trait) This all sounds right to me. Re. `apply_expressions`, most of the reasons for the [revert PR](https://github.com/apache/datafusion/pull/22437) are still true today - the method isn't called from vanilla datafusion - it cannot not have a default implementation, so every ExecutionPlan is forced to implement it - it's tricky to implement because it involves tree node recursion and the behavior of the method can change often (ex. different expressions will be traversed depending on if its called before or after the filter pushdown optimizer pass) The main difference today is we have a concrete use-case. Is this enough to add `apply_expressions` back? I think we would just need to revert the revert PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
