adriangb commented on PR #15770: URL: https://github.com/apache/datafusion/pull/15770#issuecomment-2971441638
> We might be able to split the filter pushdown into two steps: static (cannot make assumptions about reference links but can modify the plan tree, e.g. for FitlerExec) and dynamic (can make reference links but cannot modify the plan tree, e.g. SortExec and HashJoin). @alamb I had not thought of this before but it does seem interesting. we'd basically add a new (simpler) rule called `PushDownDynamicFilters` or something like that. we'd also have to add a new method to `ExecutionPlan::gather_dynamic_filters_for_pushdown` and `SortExec` / `HashJoinExec` / any operator that wants to push down a filter that depends on the plan's structure / has references back to itself uses this method instead of the existing pushdown methods. then this can be run after `EnforceSorting`. I think the main con to this would be if an operator wanted to push down a dyanmic filter _and_ modify the plan structure. I don't think it will want to remove itself from the plan structure like `FilterExec` does, but I could see it wanting to add a node (e.g. if we implemented https://github.com/apache/datafusion/pull/15697 by having `SortExec` insert a `FilterExec` underneath itself). the big pro here is that I think it makes a clear distinction of capabilities and resolves th e fragility of rewriting a plan while keeping track of references between nodes. what do you think of that approach vs. the approach we have here of teaching `EnforceSorting` how to pass around dynamic filters? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
