adriangb commented on issue #23814: URL: https://github.com/apache/datafusion/issues/23814#issuecomment-5050800628
> Breaking change. It must be implemented by every ExecutionPlan. We cannot default to vec![] because ExecutionPlan nodes may fail to declare their dynamic filters. I think what made the previous implementation tricky was that it required declaring all filter expressions and the API was so generic that a reasonable consumer could rely on it in ways that would break if it was not implemented properly. This more narrow version is at least safer if not safe in that sense: dynamic filters are generally optional performance improvements. Their absence or mishandling doesn't result in incorrect results or errors, just in performance degradation. So maybe defaulting to `vec![]` is a viable option. For `DynamicFilterNodeBehavior` and the `Producer`/`Consumer` split: is it necessary to know the relationship between nodes? I.e. if I have a `SortExec` and a `HashJoinExec` they will both declare themselves as producers, then there will be 1 or maybe 2 `DataSourceExec` that are consumers. But if the plan is being split up and distributed amongst nodes I imagine it would be necessary or at last helpful to know which producers correspond to which consumers? If not you'd have to broadcast updates from all producers to all consumers. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
