kumarUjjawal commented on issue #18150: URL: https://github.com/apache/datafusion/issues/18150#issuecomment-3763569438
> > [@Jefffrey](https://github.com/Jefffrey) I was thinking of going in the direction of adding an extract_async_exprs() method to the ExecutionPlan trait, which would be similar pattern of `gather_filters_for_pushdown()`, where each operator implements its own async expression extraction. A simple physical optimizer rule (ExtractAsyncFuncs) would then call this method on each node and insert AsyncFuncExec where needed. > > This would be the logic per-operator, will alos fixe the [#18149](https://github.com/apache/datafusion/issues/18149) schema mismatch bug (by properly reconstructing aggregates), and removes async handling complexity from the physical planner. > > Did you give this any thought? I can explore further if you have any feedback on this. > > That does sound like a potential avenue to explore; my only concern is it would be a physical optimizer rule that would become essential. I don't know what the stance is on physical optimizer rules, if they should stick purely to optimizing or if its fine that they act as rewriting into valid plans 🤔 We can follow the precedent from the EnforceDistribution/EnforceSorting and add ExtractAsyncFuncs as an essential optimizer rule, but as you said it will add another essential rule. We can go in other direction and keep the async handling as is with just some refactoring, we can avoid the essential optimizer rules, I don't know how it improves the current situation apart from cleaner codes. If we want broader change we can make the planner create a marker wrapper around inputs with async expressions; then the optimizer will unwraps/transform them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
