gstvg commented on PR #22097: URL: https://github.com/apache/datafusion/pull/22097#issuecomment-4415881744
> Is there cases this can happen with custom user expressions or operators that would fail with lambda? > > If that is the case I think we should try to see how we can make lambda expressions more resilient and work out of the box like every other expression Lambdas should work with any expression that abide to the `PhysicalExpr` contract, which is not the case for async functions. And it also should be compatible with any operator as it does abide to the `PhysicalExpr` contract. IMHO Is not that lambdas causes async funcs or `AsyncFuncExec` to fail, but the rather unique async function requirements, that during physical planning moves lambda variables out of it's lambda (first error case) and creates a schema that contains more columns than the one used during other parts of physical planning (second error case, also the cause of #18149 I believe) Anything that moves a lambda variable out of it's lambda make it impossible to evaluate it and I can't see nothing that the lambda implementation can do to avoid that. We could hide the lamba body from the normal tree node traversals (and add new _with_lambda methods to the tree node trait) so that the async physical planner would not move it, but then the async function `PhysicalExpr::evaluate` would be called and would return an error. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
