Kimahriman commented on PR #40473: URL: https://github.com/apache/spark/pull/40473#issuecomment-1474848224
Just seeing this group of PRs (most notably https://github.com/apache/spark/pull/39046), was there a real reason `NamedLambdaVariable` was added into all this mix? If I understand right, it effectively eliminates all subexpression elimination involving higher-order functions, even though it's perfectly valid to pull out a complete high-order function, you just can't pull out the `LambdaFunction` by itself. Currently the check for `CodegenFallback` is what prevents the `LambdaFunction`'s from being considered for subexpression elimination. Quick plug for my 1.5 year old PR for adding codegen to HOCs https://github.com/apache/spark/pull/34558 simply adds `HigherOrderFunction` as a special case to only consider the arguments and not the functions themselves for subexpression elimination -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
