sunchao commented on code in PR #34558:
URL: https://github.com/apache/spark/pull/34558#discussion_r3299655506
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala:
##########
@@ -354,6 +447,49 @@ case class ArrayTransform(
result
}
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ ctx.withLambdaVars(Seq(elementVar) ++ indexVar, varCodes => {
+ val elementCode = varCodes.head
+ val indexCode = varCodes.tail.headOption
+
+ nullSafeCodeGen(ctx, ev, arg => {
+ val numElements = ctx.freshName("numElements")
+ val arrayData = ctx.freshName("arrayData")
+ val i = ctx.freshName("i")
+
+ val initialization = CodeGenerator.createArrayData(
+ arrayData, dataType.elementType, numElements, s" $prettyName
failed.")
+
+ val functionCode = function.genCode(ctx)
Review Comment:
[P2] Rebind fallback lambda variables before code generation
The generated path uses the original lambda tree here, while interpreted
evaluation deliberately uses `functionsForEval` to replace separately
instantiated `NamedLambdaVariable`s with the bound argument instance by
`exprId`. As a result, a valid resolved lambda containing a `CodegenFallback`
expression can succeed interpreted but fail under codegen: I reproduced
`ArrayTransform(array(1, 2, 3), LambdaFunction(CodegenFallbackExpr(detachedArg
+ 1), Seq(arg)))`, where `detachedArg` has the same `exprId` as `arg` but a
different `AtomicReference`; `checkEvaluation` passes interpreted evaluation
and then throws `NullPointerException` in
`GeneratedClass$SpecificMutableProjection.apply`. Please generate the rebound
function tree (and keep fallback atomic state synchronized) so codegen
preserves the existing lambda-binding semantics.
_[ :robot: posted by Codex on behalf of sunchao using the code-review-for-me
skill :robot: ]_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
