Kimahriman commented on code in PR #34558:
URL: https://github.com/apache/spark/pull/34558#discussion_r3303960462
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala:
##########
@@ -354,6 +447,49 @@ case class ArrayTransform(
result
}
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ ctx.withLambdaVars(Seq(elementVar) ++ indexVar, varCodes => {
+ val elementCode = varCodes.head
+ val indexCode = varCodes.tail.headOption
+
+ nullSafeCodeGen(ctx, ev, arg => {
+ val numElements = ctx.freshName("numElements")
+ val arrayData = ctx.freshName("arrayData")
+ val i = ctx.freshName("i")
+
+ val initialization = CodeGenerator.createArrayData(
+ arrayData, dataType.elementType, numElements, s" $prettyName
failed.")
+
+ val functionCode = function.genCode(ctx)
Review Comment:
Fixed in `fe004a5435e`.
The generated HOF paths now mirror interpreted evaluation by generating from
the rebound lambda body (`functionForEval` / `functionsForEval`) instead of the
original lambda tree. This preserves the existing exprId-based lambda variable
rebinding semantics while keeping the change scoped to HOF codegen.
I added a Catalyst regression that constructs a resolved `ArrayTransform`
lambda where a `CodegenFallbackExpr` references a detached
`NamedLambdaVariable` with the same `exprId` as the bound argument. The test
failed before the fix with the generated-path NPE described in this comment,
and passes after the fix.
Verified with:
```bash
build/sbt 'catalyst/testOnly
org.apache.spark.sql.catalyst.expressions.HigherOrderFunctionsSuite -- -z
"ArrayTransform codegen rebinds fallback lambda variables by expression ID"'
build/sbt 'catalyst/testOnly
org.apache.spark.sql.catalyst.expressions.HigherOrderFunctionsSuite -- -z
"ArrayTransform"'
```
Results: focused regression passed (`1` test), ArrayTransform slice passed
(`3` tests).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
