dongjoon-hyun opened a new pull request, #59968: URL: https://github.com/apache/spark/pull/59968
### What changes were proposed in this pull request? This PR aims to use constant names when the codegen of higher-order functions registers lambda variables via `CodegenContext.addReferenceObj`, instead of the lambda variable names of the query. | Lambda variable | Before | After | |---|---|---| | Element | `elementVar.name` | `"lambdaElement"` | | Index | `indexVar.name` | `"lambdaIndex"` | | Accumulator for merge | `accForMergeVar.name` | `"lambdaAccForMerge"` | | Accumulator for finish | `accForFinishVar.name` | `"lambdaAccForFinish"` | In addition, this PR documents in the Scaladoc of `addReferenceObj` that `objName` must be a constant. ### Why are the changes needed? `addReferenceObj` writes `objName` into a comment of the generated Java source, like `((Class) references[0] /* objName */)`. Since SPARK-37019 added the codegen of array higher-order functions, this comment has held the lambda variable names of the query. This causes two issues. 1. A valid lambda variable name can break the generated source. For example, the generated class fails to compile when the name contains `*/`. Spark falls back and returns correct results, but the plan loses whole-stage codegen and errors are logged for each query. 2. The generated source depends on the name, so a compiled class cannot be reused across names. The DataFrame API creates a fresh name such as `x_1` from a global counter on each call, so running the same `transform` query again compiles its 2 classes again. ### Does this PR introduce _any_ user-facing change? Yes, this is a bug fix for Apache Spark [4.3.0](https://github.com/apache/spark/releases/tag/v4.3.0) (2026-10-27). - Queries with such lambda variable names keep whole-stage codegen and no longer log compilation errors. Query results do not change. - `EXPLAIN CODEGEN` shows the constant names instead of the lambda variable names in the comments of the generated source. ### How was this patch tested? Pass the CIs with the newly added test cases. They fail without the main code change. - `HigherOrderFunctionsSuite` - `SPARK-60760: The generated code does not hold the names of lambda variables` - `WholeStageCodegenSuite` - `SPARK-60760: Lambda variable names are not in the generated code` - `SPARK-60760: A new lambda variable name does not compile a generated class again` ``` build/sbt 'catalyst/testOnly *HigherOrderFunctionsSuite -- -z "SPARK-60760"' build/sbt 'sql/testOnly org.apache.spark.sql.execution.WholeStageCodegenSuite -- -z "SPARK-60760"' ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
