txwyy123 opened a new issue, #59842:
URL: https://github.com/apache/spark/issues/59842
### Describe the bug
Raising `spark.sql.codegen.maxFields` above 100 to enable whole-stage
codegen for wide schemas causes **SortMergeJoin codegen to fail compilation**
on Spark master:
```
org.codehaus.janino.InternalCompilerException: Compiling
"GeneratedIteratorForCodegenStage1":
CompileException: File 'generated.java', Line XX, Column YY:
Unknown variable or type "smj_bufferedRow_1"
```
The root cause: when generated code exceeds the class-size threshold,
Spark's `addNewFunctionToClass` mechanism moves code to a **nested inner
class**. The inner class cannot access local variables (`smj_bufferedRow_1`,
`smj_value_4`, etc.) declared in the outer method's scope, causing a Janino
compilation failure.
### Reproduction
```scala
import org.apache.spark.sql.functions._
spark.conf.set("spark.sql.codegen.maxFields", "200")
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "-1") // force SMJ
val numCols = 40
val left = (1 to numCols).foldLeft(spark.range(100).toDF("id")) {
(df, i) => df.withColumn(s"left_col_$i", lit(i.toLong))
}
val right = (1 to numCols).foldLeft(spark.range(100).toDF("id")) {
(df, i) => df.withColumn(s"right_col_$i", lit(i.toLong))
}
left.join(right, "id").collect()
// CompileException: Unknown variable or type "smj_bufferedRow_1"
```
### Why this is masked today
The default `spark.sql.codegen.maxFields = 100` disables codegen entirely
for schemas wider than 100 fields, so the code never reaches the point where
`addNewFunctionToClass` splits the inner class. Users who raise `maxFields` (a
documented, public config) to get codegen benefits on wider schemas will hit
this bug.
### Expected behavior
Wide SortMergeJoin queries should either:
1. Successfully compile with whole-stage codegen when `maxFields` is raised,
or
2. Gracefully fall back to the Volcano iterator model instead of throwing a
`CompileException`
### Spark version
master (as of October 2026). Also reproducible on 3.5.x branch.
### Environment
Any environment where `maxFields` is raised above 100 and the join output is
wide enough to trigger class-size splitting.
### Related
- SPARK-33301: codegen expression splitting for CASE WHEN (the prior art for
splitting within whole-stage codegen)
- SPARK-59787: wide join output extraction splitting (addresses the 8K
method-size cliff, not this class-size compilation bug)
- SPARK-60431: wide projection splitting (same family of issues)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
