Raviston6296 commented on issue #59842: URL: https://github.com/apache/spark/issues/59842#issuecomment-6560849916
I couldn't reproduce this on current master (`4fecd73323f`, which includes SPARK-60403). The snippet in the description compiles with whole-stage codegen for SortMergeJoin at 40 to 400 columns per side with `maxFields` up to 1000, AQE on and off. It also compiles when functions are forced into nested classes: inner, left and right outer joins, with a 2000-branch CASE WHEN under `spark.sql.codegen.wholeStage.splitExpressions.methodLimit=0`. With #59841 applied on top of master, the nested-class cases fail with `Code grows beyond 64 KB` in `processNext()`. Could you share the exact commit and generated code where you saw `Unknown variable or type "smj_bufferedRow_1"`? It may come from the changes in #59841 rather than master. Also note that `spark.sql.codegen.maxFields` is an internal config, and with the default `spark.sql.codegen.fallback=true` a compile failure falls back to non-codegen execution rather than failing the query. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
