Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/19083#discussion_r142000625
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
---
@@ -1074,13 +1078,30 @@ object CodeGenerator extends Logging {
logInfo(s"\n${CodeFormatter.format(code, maxLines)}")
throw new CompileException(msg, e.getLocation)
}
+
+ // Check if compiled code has a too large function
+ methodsToByteCodeSize.foreach { case (name, byteCodeSize) =>
+ if (byteCodeSize > SQLConf.get.hugeMethodLimit) {
+ val clazzName = evaluator.getClazz.getSimpleName
+ val methodName = name.replace("$", "")
+ val msg = s"failed to compile: the size of $clazzName.$methodName
was $byteCodeSize and " +
+ "this value went over the limit
`spark.sql.codegen.hugeMethodLimit`" +
+ s"(${SQLConf.get.hugeMethodLimit}). To avoid this error, you can
make this limit higher."
+ logError(msg)
+ val maxLines = SQLConf.get.loggingMaxLinesForCodegen
+ logInfo(s"\n${CodeFormatter.format(code, maxLines)}")
+ throw new IllegalArgumentException(msg)
--- End diff --
`CompileException` is better?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
