Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/19440#discussion_r143086096
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegenExec.scala
---
@@ -392,12 +392,16 @@ case class WholeStageCodegenExec(child: SparkPlan)
extends UnaryExecNode with Co
// Check if compiled code has a too large function
if (maxCodeSize > sqlContext.conf.hugeMethodLimit) {
- logWarning(s"Found too long generated codes and JIT optimization
might not work: " +
- s"the bytecode size was $maxCodeSize, this value went over the
limit " +
+ logInfo(s"Found too long generated codes and JIT optimization might
not work: " +
+ s"the bytecode size ($maxCodeSize) is above the limit " +
s"${sqlContext.conf.hugeMethodLimit}, and the whole-stage codegen
was disabled " +
s"for this plan. To avoid this, you can raise the limit " +
- s"${SQLConf.WHOLESTAGE_HUGE_METHOD_LIMIT.key}:\n$treeString")
- return child.execute()
+ s"`${SQLConf.WHOLESTAGE_HUGE_METHOD_LIMIT.key}`:\n$treeString")
+ child match {
+ // For batch file source scan, we should continue executing it
+ case f: FileSourceScanExec if f.supportsBatch => // do nothing
--- End diff --
I feel a little weird `WholeStageCodegenExec` has specific error handling
for each spark plan. Could we handle this error inside `FileSourceScanExec`?
For example, how about checking if `parent.isInstanceOf[WholeStageCodegenExec]`
in `FileSourceScanExec`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
