[
https://issues.apache.org/jira/browse/SPARK-21337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077952#comment-16077952
]
Kazuaki Ishizaki commented on SPARK-21337:
------------------------------------------
In the master branch, I cannot see a huge dump and did not get a failure.
Should we have to backport a fix into 2.1.1 if I am correct?
{code}
test("split complex single column expressions") {
val cases = 50
val conditionClauses = 20
// Generate an individual case
def generateCase(n: Int): (Expression, Expression) = {
val condition = (1 to conditionClauses)
.map(c => EqualTo(BoundReference(0, StringType, false),
Literal(s"$c:$n")))
.reduceLeft[Expression]((l, r) => Or(l, r))
(condition, Literal(n))
}
val expression = CaseWhen((1 to cases).map(generateCase(_)))
// Currently this throws a java.util.concurrent.ExecutionException wrapping
a
// org.codehaus.janino.JaninoRuntimeException: Code of method XXX of class
YYY grows beyond 64 KB
val plan = GenerateMutableProjection.generate(Seq(expression))
}
{code}
> SQL which has large ‘case when’ expressions may cause code generation beyond
> 64KB
> ---------------------------------------------------------------------------------
>
> Key: SPARK-21337
> URL: https://issues.apache.org/jira/browse/SPARK-21337
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.1.1
> Environment: spark-2.1.1-hadoop-2.6.0-cdh-5.4.2
> Reporter: fengchaoge
> Fix For: 2.1.1
>
>
> when there are large 'case when ' expressions in spark sql,the CodeGenerator
> failed to compile it.
> Error message is followed by a huge dump of generated source code,at last
> failed.
> java.util.concurrent.ExecutionException: java.lang.Exception: failed to
> compile: org.codehaus.janino.JaninoRuntimeException: Code of method
> "apply_9$(Lorg/apache/spark/sql/catalyst/expressions/GeneratedClass$SpecificUnsafeProjection;Lorg/apache/spark/sql/catalyst/InternalRow;)V"
> of class
> "org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection"
> grows beyond 64 KB.
> It seems like SPARK-13242 has solved this problem in spark-1.6.1,however it
> apparence in spark-2.1.1 again.
> https://issues.apache.org/jira/browse/SPARK-13242.
> is there something wrong ?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
