hailong wang created FLINK-19363:
------------------------------------
Summary: Code of split method grows beyond 64 KB
Key: FLINK-19363
URL: https://issues.apache.org/jira/browse/FLINK-19363
Project: Flink
Issue Type: Improvement
Components: Table SQL / Planner
Affects Versions: 1.11.0
Reporter: hailong wang
Fix For: 1.12.0
For now, If the total size beyond 64kb, we will split one field into one field.
But when sql is complicated, one field size will grow 64kb.
{code:java}
Caused by: org.codehaus.janino.InternalCompilerException: Compiling
"StreamExecCalc$4436": Code of method "split$4435$(LStreamExecCalc$4436;)V" of
class "StreamExecCalc$4436" grows beyond 64 KBCaused by:
org.codehaus.janino.InternalCompilerException: Compiling "StreamExecCalc$4436":
Code of method "split$4435$(LStreamExecCalc$4436;)V" of class
"StreamExecCalc$4436" grows beyond 64 KB at
org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:382) at
org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:237) at
org.codehaus.janino.SimpleCompiler.compileToClassLoader(SimpleCompiler.java:465)
at org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:216) at
org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:207) at
org.codehaus.commons.compiler.Cookable.cook(Cookable.java:80) at
org.codehaus.commons.compiler.Cookable.cook(Cookable.java:75) at
org.apache.flink.table.runtime.generated.CompileUtils.doCompile(CompileUtils.java:78)
... 24 moreCaused by: org.codehaus.janino.InternalCompilerException: Code of
method "split$4435$(LStreamExecCalc$4436;)V" of class "StreamExecCalc$4436"
grows beyond 64 KB at
org.codehaus.janino.CodeContext.makeSpace(CodeContext.java:1009) at
org.codehaus.janino.CodeContext.write(CodeContext.java:918) at
org.codehaus.janino.CodeContext.writeBranch(CodeContext.java:1038)
{code}
By this case, we should split one field to multi functions.
My solution is as follow,
{code:java}
override def visitCall(call: RexCall): GeneratedExpression = {
val resultType = FlinkTypeFactory.toLogicalType(call.getType)
// convert operands and help giving untyped NULL literals a type
val operands = call.getOperands.zipWithIndex.map {
// this helps e.g. for AS(null)
// we might need to extend this logic in case some rules do not create
typed NULLs
case (operandLiteral: RexLiteral, 0) if
operandLiteral.getType.getSqlTypeName == SqlTypeName.NULL &&
call.getOperator.getReturnTypeInference == ReturnTypes.ARG0 =>
generateNullLiteral(resultType, ctx.nullCheck)
case (o@_, _) => o.accept(this)
}
// when function is too large, we split it.
val exp = generateCallExpression(ctx, call.getOperator, operands, resultType)
if (exp.code.length > maxGeneratedCodeLength) {
ctx.setCodeSplit()
val methodName = newName("callSplit")
val resultTypeTerm = boxedTypeTermForType(exp.resultType)
val callResultTerm = ctx.addReusableLocalVariable(resultTypeTerm,
"callSplitResultTerm")
val callNullResultTerm = ctx.addReusableLocalVariable("boolean",
"callSplitNullResultTerm")
val method =
s"""
|private void $methodName() throws Exception {
| ${exp.code}
| ${callResultTerm}= ${exp.resultTerm};
| ${callNullResultTerm}= ${exp.nullTerm};
|}
|""".stripMargin
val methodCode =
s"""
|$methodName();
|""".stripMargin
ctx.addReusableMember(method)
return new GeneratedExpression(callResultTerm, callNullResultTerm,
methodCode, exp.resultType)
}
exp
}
{code}
When function beyong maxGeneratedCodeLength, we split it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
