[
https://issues.apache.org/jira/browse/CALCITE-3094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17140508#comment-17140508
]
Thomas Rebele commented on CALCITE-3094:
----------------------------------------
I would not apply a transformation to the generated code, but generate
different code in the first place. During code generation the length of the
left and right array are known, so it would be easy to generate the according
System.arraycopy(...) calls. I see three points in favor of my proposal:
* It is easy to implement (though I haven't found the method that generates
the method for concatenating arrays yet).
* It fixes the cause of the exception for this particular query. Maybe we can
find other fixes when other generated methods hit the 64kB limit.
* It is more performant for inputs with many fields. I did a JMH benchmark.
For |left| = |right| >= 500 fields it shows a speedup of >20x with
System.arraycopy(...). For 100 to 50 fields the speedup is still 3-4x. The
break even point is somewhere between 10 and 50 fields.
*Benchmark results:* (source code for benchmark [^CTE-3094.zip])
With 500ms per iteration (should be redone with a higher time limit to get more
reliable results):
{code:java}
Benchmark (fields) Mode Cnt Score Error Units
MyBenchmark.arraycopy 1 thrpt 25 36125106.608 ± 1697823.670 ops/s
MyBenchmark.arraycopy 5 thrpt 25 35451604.912 ± 329986.258 ops/s
MyBenchmark.arraycopy 10 thrpt 25 35102765.686 ± 428249.309 ops/s
MyBenchmark.arraycopy 50 thrpt 25 24356938.619 ± 260712.335 ops/s
MyBenchmark.arraycopy 100 thrpt 25 13577289.534 ± 51174.663 ops/s
MyBenchmark.newArray 1 thrpt 25 102568455.693 ± 1548429.031 ops/s
MyBenchmark.newArray 5 thrpt 25 66322353.185 ± 3822598.128 ops/s
MyBenchmark.newArray 10 thrpt 25 37661074.279 ± 268101.214 ops/s
MyBenchmark.newArray 50 thrpt 25 7955575.721 ± 81694.888 ops/s
MyBenchmark.newArray 100 thrpt 25 2964480.660 ± 123294.506 ops/s
{code}
With 10s per iteration (default settings):
{code:java}
Benchmark (fields) Mode Cnt Score Error Units
MyBenchmark.arraycopy 3001 thrpt 25 535075.323 ± 984.691 ops/s
MyBenchmark.newArray 3001 thrpt 25 23586.443 ± 189.309 ops/s
Benchmark (fields) Mode Cnt Score Error Units
MyBenchmark.arraycopy 500 thrpt 25 2834626.364 ± 11407.411 ops/s
MyBenchmark.arraycopy 1000 thrpt 25 1650213.571 ± 2468.686 ops/s
MyBenchmark.newArray 500 thrpt 25 138037.870 ± 791.348 ops/s
MyBenchmark.newArray 1000 thrpt 25 70199.080 ± 404.447 ops/s
{code}
> Code of method grows beyond 64 KB
> ---------------------------------
>
> Key: CALCITE-3094
> URL: https://issues.apache.org/jira/browse/CALCITE-3094
> Project: Calcite
> Issue Type: Bug
> Components: core, jdbc-adapter
> Affects Versions: 1.19.0
> Reporter: Pablo
> Priority: Critical
> Attachments: CTE-3094.zip
>
>
> I am running simple queries like these:
> {{SELECT * FROM t0 INNER JOIN t1 ON t0.`f000000` = t1.`f000003`"}}
>
> They are running ok, but, when t1 and t0 have about 3290 fields, I get this
> error.
> Is there anything I can do? I can't do joins with this amount of fields?
>
> Thank you in advance
>
> {noformat}
> Caused by: java.lang.RuntimeException: Error while compiling generated Java
> code:
> public org.apache.calcite.linq4j.Enumerable bind(final
> org.apache.calcite.DataContext root) {
> final org.apache.calcite.rel.RelNode v2stashed =
> (org.apache.calcite.rel.RelNode) root.get("v2stashed");
> final org.apache.calcite.rel.RelNode v1stashed =
> (org.apache.calcite.rel.RelNode) root.get("v1stashed");
> final org.apache.calcite.interpreter.Interpreter interpreter = new
> org.apache.calcite.interpreter.Interpreter(
> root,
> v1stashed);
> final org.apache.calcite.interpreter.Interpreter interpreter0 = new
> org.apache.calcite.interpreter.Interpreter(
> root,
> v2stashed);
> return interpreter.join(interpreter0, new
> org.apache.calcite.linq4j.function.Function1() {
> public Byte apply(Object[] v1) {
> return (Byte) v1[0];
> }
> public Object apply(Object v1) {
> return apply(
> (Object[]) v1);
> }
> }
> , new org.apache.calcite.linq4j.function.Function1() {
> public Byte apply(Object[] v1) {
> return (Byte) v1[3];
> }
> public Object apply(Object v1) {
> return apply(
> (Object[]) v1);
> }
> }
> , new org.apache.calcite.linq4j.function.Function2() {
> public Object[] apply(Object[] left, Object[] right) {
> return new Object[] {
> left[0],
> left[1],
> left[2],
> [...elided ...]
> left[3298],
> left[3299],
> right[0],
> right[1],
> [...elided...]]
> right[3295],
> right[3296],
> right[3297],
> right[3298],
> right[3299]};
> }
> public Object[] apply(Object left, Object right) {
> return apply(
> (Object[]) left,
> (Object[]) right);
> }
> }
> , null, false, false);
> }
> public Class getElementType() {
> return java.lang.Object[].class;
> }
> at org.apache.calcite.avatica.Helper.wrap(Helper.java:37)
> at
> org.apache.calcite.adapter.enumerable.EnumerableInterpretable.toBindable(EnumerableInterpretable.java:128)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl$CalcitePreparingStmt.implement(CalcitePrepareImpl.java:1233)
> at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:332)
> at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:231)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:767)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:631)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:601)
> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:229)
> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement_(CalciteConnectionImpl.java:211)
> ... 80 more
> Caused by: org.codehaus.janino.InternalCompilerException: Compiling "Baz":
> Code of method
> "apply([Ljava/lang/Object;[Ljava/lang/Object;)[Ljava/lang/Object;" of class
> "Baz$3" grows beyond 64 KB
> at org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:382)
> at org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:237)
> at
> org.codehaus.janino.SimpleCompiler.compileToClassLoader(SimpleCompiler.java:465)
> at
> org.codehaus.janino.ClassBodyEvaluator.compileToClass(ClassBodyEvaluator.java:313)
> at org.codehaus.janino.ClassBodyEvaluator.cook(ClassBodyEvaluator.java:235)
> at org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:207)
> at org.codehaus.commons.compiler.Cookable.cook(Cookable.java:50)
> at
> org.codehaus.janino.ClassBodyEvaluator.createInstance(ClassBodyEvaluator.java:347)
> at
> org.apache.calcite.adapter.enumerable.EnumerableInterpretable.getBindable(EnumerableInterpretable.java:162)
> at
> org.apache.calcite.adapter.enumerable.EnumerableInterpretable.toBindable(EnumerableInterpretable.java:125)
> ... 88 more
> Caused by: org.codehaus.janino.InternalCompilerException: Code of method
> "apply([Ljava/lang/Object;[Ljava/lang/Object;)[Ljava/lang/Object;" of class
> "Baz$3" grows beyond 64 KB
> at org.codehaus.janino.CodeContext.makeSpace(CodeContext.java:1048)
> at org.codehaus.janino.CodeContext.write(CodeContext.java:925)
> at org.codehaus.janino.UnitCompiler.writeOpcode(UnitCompiler.java:12291)
> at org.codehaus.janino.UnitCompiler.pushConstant(UnitCompiler.java:10746)
> at org.codehaus.janino.UnitCompiler.compileGetValue(UnitCompiler.java:5616)
> at org.codehaus.janino.UnitCompiler.compileGet2(UnitCompiler.java:5592)
> at org.codehaus.janino.UnitCompiler.access$9700(UnitCompiler.java:215)
> at
> org.codehaus.janino.UnitCompiler$16.visitNewInitializedArray(UnitCompiler.java:4434)
> at
> org.codehaus.janino.UnitCompiler$16.visitNewInitializedArray(UnitCompiler.java:4396)
> at org.codehaus.janino.Java$NewInitializedArray.accept(Java.java:5373)
> at org.codehaus.janino.UnitCompiler.compileGet(UnitCompiler.java:4396)
> at org.codehaus.janino.UnitCompiler.compileGetValue(UnitCompiler.java:5662)
> at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:2649)
> at org.codehaus.janino.UnitCompiler.access$2800(UnitCompiler.java:215)
> at
> org.codehaus.janino.UnitCompiler$6.visitReturnStatement(UnitCompiler.java:1504)
> at
> org.codehaus.janino.UnitCompiler$6.visitReturnStatement(UnitCompiler.java:1487)
> at org.codehaus.janino.Java$ReturnStatement.accept(Java.java:3563)
> at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1487)
> at org.codehaus.janino.UnitCompiler.compileStatements(UnitCompiler.java:1567)
> at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:3388)
> at
> org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1357)
> at
> org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1330)
> at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:822)
> at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:981)
> at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:951)
> at org.codehaus.janino.UnitCompiler.access$200(UnitCompiler.java:215)
> at
> org.codehaus.janino.UnitCompiler$2.visitAnonymousClassDeclaration(UnitCompiler.java:409)
> at
> org.codehaus.janino.UnitCompiler$2.visitAnonymousClassDeclaration(UnitCompiler.java:406)
> at org.codehaus.janino.Java$AnonymousClassDeclaration.accept(Java.java:1149)
> at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:406)
> at org.codehaus.janino.UnitCompiler.compileGet2(UnitCompiler.java:5509)
> at org.codehaus.janino.UnitCompiler.access$9500(UnitCompiler.java:215)
> at
> org.codehaus.janino.UnitCompiler$16.visitNewAnonymousClassInstance(UnitCompiler.java:4432)
> at
> org.codehaus.janino.UnitCompiler$16.visitNewAnonymousClassInstance(UnitCompiler.java:4396)
> at org.codehaus.janino.Java$NewAnonymousClassInstance.accept(Java.java:5238)
> at org.codehaus.janino.UnitCompiler.compileGet(UnitCompiler.java:4396)
> at org.codehaus.janino.UnitCompiler.compileGetValue(UnitCompiler.java:5662)
> at org.codehaus.janino.UnitCompiler.compileGet2(UnitCompiler.java:5182)
> at org.codehaus.janino.UnitCompiler.access$9100(UnitCompiler.java:215)
> at
> org.codehaus.janino.UnitCompiler$16.visitMethodInvocation(UnitCompiler.java:4423)
> at
> org.codehaus.janino.UnitCompiler$16.visitMethodInvocation(UnitCompiler.java:4396)
> at org.codehaus.janino.Java$MethodInvocation.accept(Java.java:5073)
> at org.codehaus.janino.UnitCompiler.compileGet(UnitCompiler.java:4396)
> at org.codehaus.janino.UnitCompiler.compileGetValue(UnitCompiler.java:5662)
> at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:2649)
> at org.codehaus.janino.UnitCompiler.access$2800(UnitCompiler.java:215)
> at
> org.codehaus.janino.UnitCompiler$6.visitReturnStatement(UnitCompiler.java:1504)
> at
> org.codehaus.janino.UnitCompiler$6.visitReturnStatement(UnitCompiler.java:1487)
> at org.codehaus.janino.Java$ReturnStatement.accept(Java.java:3563)
> at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1487)
> at org.codehaus.janino.UnitCompiler.compileStatements(UnitCompiler.java:1567)
> at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:3388)
> at
> org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1357)
> at
> org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1330)
> at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:822)
> at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:432)
> at org.codehaus.janino.UnitCompiler.access$400(UnitCompiler.java:215)
> at
> org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:411)
> at
> org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:406)
> at
> org.codehaus.janino.Java$PackageMemberClassDeclaration.accept(Java.java:1414)
> at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:406)
> at org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:378)
> ... 97 more
> {noformat}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
