[ https://issues.apache.org/jira/browse/CALCITE-2792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16946209#comment-16946209 ]
Julian Hyde commented on CALCITE-2792: -------------------------------------- The balanced tree is a clever idea but it seems to be masking the problem. The real solution is to not create huge expressions. So, a fix to CALCITE-2696 would be good. Also, I think it would be useful to create a "$HARD_IN" function that has the same arguments as IN but is not expanded to OR. It can pass through the planning process unchanged. Users wouldn't reference it in their SQL, but SqlToRelConverter could introduce it based on a parameter setting. Because it has a large argument list it would be "flat" and not susceptible to StackOverflowError. People could add some conservative optimizations. > Stackoverflow while evaluating filter with large number of OR conditions > ------------------------------------------------------------------------ > > Key: CALCITE-2792 > URL: https://issues.apache.org/jira/browse/CALCITE-2792 > Project: Calcite > Issue Type: Bug > Components: core > Affects Versions: 1.18.0 > Reporter: Dirk Mahler > Priority: Major > Labels: pull-request-available > Attachments: calcite-stackoverflow.zip > > Time Spent: 0.5h > Remaining Estimate: 0h > > As a workaround for CALCITE-2696 we're currently using OR conditions for > filtering values, e.g. instead of > {noformat} > ... WHERE value2 IN (1,2,3) > {noformat} > {noformat} > ... WHERE value2=1 OR value2=2 OR value2=3 > {noformat} > We're now hitting a StackOverflowError because the number of values in the > filter grows quite large (i.e. 1000-3000) and obviously the evaluation > recursive: > {noformat} > java.lang.StackOverflowError > at java.util.AbstractCollection.toArray(AbstractCollection.java:176) > at > org.apache.calcite.sql.util.SqlShuttle$CallCopyingArgHandler.<init>(SqlShuttle.java:111) > at > org.apache.calcite.sql.validate.SqlValidatorImpl$Expander.visitScoped(SqlValidatorImpl.java:5699) > at > org.apache.calcite.sql.validate.SqlScopedShuttle.visit(SqlScopedShuttle.java:50) > at > org.apache.calcite.sql.validate.SqlScopedShuttle.visit(SqlScopedShuttle.java:33) > at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:138) > at > org.apache.calcite.sql.util.SqlShuttle$CallCopyingArgHandler.visitChild(SqlShuttle.java:134) > at > org.apache.calcite.sql.util.SqlShuttle$CallCopyingArgHandler.visitChild(SqlShuttle.java:101) > at org.apache.calcite.sql.SqlOperator.acceptCall(SqlOperator.java:865) > at > org.apache.calcite.sql.validate.SqlValidatorImpl$Expander.visitScoped(SqlValidatorImpl.java:5701) > at > org.apache.calcite.sql.validate.SqlScopedShuttle.visit(SqlScopedShuttle.java:50) > at > org.apache.calcite.sql.validate.SqlScopedShuttle.visit(SqlScopedShuttle.java:33) > at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:138) > at > org.apache.calcite.sql.util.SqlShuttle$CallCopyingArgHandler.visitChild(SqlShuttle.java:134) > at > org.apache.calcite.sql.util.SqlShuttle$CallCopyingArgHandler.visitChild(SqlShuttle.java:101) > at org.apache.calcite.sql.SqlOperator.acceptCall(SqlOperator.java:865) > at > org.apache.calcite.sql.validate.SqlValidatorImpl$Expander.visitScoped(SqlValidatorImpl.java:5701) > at > org.apache.calcite.sql.validate.SqlScopedShuttle.visit(SqlScopedShuttle.java:50) > at > org.apache.calcite.sql.validate.SqlScopedShuttle.visit(SqlScopedShuttle.java:33) > at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:138) > ... > {noformat} > We tried to increase the stack size of the virtual machine (-Xss8000k) but > this leads to a problem with Janino: > {noformat} > java.sql.SQLException: exception while executing query: Compiling "Buzz": > Code of method > "execute(Lorg/apache/calcite/interpreter/Context;[Ljava/lang/Object;)V" of > class "Buzz" grows beyond 64 KB > at org.apache.calcite.avatica.Helper.createException(Helper.java:56) > at org.apache.calcite.avatica.Helper.createException(Helper.java:41) > at > org.apache.calcite.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:577) > at > org.apache.calcite.avatica.AvaticaPreparedStatement.executeQuery(AvaticaPreparedStatement.java:137) > at > org.apache.calcite.issue.StackOverflowTest.stackOverflow(StackOverflowTest.java:45) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > at org.junit.runner.JUnitCore.run(JUnitCore.java:137) > at > com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68) > at > com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47) > at > com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242) > at > com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70) > Caused by: org.codehaus.janino.InternalCompilerException: Compiling "Buzz": > Code of method > "execute(Lorg/apache/calcite/interpreter/Context;[Ljava/lang/Object;)V" of > class "Buzz" grows beyond 64 KB > at org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:382) > at org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:237) > at > org.codehaus.janino.SimpleCompiler.compileToClassLoader(SimpleCompiler.java:465) > at > org.codehaus.janino.ClassBodyEvaluator.compileToClass(ClassBodyEvaluator.java:313) > at > org.codehaus.janino.ClassBodyEvaluator.cook(ClassBodyEvaluator.java:235) > at org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:207) > at org.codehaus.commons.compiler.Cookable.cook(Cookable.java:50) > at > org.codehaus.janino.ClassBodyEvaluator.createInstance(ClassBodyEvaluator.java:347) > at > org.apache.calcite.interpreter.JaninoRexCompiler.getScalar(JaninoRexCompiler.java:176) > at > org.apache.calcite.interpreter.JaninoRexCompiler.baz(JaninoRexCompiler.java:153) > at > org.apache.calcite.interpreter.JaninoRexCompiler.compile(JaninoRexCompiler.java:111) > at > org.apache.calcite.interpreter.Interpreter$CompilerImpl.compile(Interpreter.java:487) > at > org.apache.calcite.interpreter.Nodes$CoreCompiler.compile(Nodes.java:42) > at > org.apache.calcite.interpreter.TableScanNode.createEnumerable(TableScanNode.java:266) > at > org.apache.calcite.interpreter.TableScanNode.createProjectableFilterable(TableScanNode.java:233) > at > org.apache.calcite.interpreter.TableScanNode.create(TableScanNode.java:81) > at > org.apache.calcite.interpreter.Nodes$CoreCompiler.visit(Nodes.java:69) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.calcite.util.ReflectUtil.invokeVisitorInternal(ReflectUtil.java:257) > at > org.apache.calcite.util.ReflectUtil.invokeVisitor(ReflectUtil.java:214) > at > org.apache.calcite.util.ReflectUtil$1.invokeVisitor(ReflectUtil.java:464) > at > org.apache.calcite.interpreter.Interpreter$CompilerImpl.visit(Interpreter.java:451) > at > org.apache.calcite.interpreter.Nodes$CoreCompiler.visit(Nodes.java:42) > at > org.apache.calcite.interpreter.Interpreter$CompilerImpl.visitRoot(Interpreter.java:405) > at > org.apache.calcite.interpreter.Interpreter.<init>(Interpreter.java:88) > at Baz.bind(Unknown Source) > at > org.apache.calcite.jdbc.CalcitePrepare$CalciteSignature.enumerable(CalcitePrepare.java:355) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.enumerable(CalciteConnectionImpl.java:309) > at > org.apache.calcite.jdbc.CalciteMetaImpl._createIterable(CalciteMetaImpl.java:506) > at > org.apache.calcite.jdbc.CalciteMetaImpl.createIterable(CalciteMetaImpl.java:497) > at > org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:182) > at > org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:64) > at > org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:43) > at > org.apache.calcite.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:573) > ... 26 more > Caused by: org.codehaus.janino.InternalCompilerException: Code of method > "execute(Lorg/apache/calcite/interpreter/Context;[Ljava/lang/Object;)V" of > class "Buzz" grows beyond 64 KB > at org.codehaus.janino.CodeContext.makeSpace(CodeContext.java:1048) > at org.codehaus.janino.CodeContext.write(CodeContext.java:925) > at org.codehaus.janino.UnitCompiler.writeByte(UnitCompiler.java:12275) > at org.codehaus.janino.UnitCompiler.load(UnitCompiler.java:11936) > at org.codehaus.janino.UnitCompiler.load(UnitCompiler.java:11926) > at org.codehaus.janino.UnitCompiler.compileGet2(UnitCompiler.java:4465) > at org.codehaus.janino.UnitCompiler.access$8000(UnitCompiler.java:215) > at > org.codehaus.janino.UnitCompiler$16$1.visitLocalVariableAccess(UnitCompiler.java:4408) > at > org.codehaus.janino.UnitCompiler$16$1.visitLocalVariableAccess(UnitCompiler.java:4400) > at org.codehaus.janino.Java$LocalVariableAccess.accept(Java.java:4274) > at > org.codehaus.janino.UnitCompiler$16.visitLvalue(UnitCompiler.java:4400) > at > org.codehaus.janino.UnitCompiler$16.visitLvalue(UnitCompiler.java:4396) > at org.codehaus.janino.Java$Lvalue.accept(Java.java:4148) > at org.codehaus.janino.UnitCompiler.compileGet(UnitCompiler.java:4396) > at org.codehaus.janino.UnitCompiler.compileGet2(UnitCompiler.java:4461) > at org.codehaus.janino.UnitCompiler.access$7500(UnitCompiler.java:215) > at > org.codehaus.janino.UnitCompiler$16$1.visitAmbiguousName(UnitCompiler.java:4403) > at > org.codehaus.janino.UnitCompiler$16$1.visitAmbiguousName(UnitCompiler.java:4400) > at org.codehaus.janino.Java$AmbiguousName.accept(Java.java:4224) > at > org.codehaus.janino.UnitCompiler$16.visitLvalue(UnitCompiler.java:4400) > at > org.codehaus.janino.UnitCompiler$16.visitLvalue(UnitCompiler.java:4396) > at org.codehaus.janino.Java$Lvalue.accept(Java.java:4148) > at org.codehaus.janino.UnitCompiler.compileGet(UnitCompiler.java:4396) > at > org.codehaus.janino.UnitCompiler.compileGetValue(UnitCompiler.java:5662) > at > org.codehaus.janino.UnitCompiler.compileBoolean2(UnitCompiler.java:4120) > at org.codehaus.janino.UnitCompiler.access$6600(UnitCompiler.java:215) > at > org.codehaus.janino.UnitCompiler$14.visitBinaryOperation(UnitCompiler.java:3957) > at > org.codehaus.janino.UnitCompiler$14.visitBinaryOperation(UnitCompiler.java:3935) > at org.codehaus.janino.Java$BinaryOperation.accept(Java.java:4864) > at > org.codehaus.janino.UnitCompiler.compileBoolean(UnitCompiler.java:3935) > at > org.codehaus.janino.UnitCompiler.compileBoolean2(UnitCompiler.java:4078) > at org.codehaus.janino.UnitCompiler.access$6600(UnitCompiler.java:215) > at > org.codehaus.janino.UnitCompiler$14.visitBinaryOperation(UnitCompiler.java:3957) > at > org.codehaus.janino.UnitCompiler$14.visitBinaryOperation(UnitCompiler.java:3935) > at org.codehaus.janino.Java$BinaryOperation.accept(Java.java:4864) > {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005)