[
https://issues.apache.org/jira/browse/CALCITE-4054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17128998#comment-17128998
]
Ruben Q L edited comment on CALCITE-4054 at 6/9/20, 10:48 AM:
--------------------------------------------------------------
The exception is thrown by the code generated by
{{EnumerableTableSpool#implement}}:
{code:java}
@Override public Result implement(EnumerableRelImplementor implementor,
Prefer pref) {
...
// ModifiableTable t = (ModifiableTable)
root.getRootSchema().getTable(tableName);
// return lazyCollectionSpool(t.getModifiableCollection(), <inputExp>);
BlockBuilder builder = new BlockBuilder();
...
{code}
The call {{root.getRootSchema().getTable(tableName)}} does not find the table
and returns {{null}}, hence the subsequent call to
{{getModifiableCollection()}} throws the NPE.
Normally, the transient table gets added to the root schema when its enumerable
is create, via {{ListTransientTable#scan}}:
{code:java}
@Override public Enumerable<Object[]> scan(DataContext root) {
// add the table into the schema, so that it is accessible by any potential
operator
root.getRootSchema().add(name, this);
...
{code}
Therefore, we should expect the table to be correctly added to the root schema
when we build the query result's enumerable. This is usually the case, since
each operator in Enumerable convention builds and returns its enumerable. But,
there is an exception with EnumerableCorrelate. The right-hand-side of
EnumerableCorrelate does not build an Enumerable, instead it builds a
*function* that, given a certain correlated value, returns the corresponding
enumerable for the RHS evaluation with that value (see the implementation of
EnumerableDefaults#correlateJoin). Therefore, in that situation where the
ListTransientScan is on the RHS of the EnumerableCorrelate, its scan method has
not been yet executed, hence the table has not been added to the schema, and we
reach the NPE when EnumerableTableSpool's dynamic code runs.
was (Author: rubenql):
The exception is thrown by the code generated by
{{EnumerableTableSpool#implement}}:
{code:java}
@Override public Result implement(EnumerableRelImplementor implementor,
Prefer pref) {
...
// ModifiableTable t = (ModifiableTable)
root.getRootSchema().getTable(tableName);
// return lazyCollectionSpool(t.getModifiableCollection(), <inputExp>);
BlockBuilder builder = new BlockBuilder();
...
{code}
The call {{root.getRootSchema().getTable(tableName)}} does not find the table
and returns {{null}}, hence the subsequent call to
{{getModifiableCollection()}} throws the NPE.
Normally, the transient table gets added to the root schema when its enumerable
is create, via {{ListTransientTable#scan}}:
{code:java}
@Override public Enumerable<Object[]> scan(DataContext root) {
// add the table into the schema, so that it is accessible by any potential
operator
root.getRootSchema().add(name, this);
...
{code}
Therefore, we should expect the table to be correctly added to the root schema
when we build the query result's enumerable. This is usually the case, since
each operator in Enumerable convention builds and returns its enumerable. But,
there is an exception with EnumerableCorrelate. The right-hand-side of
EnumerableCorrelate does not build an Enumerable, instead it builds a
*function* that, given a certain correlated value, returns the corresponding
enumerable for the RHS evaluation with that value (see the implementation of
EnumerableDefaults#correlateJoin). Therefore, in that situation where the
ListTransientScan is on the RHS of the EnumerableCorrelate, its scan method has
not been yet executed, hence the table has not been added to the schema, and we
reach the NPE when EnumerableTableSpool build its dynamic code.
> RepeatUnion containing a Correlate with a transientScan on its RHS causes NPE
> -----------------------------------------------------------------------------
>
> Key: CALCITE-4054
> URL: https://issues.apache.org/jira/browse/CALCITE-4054
> Project: Calcite
> Issue Type: Bug
> Components: core
> Affects Versions: 1.23.0
> Reporter: Ruben Q L
> Priority: Minor
>
> This is not a very urgent problem, since this is an exception on an operator
> tagged as "Experimental", and several "not-so-usual" circumstances must occur
> in order to reach the NPE:
> - We need a "recursive plan", i.e. a RepeatUnion with a TransientScan (see
> examples in EnumerableRepeatUnionTest.java).
> - Inside the repeat union, there must be a Correlate, or a Join that gets
> implemented as a Correlate due to JoinToCorrelateRule.
> - We have the TransientScan (i.e. the scan on the transient table) on the
> right-hand-side of the correlate/join. Normally, transientScan always appear
> on the LHS of a correlate/join (because when the plan is built they need the
> top of the RelBuilder's stack to get its rowType); however, it can get on the
> RHS due to JoinCommuteRule.
> We can force these conditions and reproduce the issue with the following unit
> test (in EnumerableRepeatUnionTest.java):
> {code:java}
> @Test void testRepeatUnionWithCorrelateWithTransientScanOnItsRight() {
> CalciteAssert.that()
> .with(CalciteConnectionProperty.LEX, Lex.JAVA)
> .with(CalciteConnectionProperty.FORCE_DECORRELATE, false)
> .withSchema("s", new ReflectiveSchema(new HierarchySchema()))
> .query("?")
> .withHook(Hook.PLANNER, (Consumer<RelOptPlanner>) planner -> {
> planner.addRule(JoinToCorrelateRule.INSTANCE);
> planner.removeRule(JoinCommuteRule.INSTANCE);
> planner.removeRule(EnumerableRules.ENUMERABLE_MERGE_JOIN_RULE);
> planner.removeRule(EnumerableRules.ENUMERABLE_JOIN_RULE);
> })
> .withRel(builder -> {
> builder
> // WITH RECURSIVE delta(empid, name) as (
> // SELECT empid, name FROM emps WHERE empid = 2
> // UNION ALL
> // SELECT e.empid, e.name FROM delta d
> // JOIN hierarchies h ON d.empid =
> h.managerid
> // JOIN emps e ON
> h.subordinateid = e.empid
> // )
> // SELECT empid, name FROM delta
> .scan("s", "emps")
> .filter(
> builder.equals(
> builder.field("empid"),
> builder.literal(2)))
> .project(
> builder.field("emps", "empid"),
> builder.field("emps", "name"))
> .transientScan("#DELTA#");
> RelNode transientScan = builder.build(); // pop the transientScan
> to use it later
> builder
> .scan("s", "hierarchies")
> .push(transientScan) // use the transientScan as right input of
> the join
> .join(
> JoinRelType.INNER,
> builder.equals(
> builder.field(2, "#DELTA#", "empid"),
> builder.field(2, "hierarchies", "managerid")))
> .scan("s", "emps")
> .join(
> JoinRelType.INNER,
> builder.equals(
> builder.field(2, "hierarchies", "subordinateid"),
> builder.field(2, "emps", "empid")))
> .project(
> builder.field("emps", "empid"),
> builder.field("emps", "name"))
> .repeatUnion("#DELTA#", true);
> return builder.build();
> })
> .explainHookMatches(""
> + "EnumerableRepeatUnion(all=[true])\n"
> + " EnumerableTableSpool(readType=[LAZY], writeType=[LAZY],
> table=[[#DELTA#]])\n"
> + " EnumerableCalc(expr#0..4=[{inputs}], expr#5=[2],
> expr#6=[=($t0, $t5)], empid=[$t0], name=[$t2], $condition=[$t6])\n"
> + " EnumerableTableScan(table=[[s, emps]])\n"
> + " EnumerableTableSpool(readType=[LAZY], writeType=[LAZY],
> table=[[#DELTA#]])\n"
> + " EnumerableCalc(expr#0..8=[{inputs}], empid=[$t4],
> name=[$t6])\n"
> + " EnumerableCorrelate(correlation=[$cor1],
> joinType=[inner], requiredColumns=[{1}])\n"
> // It is important to have EnumerableCorrelate + #DELTA# table
> scan on its right
> + " EnumerableCorrelate(correlation=[$cor0],
> joinType=[inner], requiredColumns=[{0}])\n"
> + " EnumerableTableScan(table=[[s, hierarchies]])\n"
> + " EnumerableCalc(expr#0..1=[{inputs}], expr#2=[$cor0],
> expr#3=[$t2.managerid], expr#4=[=($t0, $t3)], proj#0..1=[{exprs}],
> $condition=[$t4])\n"
> + " EnumerableInterpreter\n"
> + " BindableTableScan(table=[[#DELTA#]])\n"
> + " EnumerableCalc(expr#0..4=[{inputs}], expr#5=[$cor1],
> expr#6=[$t5.subordinateid], expr#7=[=($t6, $t0)], proj#0..4=[{exprs}],
> $condition=[$t7])\n"
> + " EnumerableTableScan(table=[[s, emps]])\n")
> .returnsUnordered(""
> + "empid=2; name=Emp2\n"
> + "empid=3; name=Emp3\n"
> + "empid=5; name=Emp5");
> }
> {code}
> Which fails with NPE on the dynamic generated code:
> {code}
> Error while executing SQL "?": null
> java.sql.SQLException: Error while executing SQL "?": null
> at org.apache.calcite.avatica.Helper.createException(Helper.java:56)
> at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:163)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227)
> at
> org.apache.calcite.test.CalciteAssert.assertQuery(CalciteAssert.java:532)
> at
> org.apache.calcite.test.CalciteAssert$AssertQuery.lambda$returns$1(CalciteAssert.java:1511)
> at
> org.apache.calcite.test.CalciteAssert$AssertQuery.withConnection(CalciteAssert.java:1450)
> at
> org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1509)
> at
> org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1499)
> at
> org.apache.calcite.test.CalciteAssert$AssertQuery.returnsUnordered(CalciteAssert.java:1521)
> at
> org.apache.calcite.test.enumerable.EnumerableRepeatUnionTest.testRepeatUnionWithCorrelateWithTransientScanOnItsRight(EnumerableRepeatUnionTest.java:296)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:675)
> at
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:125)
> at
> org.junit.jupiter.engine.extension.TimeoutInvocation.proceed(TimeoutInvocation.java:46)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:139)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:131)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:81)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:104)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:62)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:43)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:35)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
> at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:202)
> at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:198)
> at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
> at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:69)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:135)
> at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
> at
> org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
> at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
> at
> org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:171)
> at
> org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService.invokeAll(ForkJoinPoolHierarchicalTestExecutorService.java:115)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
> at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
> at
> org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
> at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
> at
> org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:171)
> at
> org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService.invokeAll(ForkJoinPoolHierarchicalTestExecutorService.java:115)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
> at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
> at
> org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
> at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
> at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
> at
> org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:171)
> at java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
> at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
> at
> java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
> at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
> at
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
> Caused by: java.lang.NullPointerException
> at Baz.bind(Unknown Source)
> at
> org.apache.calcite.jdbc.CalcitePrepare$CalciteSignature.enumerable(CalcitePrepare.java:355)
> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.enumerable(CalciteConnectionImpl.java:315)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl._createIterable(CalciteMetaImpl.java:507)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl.createIterable(CalciteMetaImpl.java:498)
> at
> org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:182)
> at
> org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:64)
> at
> org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:43)
> at
> org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:667)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:567)
> at
> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
> ... 66 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
