Niels Pardon created CALCITE-6261: ------------------------------------- Summary: RelBuilder.aggregate() field pruning does not use permuted field indices when used with force project and duplicate agg calls Key: CALCITE-6261 URL: https://issues.apache.org/jira/browse/CALCITE-6261 Project: Calcite Issue Type: Bug Components: core Affects Versions: 1.36.0, 1.35.0 Reporter: Niels Pardon
We were running into issues with RelBuilder.aggregate() when we tried to upgrade our code from Calcite 1.32.0 to 1.36.0 and after some debugging it seems this is caused by changes introduced with CALCITE-4334. I was able to put together this test case for RelBuilderTest reproducing the issue: {code:java} @Test void testAggregateForceProject() { final Function<RelBuilder, RelBuilder> f1 = builder -> builder.scan("EMP") .project( ImmutableList.of( builder.field("EMPNO"), builder.field("ENAME"), builder.field("JOB"), builder.field("MGR"), builder.field("HIREDATE"), builder.field("SAL"), builder.field("COMM"), builder.field("DEPTNO")), ImmutableList.of(), true); final Function<RelBuilder, RelBuilder> f2 = builder -> builder.aggregate( builder.groupKey(builder.field("MGR")), builder.avg(false, "SALARY_AVG", builder.field("SAL")), builder.sum(false, "SALARY_SUM", builder.field("SAL")), builder.avg(false, "SALARY_MEAN", builder.field("SAL"))); final String expected = "" + "LogicalProject(MGR=[$0], SALARY_AVG=[$1], SALARY_SUM=[$2], SALARY_MEAN=[$1])\n" + " LogicalAggregate(group=[{0}], SALARY_AVG=[AVG($1)], SALARY_SUM=[SUM($1)])\n" + " LogicalProject(MGR=[$3], SAL=[$5])\n" + " LogicalTableScan(table=[[scott, EMP]])\n"; assertThat(f2.apply(f1.apply(createBuilder())).build(), hasTree(expected)); } {code} With Calcite 1.36.0 you will get an AssertionError: {code:java} java.lang.AssertionError at org.apache.calcite.rel.core.Aggregate.<init>(Aggregate.java:175) at org.apache.calcite.rel.logical.LogicalAggregate.<init>(LogicalAggregate.java:72) at org.apache.calcite.rel.logical.LogicalAggregate.create_(LogicalAggregate.java:144) at org.apache.calcite.rel.logical.LogicalAggregate.create(LogicalAggregate.java:116) at org.apache.calcite.rel.core.RelFactories$AggregateFactoryImpl.createAggregate(RelFactories.java:333) at org.apache.calcite.tools.RelBuilder.aggregate_(RelBuilder.java:2576) at org.apache.calcite.tools.RelBuilder.aggregate_(RelBuilder.java:2523) at org.apache.calcite.tools.RelBuilder.aggregate(RelBuilder.java:2335) at org.apache.calcite.test.RelBuilderTest.lambda$testAggregateForceProject$36(RelBuilderTest.java:1509) at org.apache.calcite.test.RelBuilderTest.testAggregateForceProject(RelBuilderTest.java:1522) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727) at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60) at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131) at org.junit.jupiter.engine.extension.SameThreadTimeoutInvocation.proceed(SameThreadTimeoutInvocation.java:45) at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156) at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147) at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86) at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103) at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93) at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106) at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64) at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45) at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37) at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92) at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:217) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:213) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:138) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:68) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141) at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138) at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95) at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185) at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService.invokeAll(ForkJoinPoolHierarchicalTestExecutorService.java:129) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141) at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138) at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95) at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185) at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService.invokeAll(ForkJoinPoolHierarchicalTestExecutorService.java:129) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141) at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138) at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95) at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185) at java.base/java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189) at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183) {code} The issue is caused by the introduction of groupSet2 and groupSets2 which now contain the permuted field indices while groupSet/groupSets still uses the unchanged field indices and the last portion of the aggregate_() method in RelBuilder still using the original groupSet and groupSets variables instead of the newly introduced groupSet2 / groupSets2: https://github.com/apache/calcite/blob/abf462a7ddab11a44610827b0cd69510c099d9b6/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L2509-L2528 -- This message was sent by Atlassian Jira (v8.20.10#820010)