[jira] [Commented] (CALCITE-3244) RelDecorrelator unable to decorrelate expression with filter and aggregate on top
[
https://issues.apache.org/jira/browse/CALCITE-3244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16917529#comment-16917529
]
Volodymyr Vysotskyi commented on CALCITE-3244:
--
[~danny0405], currently I don't have any idea how to fix this issue. Some time
ago I have played with enforcing three-value-logic for such types of
subqueries, but it didn't help.
> RelDecorrelator unable to decorrelate expression with filter and aggregate on
> top
> -
>
> Key: CALCITE-3244
> URL: https://issues.apache.org/jira/browse/CALCITE-3244
> Project: Calcite
> Issue Type: Improvement
>Affects Versions: 1.20.0
>Reporter: benj
>Assignee: Danny Chan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Some very useful type of requests currently failed with:
> {code:java}
> SYSTEM ERROR: UnsupportedOperationException: Adding Implicit RowID column is
> not supported for ValuesPrel operator
> {code}
> Examples from DRILL-7050:
> {code:sql}
> select t1.id,
> (select count(t2.id)
> from (
> select 1 as id
> union all
> select 2 as id
> ) t2 where t2.id = t1.id)
> from (
> select 1 as id
> union all
> select 2 as id
> ) t1
> {code}
> {code:sql}
> SELECT t,
> (SELECT count(*) FROM
> (SELECT split(r,' ') AS r FROM
> (SELECT sub.t AS r)) AS x
> ,LATERAL(SELECT $unnest AS u FROM unnest(x.r))
> /* WHERE ... */) t2
> FROM
> (SELECT 'unnest is useful' AS t) sub
> {code}
>
> _Please note that in 1.18 the error for these requests was:_
> {code:java}
> Error: PLAN ERROR: Cannot convert RexNode to equivalent Drill expression.
> RexNode Class: org.apache.calcite.rex.RexCorrelVariable, RexNode Digest: $cor0
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Comment Edited] (CALCITE-3244) RelDecorrelator unable to decorrelate expression with filter and aggregate on top
[
https://issues.apache.org/jira/browse/CALCITE-3244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916763#comment-16916763
]
Volodymyr Vysotskyi edited comment on CALCITE-3244 at 8/27/19 2:45 PM:
---
I have checked your branch, and problem with types wasn't fixed fully, so for
example, the following test fails:
{code:java}
@Test public void testDecorrelateExists3() throws Exception {
final String sql = "select \n"
+ " exists (select min(t2.id) \n"
+ "from (values(1), (2)) t2(id) where t2.id = t1.id)\n"
+ "from (values(3), (4)) t1(id)";
checkSubQuery(sql).withLateDecorrelation(true).check();
}
{code}
{noformat}
java.lang.AssertionError: Cannot add expression of different type to set:
set type is RecordType(INTEGER NOT NULL ID, BOOLEAN i) NOT NULL
expression type is RecordType(INTEGER NOT NULL ID, BOOLEAN NOT NULL i) NOT NULL
set is
rel#44:LogicalCorrelate(left=HepRelVertex#36,right=HepRelVertex#43,correlation=$cor0,joinType=left,requiredColumns={0})
expression is LogicalProject(ID=[$0], i=[true])
LogicalCorrelate(correlation=[$cor0], joinType=[left], requiredColumns=[{0}])
LogicalValues(tuples=[[{ 3 }, { 4 }]])
LogicalAggregate(group=[{}], EXPR$0=[MIN($0)])
LogicalFilter(condition=[=($0, $cor0.ID)])
LogicalValues(tuples=[[{ 1 }, { 2 }]])
at
org.apache.calcite.plan.RelOptUtil.verifyTypeEquivalence(RelOptUtil.java:383)
at
org.apache.calcite.plan.hep.HepRuleCall.transformTo(HepRuleCall.java:57)
at
org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:236)
at
org.apache.calcite.sql2rel.RelDecorrelator$AdjustProjectForCountAggregateRule.onMatch2(RelDecorrelator.java:2528)
at
org.apache.calcite.sql2rel.RelDecorrelator$AdjustProjectForCountAggregateRule.onMatch(RelDecorrelator.java:2436)
at
org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:319)
at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:560)
at
org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:419)
at
org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:256)
at
org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:127)
at
org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:215)
at
org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:202)
at
org.apache.calcite.sql2rel.RelDecorrelator.decorrelate(RelDecorrelator.java:250)
at
org.apache.calcite.sql2rel.RelDecorrelator.decorrelateQuery(RelDecorrelator.java:215)
at
org.apache.calcite.test.RelOptTestBase.checkPlanning(RelOptTestBase.java:213)
at
org.apache.calcite.test.RelOptTestBase$Sql.check(RelOptTestBase.java:338)
at
org.apache.calcite.test.RelOptTestBase$Sql.check(RelOptTestBase.java:321)
at
org.apache.calcite.test.RelOptRulesTest.testDecorrelateExists2(RelOptRulesTest.java:5626)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at
com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
at
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at
[jira] [Commented] (CALCITE-3244) RelDecorrelator unable to decorrelate expression with filter and aggregate on top
[
https://issues.apache.org/jira/browse/CALCITE-3244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916763#comment-16916763
]
Volodymyr Vysotskyi commented on CALCITE-3244:
--
I have checked your branch, and the following test fails:
{code:java}
@Test public void testDecorrelateExists3() throws Exception {
final String sql = "select \n"
+ " exists (select min(t2.id) \n"
+ "from (values(1), (2)) t2(id) where t2.id = t1.id)\n"
+ "from (values(3), (4)) t1(id)";
checkSubQuery(sql).withLateDecorrelation(true).check();
}
{code}
{noformat}
java.lang.AssertionError: Cannot add expression of different type to set:
set type is RecordType(INTEGER NOT NULL ID, BOOLEAN i) NOT NULL
expression type is RecordType(INTEGER NOT NULL ID, BOOLEAN NOT NULL i) NOT NULL
set is
rel#44:LogicalCorrelate(left=HepRelVertex#36,right=HepRelVertex#43,correlation=$cor0,joinType=left,requiredColumns={0})
expression is LogicalProject(ID=[$0], i=[true])
LogicalCorrelate(correlation=[$cor0], joinType=[left], requiredColumns=[{0}])
LogicalValues(tuples=[[{ 3 }, { 4 }]])
LogicalAggregate(group=[{}], EXPR$0=[MIN($0)])
LogicalFilter(condition=[=($0, $cor0.ID)])
LogicalValues(tuples=[[{ 1 }, { 2 }]])
at
org.apache.calcite.plan.RelOptUtil.verifyTypeEquivalence(RelOptUtil.java:383)
at
org.apache.calcite.plan.hep.HepRuleCall.transformTo(HepRuleCall.java:57)
at
org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:236)
at
org.apache.calcite.sql2rel.RelDecorrelator$AdjustProjectForCountAggregateRule.onMatch2(RelDecorrelator.java:2528)
at
org.apache.calcite.sql2rel.RelDecorrelator$AdjustProjectForCountAggregateRule.onMatch(RelDecorrelator.java:2436)
at
org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:319)
at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:560)
at
org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:419)
at
org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:256)
at
org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:127)
at
org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:215)
at
org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:202)
at
org.apache.calcite.sql2rel.RelDecorrelator.decorrelate(RelDecorrelator.java:250)
at
org.apache.calcite.sql2rel.RelDecorrelator.decorrelateQuery(RelDecorrelator.java:215)
at
org.apache.calcite.test.RelOptTestBase.checkPlanning(RelOptTestBase.java:213)
at
org.apache.calcite.test.RelOptTestBase$Sql.check(RelOptTestBase.java:338)
at
org.apache.calcite.test.RelOptTestBase$Sql.check(RelOptTestBase.java:321)
at
org.apache.calcite.test.RelOptRulesTest.testDecorrelateExists2(RelOptRulesTest.java:5626)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at
com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
at
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
{noformat}
> RelDecorrelator unable to decorrelate expression with filter and aggregate on
> top
>
[jira] [Commented] (CALCITE-3244) RelDecorrelator unable to decorrelate expression with filter and aggregate on top
[
https://issues.apache.org/jira/browse/CALCITE-3244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916743#comment-16916743
]
Volodymyr Vysotskyi commented on CALCITE-3244:
--
I have also worked on this issue, here is my branch with the changes
[https://github.com/vvysotskyi/calcite/tree/CALCITE-3244] (without modifying
some tests baselines and there are some similar things to things you did, but
there are some additional changes connected with attempts to fix another issue).
Regarding disabled unit test, fix for this issue will cause wrong results for
queries which currently work, though without decorrelation. Looks like
[~julianhyde] predicted this issue in his comment above and we should fix it
with current issue.
> RelDecorrelator unable to decorrelate expression with filter and aggregate on
> top
> -
>
> Key: CALCITE-3244
> URL: https://issues.apache.org/jira/browse/CALCITE-3244
> Project: Calcite
> Issue Type: Improvement
>Affects Versions: 1.20.0
>Reporter: benj
>Assignee: Danny Chan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Some very useful type of requests currently failed with:
> {code:java}
> SYSTEM ERROR: UnsupportedOperationException: Adding Implicit RowID column is
> not supported for ValuesPrel operator
> {code}
> Examples from DRILL-7050:
> {code:sql}
> select t1.id,
> (select count(t2.id)
> from (
> select 1 as id
> union all
> select 2 as id
> ) t2 where t2.id = t1.id)
> from (
> select 1 as id
> union all
> select 2 as id
> ) t1
> {code}
> {code:sql}
> SELECT t,
> (SELECT count(*) FROM
> (SELECT split(r,' ') AS r FROM
> (SELECT sub.t AS r)) AS x
> ,LATERAL(SELECT $unnest AS u FROM unnest(x.r))
> /* WHERE ... */) t2
> FROM
> (SELECT 'unnest is useful' AS t) sub
> {code}
>
> _Please note that in 1.18 the error for these requests was:_
> {code:java}
> Error: PLAN ERROR: Cannot convert RexNode to equivalent Drill expression.
> RexNode Class: org.apache.calcite.rex.RexCorrelVariable, RexNode Digest: $cor0
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Resolved] (CALCITE-3138) RelStructuredTypeFlattener doesn't restructure ROW type fields
[
https://issues.apache.org/jira/browse/CALCITE-3138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi resolved CALCITE-3138.
--
Resolution: Fixed
Fixed in
[1e62d3d|https://github.com/apache/calcite/commit/1e62d3d64fc217d14016702237b4f8d56b3683f2].
Thanks, [~IhorHuzenko], for fixing this issue and [~danny0405] for the review!
> RelStructuredTypeFlattener doesn't restructure ROW type fields
> ---
>
> Key: CALCITE-3138
> URL: https://issues.apache.org/jira/browse/CALCITE-3138
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Haisheng Yuan
>Assignee: Igor Guzenko
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.21.0
>
> Attachments: ROW_repro.patch
>
> Time Spent: 7h 20m
> Remaining Estimate: 0h
>
> 1) RelStructuredTypeFlattener.restructureFields(structType) doesn't support
> ROW type. However, ROW type is flattened by
> RelStructuredTypeFlattener just like struct. So when user queries one column
> with complex type ROW, after flattening and restructuring
> top level project returns reference to first inner primitive field of the
> requested column.
> 2) Another finding is related to ITEM expression applied to array of structs
> column. For example, let's imagine a table with column of
> type ARRAY>>. When user requests is SQL
> array_column[1], Calcite generates ITEM($0, 1) , where $0 is ref to array
> column
> from Scan and 1 is index literal. Current flattener generates two field acess
> expressions ITEM($0, 1).a, ITEM($0, 1).b but dont take into account
> that ITEM($0, 1).b returns struct which also should be flattened.
> 3) In some cases applying of ITEM after flattenning is no longer possible.
> For example, consider column with type
> STRUCT>. User requests column['b'] in query and
> Calcite creates ITEM($0,'b').
> After flattenning Scan is covered by LogicalProject($0.a, $0.b.x, $0.b.y) and
> the old projection ITEM($0,'b') can't
> be applied anymore. So now it should be converted to refer only subset of
> fields ($1,$2) from flattening project.
> UPDATES IN EXPECTED TEST RESULTS:
> --
> TEST CASE: SqlToRelConverterTest.testNestedColumnType()
> {code}
> select empa.home_address.zip from sales.emp_address empa where
> empa.home_address.city = 'abc'
> {code}
> OLD RESULT:
> {code}
> LogicalProject(ZIP=[$4])
> LogicalFilter(condition=[=($3, 'abc':VARCHAR(20))])
> LogicalProject(EMPNO=[$0], STREET=[$1.STREET], CITY=[$1.CITY],
> ZIP=[$1.ZIP], STATE=[$1.STATE], STREET5=[$2.STREET], CITY6=[$2.CITY],
> ZIP7=[$2.ZIP], STATE8=[$2.STATE])
> LogicalTableScan(table=[[CATALOG, SALES, EMP_ADDRESS]])
> {code}
> 1. Above in logical filter, condition references to field $3 which is
> ZIP=[$1.ZIP] field from previous project,
>however in original query filtering should be done by CITY field.
> 2. Also the top level project references to $4 field, which is
> STATE=[$1.STATE] field from project, but original
>query requested ZIP field.
>
> UPDATED RESULT:
> {code}
> LogicalProject(ZIP=[$3])
> LogicalFilter(condition=[=($2, 'abc')])
> LogicalProject(EMPNO=[$0], STREET=[$1.STREET], CITY=[$1.CITY],
> ZIP=[$1.ZIP], STATE=[$1.STATE], STREET5=[$2.STREET], CITY6=[$2.CITY],
> ZIP7=[$2.ZIP], STATE8=[$2.STATE])
> LogicalTableScan(table=[[CATALOG, SALES, EMP_ADDRESS]])
> {code}
> --
> TEST CASE: SqlToRelConverterTest.testStructTypeAlias()
> {code}
> select t.r AS myRow from (select row(row(1)) r from dept) t
> {code}
> OLD RESULT:
> {code}
> LogicalProject(MYROW$$0$$0=[1])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> {code}
> 1. Inside the subselect of row(row(1)) type of returned column is
> RecordType(RecordType(INTEGER EXPR$0) EXPR$0),
>but the top level project uses flattened expression and returns to user
> literal 1 with type RecordType(INTEGER MYROW$$0$$0),
>although the type doesn't match type returned by row(row(1)) expression.
> 2. Also it's suspicious that caller expects returned column to have name
> 'myRow' but gets 'MYROW$$0$$0'.
>
> UPDATED RESULT:
> {code}
> LogicalProject(MYROW=[ROW(ROW($0))])
> LogicalProject(MYROW$$0$$0=[1])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> {code}
> --
> TEST CASE: SqlToRelConverterTest.testFlattenRecords()
> {code}
> select employees[1] from dept_nested
> {code}
> OLD RESULT:
> {code}
> LogicalProject(EXPR$0=[$0])
> LogicalProject(EXPR$0$0=[ITEM($3, 1).EMPNO], EXPR$0$1=[ITEM($3, 1).ENAME],
> EXPR$0$2=[ITEM($3,
[jira] [Comment Edited] (CALCITE-2166) Cumulative cost of RelSubset.best RelNode is increased after calling RelSubset.propagateCostImprovements() for input RelNodes
[
https://issues.apache.org/jira/browse/CALCITE-2166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16912124#comment-16912124
]
Volodymyr Vysotskyi edited comment on CALCITE-2166 at 8/21/19 11:58 AM:
Thanks, Xiening Dai, for moving this issue forward! I've also considered
similar options some time ago, and here are my thoughts regarding them:
1 & 2: Currently we store the best cost value to avoid recalculating it every
time when a new rel node is added to the rel subset. For the case when we have
to recalculate its cost, we also have to clear rel metadata cache for current
rel node, and for the case when something is changed, also recalculate parent
rel nodes costs.
I agree that #3 is too complex, and I don't have other solutions for this issue.
But in either case, it would be good if 1 or 2 option will help to fix this
issue partially without introducing significant performance degradation.
was (Author: vvysotskyi):
Thanks, Xiening Dai, for moving this issue forward! I've also considered
similar options, and here are my thoughts regarding them:
1 & 2: Currently we store the best cost value to avoid recalculating it every
time when a new rel node is added to the rel subset. For the case when we have
to recalculate its cost, we also have to clear rel metadata cache for current
rel node, and for the case when something is changed, also recalculate parent
rel nodes costs.
I agree that #3 is too complex, and I don't have other solutions for this issue.
But in either case, it would be good if 1 or 2 option will help to fix this
issue partially without introducing significant performance degradation.
> Cumulative cost of RelSubset.best RelNode is increased after calling
> RelSubset.propagateCostImprovements() for input RelNodes
> -
>
> Key: CALCITE-2166
> URL: https://issues.apache.org/jira/browse/CALCITE-2166
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.15.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Danny Chan
>Priority: Critical
>
> After calling {{RelSubset.propagateCostImprovements()}} cumulative cost of
> {{RelSubset.best}} {{RelNode}} may be increased due to the increase of the
> non-cumulative cost caused by changing of input best {{RelNode}}.
> To observe this issue, add this code:
> {code:java}
> if (subset.best != null) {
> RelOptCost bestCost = getCost(subset.best,
> RelMetadataQuery.instance());
> if (!subset.bestCost.equals(bestCost)) {
> throw new AssertionError(
> "relSubset [" + subset.getDescription()
> + "] has wrong best cost "
> + subset.bestCost + ". Correct cost is " + bestCost);
> }
> }
> {code}
> into {{VolcanoPlanner.validate()}} method (line 907).
> List of unit tests which fail with this check:
> {noformat}
> Failed tests:
>
> MaterializationTest.testJoinMaterializationUKFK9:1823->checkMaterialize:198->checkMaterialize:205->checkThatMaterialize:233
> relSubset [rel#226287:Subset#8.ENUMERABLE.[]] has wrong best cost {221.5
> rows, 128.25 cpu, 0.0 io}. Correct cost is {233.0 rows, 178.0 cpu, 0.0 io}
> ScannableTableTest.testPFPushDownProjectFilterAggregateNested:279 relSubset
> [rel#12950:Subset#5.ENUMERABLE.[]] has wrong best cost {63.8 rows, 62.308
> cpu, 0.0 io}. Correct cost is {70.4 rows, 60.404 cpu, 0.0 io}
> ScannableTableTest.testPFTableRefusesFilterCooperative:221 relSubset
> [rel#13382:Subset#2.ENUMERABLE.[]] has wrong best cost {81.0 rows, 181.01
> cpu, 0.0 io}. Correct cost is {150.5 rows, 250.505 cpu, 0.0 io}
> ScannableTableTest.testProjectableFilterableCooperative:148 relSubset
> [rel#13611:Subset#2.ENUMERABLE.[]] has wrong best cost {81.0 rows, 181.01
> cpu, 0.0 io}. Correct cost is {150.5 rows, 250.505 cpu, 0.0 io}
> ScannableTableTest.testProjectableFilterableNonCooperative:165 relSubset
> [rel#13754:Subset#2.ENUMERABLE.[]] has wrong best cost {81.0 rows, 181.01
> cpu, 0.0 io}. Correct cost is {150.5 rows, 250.505 cpu, 0.0 io}
> FrameworksTest.testUpdate:336->executeQuery:367 relSubset
> [rel#22533:Subset#2.ENUMERABLE.any] has wrong best cost {19.5 rows, 37.75
> cpu, 0.0 io}. Correct cost is {22.575 rows, 52.58 cpu, 0.0 io}
> {noformat}
> For the test {{MaterializationTest.testJoinMaterializationUKFK9}} initial
> best plan was:
> {noformat}
> EnumerableProject(empid0=[$5], empid00=[$5], deptno0=[$7]): rowcount = 15.0,
> cumulative cost = {15.0 rows, 45.0 cpu, 0.0 io}, id = 3989
> EnumerableJoin(subset=[rel#3988:Subset#34.ENUMERABLE.[]], condition=[=($1,
> $7)], joinType=[inner]): rowcount = 15.0, cumulative cost = {116.0 rows, 0.0
> cpu,
[jira] [Commented] (CALCITE-2166) Cumulative cost of RelSubset.best RelNode is increased after calling RelSubset.propagateCostImprovements() for input RelNodes
[
https://issues.apache.org/jira/browse/CALCITE-2166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16912124#comment-16912124
]
Volodymyr Vysotskyi commented on CALCITE-2166:
--
Thanks, Xiening Dai, for moving this issue forward! I've also considered
similar options, and here are my thoughts regarding them:
1 & 2: Currently we store the best cost value to avoid recalculating it every
time when a new rel node is added to the rel subset. For the case when we have
to recalculate its cost, we also have to clear rel metadata cache for current
rel node, and for the case when something is changed, also recalculate parent
rel nodes costs.
I agree that #3 is too complex, and I don't have other solutions for this issue.
But in either case, it would be good if 1 or 2 option will help to fix this
issue partially without introducing significant performance degradation.
> Cumulative cost of RelSubset.best RelNode is increased after calling
> RelSubset.propagateCostImprovements() for input RelNodes
> -
>
> Key: CALCITE-2166
> URL: https://issues.apache.org/jira/browse/CALCITE-2166
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.15.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Danny Chan
>Priority: Critical
>
> After calling {{RelSubset.propagateCostImprovements()}} cumulative cost of
> {{RelSubset.best}} {{RelNode}} may be increased due to the increase of the
> non-cumulative cost caused by changing of input best {{RelNode}}.
> To observe this issue, add this code:
> {code:java}
> if (subset.best != null) {
> RelOptCost bestCost = getCost(subset.best,
> RelMetadataQuery.instance());
> if (!subset.bestCost.equals(bestCost)) {
> throw new AssertionError(
> "relSubset [" + subset.getDescription()
> + "] has wrong best cost "
> + subset.bestCost + ". Correct cost is " + bestCost);
> }
> }
> {code}
> into {{VolcanoPlanner.validate()}} method (line 907).
> List of unit tests which fail with this check:
> {noformat}
> Failed tests:
>
> MaterializationTest.testJoinMaterializationUKFK9:1823->checkMaterialize:198->checkMaterialize:205->checkThatMaterialize:233
> relSubset [rel#226287:Subset#8.ENUMERABLE.[]] has wrong best cost {221.5
> rows, 128.25 cpu, 0.0 io}. Correct cost is {233.0 rows, 178.0 cpu, 0.0 io}
> ScannableTableTest.testPFPushDownProjectFilterAggregateNested:279 relSubset
> [rel#12950:Subset#5.ENUMERABLE.[]] has wrong best cost {63.8 rows, 62.308
> cpu, 0.0 io}. Correct cost is {70.4 rows, 60.404 cpu, 0.0 io}
> ScannableTableTest.testPFTableRefusesFilterCooperative:221 relSubset
> [rel#13382:Subset#2.ENUMERABLE.[]] has wrong best cost {81.0 rows, 181.01
> cpu, 0.0 io}. Correct cost is {150.5 rows, 250.505 cpu, 0.0 io}
> ScannableTableTest.testProjectableFilterableCooperative:148 relSubset
> [rel#13611:Subset#2.ENUMERABLE.[]] has wrong best cost {81.0 rows, 181.01
> cpu, 0.0 io}. Correct cost is {150.5 rows, 250.505 cpu, 0.0 io}
> ScannableTableTest.testProjectableFilterableNonCooperative:165 relSubset
> [rel#13754:Subset#2.ENUMERABLE.[]] has wrong best cost {81.0 rows, 181.01
> cpu, 0.0 io}. Correct cost is {150.5 rows, 250.505 cpu, 0.0 io}
> FrameworksTest.testUpdate:336->executeQuery:367 relSubset
> [rel#22533:Subset#2.ENUMERABLE.any] has wrong best cost {19.5 rows, 37.75
> cpu, 0.0 io}. Correct cost is {22.575 rows, 52.58 cpu, 0.0 io}
> {noformat}
> For the test {{MaterializationTest.testJoinMaterializationUKFK9}} initial
> best plan was:
> {noformat}
> EnumerableProject(empid0=[$5], empid00=[$5], deptno0=[$7]): rowcount = 15.0,
> cumulative cost = {15.0 rows, 45.0 cpu, 0.0 io}, id = 3989
> EnumerableJoin(subset=[rel#3988:Subset#34.ENUMERABLE.[]], condition=[=($1,
> $7)], joinType=[inner]): rowcount = 15.0, cumulative cost = {116.0 rows, 0.0
> cpu, 0.0 io}, id = 4797
> EnumerableFilter(subset=[rel#4274:Subset#47.ENUMERABLE.[0]],
> condition=[=(CAST($2):VARCHAR CHARACTER SET "ISO-8859-1" COLLATE
> "ISO-8859-1$en_US$primary", 'Bill')]): rowcount = 1.0, cumulative cost = {1.0
> rows, 1.0 cpu, 0.0 io}, id = 16522
> EnumerableTableScan(subset=[rel#158:Subset#11.ENUMERABLE.[0]],
> table=[[hr, m0]]): rowcount = 1.0, cumulative cost = {0.0 rows, 1.0 cpu, 0.0
> io}, id = 79
> EnumerableTableScan(subset=[rel#115:Subset#5.ENUMERABLE.[]], table=[[hr,
> depts]]): rowcount = 100.0, cumulative cost = {100.0 rows, 101.0 cpu, 0.0
> io}, id = 62
> {noformat}
> Its cumulative cost is \{221.5 rows, 123.75 cpu, 0.0 io}
> After applying some rules it became:
> {noformat}
> EnumerableProject(empid0=[$3], empid00=[$3],
[jira] [Commented] (CALCITE-3257) Need to clear RelMetaDataQuery cache after dumping RelNodes and Graphviz
[
https://issues.apache.org/jira/browse/CALCITE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909448#comment-16909448
]
Volodymyr Vysotskyi commented on CALCITE-3257:
--
I don't know the exact reason for adding calls to invalidate rel nodes after
the rule is applied, so I will operate only my assumptions.
I think that it was done to force recalculating rel node cost after the rule is
applied, so ideally we would have the correct cost for both unchanged and
updated rel nodes - when the rule is applied, child rel nodes were changed and
cost of current rel node was calculated considering children costs instead of
using the outdated cached value.
So ideally for the case when the tree wasn't changed, there is no need to
invalidate the cache, since cached values should be correct.
One of the issues, fixed in the scope of CALCITE-2018 was the case when several
instances caches are used in the same time - one cache instance was invalidated
after the rule is applied, but another one was used during the next rule
applying.
*As I see from the code, {{RuleQueue.dump()}} does not calculate rel nodes
cost...*
I think something similar happens in this case so invalidating the cache after
(or perhaps also before) calling {{dump()}} method will help to avoid this
issue, but the problem with coexisting several cache instances will still be
present.
Could you please check that CALCITE-2018 fixes the issue you have observed?
> Need to clear RelMetaDataQuery cache after dumping RelNodes and Graphviz
>
>
> Key: CALCITE-3257
> URL: https://issues.apache.org/jira/browse/CALCITE-3257
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Xiening Dai
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> To repro -
> 1. Set Log4J log level to TRACE. So planner will dump rel node info every
> time at rule match
> (https://github.com/apache/calcite/blob/3124a85b93ff2f1b79484c7bd4cc41835d4f1920/core/src/main/java/org/apache/calcite/plan/volcano/RuleQueue.java#L435)
> 2. Run JdbcTest.testNotExistsCorrelated. Get below exception -
> java.lang.AssertionError: rel
> [rel#63:EnumerableAggregate.ENUMERABLE.[](input=RelSubset#62,group={0})] has
> lower cost {131.0 rows, 216.0 cpu, 0.0 io} than best cost {131.5 rows, 216.0
> cpu, 0.0 io} of subset [rel#60:Subset#4.ENUMERABLE.[]]
> at org.apache.calcite.util.Litmus$1.fail(Litmus.java:31)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.isValid(VolcanoPlanner.java:889)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:852)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:869)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1928)
> at
> org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:129)
> at
> org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:236)
> at
> org.apache.calcite.rel.convert.ConverterRule.onMatch(ConverterRule.java:141)
> at
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:208)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:631)
> at
> org.apache.calcite.tools.Programs.lambda$standard$3(Programs.java:286)
> at
> org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:346)
> at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:189)
> at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:314)
> at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:231)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:638)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:502)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:472)
> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:231)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:550)
> at
> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227)
> at
> org.apache.calcite.test.CalciteAssert.assertQuery(CalciteAssert.java:522)
> at
> org.apache.calcite.test.CalciteAssert$AssertQuery.lambda$returns$1(CalciteAssert.java:1466)
> at
>
[jira] [Commented] (CALCITE-3257) Need to clear RelMetaDataQuery cache after dumping RelNodes and Graphviz
[
https://issues.apache.org/jira/browse/CALCITE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909375#comment-16909375
]
Volodymyr Vysotskyi commented on CALCITE-3257:
--
I don't think that adding a call to RelOptCluster.invalidateMetadataQuery()
after dumping rel nodes is a good idea. As pointed in one of the comments in
CALCITE-2018, this issue also happens for the case when a trace is not enabled.
The proposed fix will just hide the issue for some cases, but more important
and hard-to-investigate issues like incorrect planning and others will still be
present.
> Need to clear RelMetaDataQuery cache after dumping RelNodes and Graphviz
>
>
> Key: CALCITE-3257
> URL: https://issues.apache.org/jira/browse/CALCITE-3257
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Xiening Dai
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> To repro -
> 1. Set Log4J log level to TRACE. So planner will dump rel node info every
> time at rule match
> (https://github.com/apache/calcite/blob/3124a85b93ff2f1b79484c7bd4cc41835d4f1920/core/src/main/java/org/apache/calcite/plan/volcano/RuleQueue.java#L435)
> 2. Run JdbcTest.testNotExistsCorrelated. Get below exception -
> java.lang.AssertionError: rel
> [rel#63:EnumerableAggregate.ENUMERABLE.[](input=RelSubset#62,group={0})] has
> lower cost {131.0 rows, 216.0 cpu, 0.0 io} than best cost {131.5 rows, 216.0
> cpu, 0.0 io} of subset [rel#60:Subset#4.ENUMERABLE.[]]
> at org.apache.calcite.util.Litmus$1.fail(Litmus.java:31)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.isValid(VolcanoPlanner.java:889)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:852)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:869)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1928)
> at
> org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:129)
> at
> org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:236)
> at
> org.apache.calcite.rel.convert.ConverterRule.onMatch(ConverterRule.java:141)
> at
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:208)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:631)
> at
> org.apache.calcite.tools.Programs.lambda$standard$3(Programs.java:286)
> at
> org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:346)
> at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:189)
> at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:314)
> at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:231)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:638)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:502)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:472)
> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:231)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:550)
> at
> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227)
> at
> org.apache.calcite.test.CalciteAssert.assertQuery(CalciteAssert.java:522)
> at
> org.apache.calcite.test.CalciteAssert$AssertQuery.lambda$returns$1(CalciteAssert.java:1466)
> at
> org.apache.calcite.test.CalciteAssert$AssertQuery.withConnection(CalciteAssert.java:1398)
> at
> org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1464)
> at
> org.apache.calcite.test.CalciteAssert$AssertQuery.explainMatches(CalciteAssert.java:1561)
> at
> org.apache.calcite.test.CalciteAssert$AssertQuery.explainContains(CalciteAssert.java:1556)
> at
> org.apache.calcite.test.JdbcTest.testNotExistsCorrelated(JdbcTest.java:4562)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
>
[jira] [Updated] (CALCITE-3138) Restructuring ROW type fields is not supported
[
https://issues.apache.org/jira/browse/CALCITE-3138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi updated CALCITE-3138:
-
Fix Version/s: 1.21.0
> Restructuring ROW type fields is not supported
> --
>
> Key: CALCITE-3138
> URL: https://issues.apache.org/jira/browse/CALCITE-3138
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Haisheng Yuan
>Assignee: Igor Guzenko
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.21.0
>
> Attachments: ROW_repro.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> 1) RelStructuredTypeFlattener.restructureFields(structType) doesn't support
> ROW type. However, ROW type is flattened by
> RelStructuredTypeFlattener just like struct. So when user queries one column
> with complex type ROW, after flattening and restructuring
> top level project returns reference to first inner primitive field of the
> requested column.
> 2) Another finding is related to ITEM expression applied to array of structs
> column. For example, let's imagine a table with column of
> type ARRAY>>. When user requests is SQL
> array_column[1], Calcite generates ITEM($0, 1) , where $0 is ref to array
> column
> from Scan and 1 is index literal. Current flattener generates two field acess
> expressions ITEM($0, 1).a, ITEM($0, 1).b but dont take into account
> that ITEM($0, 1).b returns struct which also should be flattened.
> 3) In some cases applying of ITEM after flattenning is no longer possible.
> For example, consider column with type
> STRUCT>. User requests column['b'] in query and
> Calcite creates ITEM($0,'b').
> After flattenning Scan is covered by LogicalProject($0.a, $0.b.x, $0.b.y) and
> the old projection ITEM($0,'b') can't
> be applied anymore. So now it should be converted to refer only subset of
> fields ($1,$2) from flattening project.
> UPDATES IN EXPECTED TEST RESULTS:
> --
> TEST CASE: SqlToRelConverterTest.testNestedColumnType()
> {code}
> select empa.home_address.zip from sales.emp_address empa where
> empa.home_address.city = 'abc'
> {code}
> OLD RESULT:
> {code}
> LogicalProject(ZIP=[$4])
> LogicalFilter(condition=[=($3, 'abc':VARCHAR(20))])
> LogicalProject(EMPNO=[$0], STREET=[$1.STREET], CITY=[$1.CITY],
> ZIP=[$1.ZIP], STATE=[$1.STATE], STREET5=[$2.STREET], CITY6=[$2.CITY],
> ZIP7=[$2.ZIP], STATE8=[$2.STATE])
> LogicalTableScan(table=[[CATALOG, SALES, EMP_ADDRESS]])
> {code}
> 1. Above in logical filter, condition references to field $3 which is
> ZIP=[$1.ZIP] field from previous project,
>however in original query filtering should be done by CITY field.
> 2. Also the top level project references to $4 field, which is
> STATE=[$1.STATE] field from project, but original
>query requested ZIP field.
>
> UPDATED RESULT:
> {code}
> LogicalProject(ZIP=[$3])
> LogicalFilter(condition=[=($2, 'abc')])
> LogicalProject(EMPNO=[$0], STREET=[$1.STREET], CITY=[$1.CITY],
> ZIP=[$1.ZIP], STATE=[$1.STATE], STREET5=[$2.STREET], CITY6=[$2.CITY],
> ZIP7=[$2.ZIP], STATE8=[$2.STATE])
> LogicalTableScan(table=[[CATALOG, SALES, EMP_ADDRESS]])
> {code}
> --
> TEST CASE: SqlToRelConverterTest.testStructTypeAlias()
> {code}
> select t.r AS myRow from (select row(row(1)) r from dept) t
> {code}
> OLD RESULT:
> {code}
> LogicalProject(MYROW$$0$$0=[1])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> {code}
> 1. Inside the subselect of row(row(1)) type of returned column is
> RecordType(RecordType(INTEGER EXPR$0) EXPR$0),
>but the top level project uses flattened expression and returns to user
> literal 1 with type RecordType(INTEGER MYROW$$0$$0),
>although the type doesn't match type returned by row(row(1)) expression.
> 2. Also it's suspicious that caller expects returned column to have name
> 'myRow' but gets 'MYROW$$0$$0'.
>
> UPDATED RESULT:
> {code}
> LogicalProject(MYROW=[ROW(ROW($0))])
> LogicalProject(MYROW$$0$$0=[1])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> {code}
> --
> TEST CASE: SqlToRelConverterTest.testFlattenRecords()
> {code}
> select employees[1] from dept_nested
> {code}
> OLD RESULT:
> {code}
> LogicalProject(EXPR$0=[$0])
> LogicalProject(EXPR$0$0=[ITEM($3, 1).EMPNO], EXPR$0$1=[ITEM($3, 1).ENAME],
> EXPR$0$2=[ITEM($3, 1).DETAIL])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT_NESTED]])
> {code}
> 1. Given selection of element by index from array, which by definition
> contain elements of type:
> {code}
> STRUCT
[jira] [Commented] (CALCITE-3244) RelDecorrelator unable to decorrelate expression with filter and aggregate on top
[
https://issues.apache.org/jira/browse/CALCITE-3244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16907182#comment-16907182
]
Volodymyr Vysotskyi commented on CALCITE-3244:
--
[~julianhyde], your assumption about DAG helped to discover one more issue in
{{RelDecorrelator}}. The following test (added {{EXISTS}} clause)
{code:java}
@Test public void testDecorrelateExists() throws Exception {
final String sql = "select \n"
+ " exists (select count(t2.id) \n"
+ "from (values(1), (2)) t2(id) where t2.id = t1.id)\n"
+ "from (values(3), (4)) t1(id)";
checkSubQuery(sql).withLateDecorrelation(true).check();
}
{code}
fails with the error:
{noformat}
java.lang.AssertionError: Cannot add expression of different type to set:
set type is RecordType(INTEGER NOT NULL ID, BOOLEAN i) NOT NULL
expression type is RecordType(INTEGER NOT NULL ID, BOOLEAN NOT NULL i) NOT NULL
set is
rel#44:LogicalCorrelate(left=HepRelVertex#36,right=HepRelVertex#43,correlation=$cor0,joinType=left,requiredColumns={0})
expression is LogicalProject(ID=[$0], i=[true])
LogicalCorrelate(correlation=[$cor0], joinType=[left], requiredColumns=[{0}])
LogicalValues(tuples=[[{ 3 }, { 4 }]])
LogicalAggregate(group=[{}], EXPR$0=[COUNT()])
LogicalFilter(condition=[=($0, $cor0.ID)])
LogicalValues(tuples=[[{ 1 }, { 2 }]])
at
org.apache.calcite.plan.RelOptUtil.verifyTypeEquivalence(RelOptUtil.java:384)
at
org.apache.calcite.plan.hep.HepRuleCall.transformTo(HepRuleCall.java:57)
at
org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:236)
at
org.apache.calcite.sql2rel.RelDecorrelator$AdjustProjectForCountAggregateRule.onMatch2(RelDecorrelator.java:2538)
at
org.apache.calcite.sql2rel.RelDecorrelator$AdjustProjectForCountAggregateRule.onMatch(RelDecorrelator.java:2451)
at
org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:319)
at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:560)
at
org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:419)
at
org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:256)
at
org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:127)
at
org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:215)
at
org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:202)
at
org.apache.calcite.sql2rel.RelDecorrelator.decorrelate(RelDecorrelator.java:249)
at
org.apache.calcite.sql2rel.RelDecorrelator.decorrelateQuery(RelDecorrelator.java:214)
at
org.apache.calcite.test.RelOptTestBase.checkPlanning(RelOptTestBase.java:213)
at
org.apache.calcite.test.RelOptTestBase$Sql.check(RelOptTestBase.java:338)
at
org.apache.calcite.test.RelOptTestBase$Sql.check(RelOptTestBase.java:321)
at
org.apache.calcite.test.RelOptRulesTest.testDecorrelateExists(RelOptRulesTest.java:5565)
{noformat}
Regarding the original query, the plan is not changed.
> RelDecorrelator unable to decorrelate expression with filter and aggregate on
> top
> -
>
> Key: CALCITE-3244
> URL: https://issues.apache.org/jira/browse/CALCITE-3244
> Project: Calcite
> Issue Type: Improvement
>Affects Versions: 1.20.0
>Reporter: benj
>Assignee: Danny Chan
>Priority: Major
>
> Some very useful type of requests currently failed with:
> {code:java}
> SYSTEM ERROR: UnsupportedOperationException: Adding Implicit RowID column is
> not supported for ValuesPrel operator
> {code}
> Examples from DRILL-7050:
> {code:sql}
> select t1.id,
> (select count(t2.id)
> from (
> select 1 as id
> union all
> select 2 as id
> ) t2 where t2.id = t1.id)
> from (
> select 1 as id
> union all
> select 2 as id
> ) t1
> {code}
> {code:sql}
> SELECT t,
> (SELECT count(*) FROM
> (SELECT split(r,' ') AS r FROM
> (SELECT sub.t AS r)) AS x
> ,LATERAL(SELECT $unnest AS u FROM unnest(x.r))
> /* WHERE ... */) t2
> FROM
> (SELECT 'unnest is useful' AS t) sub
> {code}
>
> _Please note that in 1.18 the error for these requests was:_
> {code:java}
> Error: PLAN ERROR: Cannot convert RexNode to equivalent Drill expression.
> RexNode Class: org.apache.calcite.rex.RexCorrelVariable, RexNode Digest: $cor0
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (CALCITE-3115) Cannot add JdbcRules which have different JdbcConvention to same VolcanoPlanner's RuleSet.
[
https://issues.apache.org/jira/browse/CALCITE-3115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16907165#comment-16907165
]
Volodymyr Vysotskyi commented on CALCITE-3115:
--
Other projects including Drill also faced with this issue. It would be good to
include this fix into 1.21.
[~winipanda], are you planning to complete the fix before the release? In the
opposite case, I can pick up this issue.
> Cannot add JdbcRules which have different JdbcConvention to same
> VolcanoPlanner's RuleSet.
> --
>
> Key: CALCITE-3115
> URL: https://issues.apache.org/jira/browse/CALCITE-3115
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.19.0
>Reporter: TANG Wen-hui
>Assignee: TANG Wen-hui
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.21.0
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> When we use Calcite via JDBC to run a sql which involves two difference jdbc
> schema:
> {code:java}
> select * from (select "a",max("b") as max_b, sum("c") as sum_c from
> "test"."temp" where "d" > 10 or "b" <> 'hello' group by "a", "e", "f" having
> "a" > 100 and max("b") < 20 limit 10) t union select "a", "b","c" from
> "test2"."temp2" group by "a","b","c"
> {code}
> the sql get a plan like that:
> {code:java}
> EnumerableUnion(all=[false])
> JdbcToEnumerableConverter
> JdbcProject(a=[$0], MAX_B=[$3], SUM_C=[$4])
> JdbcSort(fetch=[10])
> JdbcFilter(condition=[<(CAST($3):BIGINT, 20)])
> JdbcAggregate(group=[{0, 4, 5}], MAX_B=[MAX($1)], SUM_C=[SUM($2)])
> JdbcFilter(condition=[AND(OR(>($3, 10), <>($1, 'hello')), >($0,
> 100))])
> JdbcTableScan(table=[[test, temp]])
> EnumerableAggregate(group=[{0, 1, 2}])
> JdbcToEnumerableConverter
> JdbcTableScan(table=[[test2, temp2]])
> {code}
> And the EnumerableAggregate for table test2.temp2 cannot be converted to
> JdbcAggregate.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (CALCITE-3115) Cannot add JdbcRules which have different JdbcConvention to same VolcanoPlanner's RuleSet.
[
https://issues.apache.org/jira/browse/CALCITE-3115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi updated CALCITE-3115:
-
Fix Version/s: 1.21.0
> Cannot add JdbcRules which have different JdbcConvention to same
> VolcanoPlanner's RuleSet.
> --
>
> Key: CALCITE-3115
> URL: https://issues.apache.org/jira/browse/CALCITE-3115
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.19.0
>Reporter: TANG Wen-hui
>Assignee: TANG Wen-hui
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.21.0
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> When we use Calcite via JDBC to run a sql which involves two difference jdbc
> schema:
> {code:java}
> select * from (select "a",max("b") as max_b, sum("c") as sum_c from
> "test"."temp" where "d" > 10 or "b" <> 'hello' group by "a", "e", "f" having
> "a" > 100 and max("b") < 20 limit 10) t union select "a", "b","c" from
> "test2"."temp2" group by "a","b","c"
> {code}
> the sql get a plan like that:
> {code:java}
> EnumerableUnion(all=[false])
> JdbcToEnumerableConverter
> JdbcProject(a=[$0], MAX_B=[$3], SUM_C=[$4])
> JdbcSort(fetch=[10])
> JdbcFilter(condition=[<(CAST($3):BIGINT, 20)])
> JdbcAggregate(group=[{0, 4, 5}], MAX_B=[MAX($1)], SUM_C=[SUM($2)])
> JdbcFilter(condition=[AND(OR(>($3, 10), <>($1, 'hello')), >($0,
> 100))])
> JdbcTableScan(table=[[test, temp]])
> EnumerableAggregate(group=[{0, 1, 2}])
> JdbcToEnumerableConverter
> JdbcTableScan(table=[[test2, temp2]])
> {code}
> And the EnumerableAggregate for table test2.temp2 cannot be converted to
> JdbcAggregate.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (CALCITE-3244) RelDecorrelator unable to decorrelate expression with filter and aggregate on top
[
https://issues.apache.org/jira/browse/CALCITE-3244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi updated CALCITE-3244:
-
Summary: RelDecorrelator unable to decorrelate expression with filter and
aggregate on top (was: Decorrelating sub-queries - Adding Implicit RowID
support)
> RelDecorrelator unable to decorrelate expression with filter and aggregate on
> top
> -
>
> Key: CALCITE-3244
> URL: https://issues.apache.org/jira/browse/CALCITE-3244
> Project: Calcite
> Issue Type: Improvement
>Affects Versions: 1.20.0
>Reporter: benj
>Priority: Major
>
> Some very useful type of requests currently failed with:
> {code:java}
> SYSTEM ERROR: UnsupportedOperationException: Adding Implicit RowID column is
> not supported for ValuesPrel operator
> {code}
> Examples from DRILL-7050:
> {code:sql}
> select t1.id,
> (select count(t2.id)
> from (
> select 1 as id
> union all
> select 2 as id
> ) t2 where t2.id = t1.id)
> from (
> select 1 as id
> union all
> select 2 as id
> ) t1
> {code}
> {code:sql}
> SELECT t,
> (SELECT count(*) FROM
> (SELECT split(r,' ') AS r FROM
> (SELECT sub.t AS r)) AS x
> ,LATERAL(SELECT $unnest AS u FROM unnest(x.r))
> /* WHERE ... */) t2
> FROM
> (SELECT 'unnest is useful' AS t) sub
> {code}
>
> _Please note that in 1.18 the error for these requests was:_
> {code:java}
> Error: PLAN ERROR: Cannot convert RexNode to equivalent Drill expression.
> RexNode Class: org.apache.calcite.rex.RexCorrelVariable, RexNode Digest: $cor0
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Comment Edited] (CALCITE-3244) Decorrelating sub-queries - Adding Implicit RowID support
[
https://issues.apache.org/jira/browse/CALCITE-3244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16906038#comment-16906038
]
Volodymyr Vysotskyi edited comment on CALCITE-3244 at 8/13/19 10:22 AM:
Thanks, [~benj641] for reporting this issue and [~julianhyde] for taking an
initial look.
Looks like the root cause why it fails in Drill is connected with the way how
Calcite decorrelates sub-queries of such type.
For the following test:
{code:java}
@Test public void testDecorrelateScalarSubQuery() throws Exception {
final String sql = "select \n"
+ "(select count(t2.id) \n"
+ "from (values(1), (2)) t2(id) where t2.id = t1.id)\n"
+ "from (values(1), (2)) t1(id)";
checkSubQuery(sql).withLateDecorrelation(true).check();
}
{code}
Calcite will generate the following plan with {{LogicalCorrelate}} operator:
{noformat}
LogicalProject(EXPR$0=[$1])
LogicalCorrelate(correlation=[$cor0], joinType=[left], requiredColumns=[{0}])
LogicalValues(tuples=[[{ 1 }, { 2 }]])
LogicalAggregate(group=[{}], EXPR$0=[COUNT()])
LogicalFilter(condition=[=($0, $cor0.ID)])
LogicalValues(tuples=[[{ 1 }, { 2 }]])
{noformat}
But Drill does not support Correlate operator except the case when it
corresponds to lateral unnest.
The issue there is that RelDecorrelator currently cannot rewrite the plan to
remove correlate operator.
was (Author: vvysotskyi):
Thanks, [~benj641] for reporting this issue and [~julianhyde] for taking an
initial look.
Looks like the root cause why it fails in Drill is connected with the way how
Calcite decorrelates sub-queries of such type.
For the following test:
{code:java}
@Test public void testDecorrelateScalarSubQuery() throws Exception {
final String sql = "select \n"
+ "(select count(t2.id) \n"
+ "from (values(1), (2)) t2(id) where t2.id = t1.id)\n"
+ "from (values(1), (2)) t1(id)";
checkSubQuery(sql).withLateDecorrelation(true).check();
}
{code}
Calcite will generate the following plan with {{LogicalCorrelate}} operator:
{noformat}
LogicalProject(EXPR$0=[$1])
LogicalCorrelate(correlation=[$cor0], joinType=[left], requiredColumns=[{0}])
LogicalValues(tuples=[[{ 1 }, { 2 }]])
LogicalAggregate(group=[{}], EXPR$0=[COUNT()])
LogicalFilter(condition=[=($0, $cor0.ID)])
LogicalValues(tuples=[[{ 1 }, { 2 }]])
{noformat}
But Drill does not support Correlate operator except the case when it
corresponds to lateral unnest so it was expected that RelDecorrelator will
rewrite the plan to remove correlate operator.
> Decorrelating sub-queries - Adding Implicit RowID support
> -
>
> Key: CALCITE-3244
> URL: https://issues.apache.org/jira/browse/CALCITE-3244
> Project: Calcite
> Issue Type: Improvement
>Affects Versions: 1.20.0
>Reporter: benj
>Priority: Major
>
> Some very useful type of requests currently failed with:
> {code:java}
> SYSTEM ERROR: UnsupportedOperationException: Adding Implicit RowID column is
> not supported for ValuesPrel operator
> {code}
> Examples from DRILL-7050:
> {code:sql}
> select t1.id,
> (select count(t2.id)
> from (
> select 1 as id
> union all
> select 2 as id
> ) t2 where t2.id = t1.id)
> from (
> select 1 as id
> union all
> select 2 as id
> ) t1
> {code}
> {code:sql}
> SELECT t,
> (SELECT count(*) FROM
> (SELECT split(r,' ') AS r FROM
> (SELECT sub.t AS r)) AS x
> ,LATERAL(SELECT $unnest AS u FROM unnest(x.r))
> /* WHERE ... */) t2
> FROM
> (SELECT 'unnest is useful' AS t) sub
> {code}
>
> _Please note that in 1.18 the error for these requests was:_
> {code:java}
> Error: PLAN ERROR: Cannot convert RexNode to equivalent Drill expression.
> RexNode Class: org.apache.calcite.rex.RexCorrelVariable, RexNode Digest: $cor0
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (CALCITE-3244) Decorrelating sub-queries - Adding Implicit RowID support
[
https://issues.apache.org/jira/browse/CALCITE-3244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16906038#comment-16906038
]
Volodymyr Vysotskyi commented on CALCITE-3244:
--
Thanks, [~benj641] for reporting this issue and [~julianhyde] for taking an
initial look.
Looks like the root cause why it fails in Drill is connected with the way how
Calcite decorrelates sub-queries of such type.
For the following test:
{code:java}
@Test public void testDecorrelateScalarSubQuery() throws Exception {
final String sql =
"select \n"
+ "(select count(t2.id) \n"
+ "from (values(1), (2)) t2(id) where t2.id = t1.id)\n"
+ "from (values(1), (2)) t1(id)";
checkSubQuery(sql).withLateDecorrelation(true).check();
}
{code}
Calcite will generate the following plan with {{LogicalCorrelate}} operator:
{noformat}
LogicalProject(EXPR$0=[$1])
LogicalCorrelate(correlation=[$cor0], joinType=[left], requiredColumns=[{0}])
LogicalValues(tuples=[[{ 1 }, { 2 }]])
LogicalAggregate(group=[{}], EXPR$0=[COUNT()])
LogicalFilter(condition=[=($0, $cor0.ID)])
LogicalValues(tuples=[[{ 1 }, { 2 }]])
{noformat}
But Drill does not support Correlate operator except the case when it
corresponds to lateral unnest so it was expected that RelDecorrelator will
rewrite the plan to remove correlate operator.
> Decorrelating sub-queries - Adding Implicit RowID support
> -
>
> Key: CALCITE-3244
> URL: https://issues.apache.org/jira/browse/CALCITE-3244
> Project: Calcite
> Issue Type: Improvement
>Affects Versions: 1.20.0
>Reporter: benj
>Priority: Major
>
> Some very useful type of requests currently failed with:
> {code:java}
> SYSTEM ERROR: UnsupportedOperationException: Adding Implicit RowID column is
> not supported for ValuesPrel operator
> {code}
> Examples from DRILL-7050:
> {code:sql}
> select t1.id,
> (select count(t2.id)
> from (
> select 1 as id
> union all
> select 2 as id
> ) t2 where t2.id = t1.id)
> from (
> select 1 as id
> union all
> select 2 as id
> ) t1
> {code}
> {code:sql}
> SELECT t,
> (SELECT count(*) FROM
> (SELECT split(r,' ') AS r FROM
> (SELECT sub.t AS r)) AS x
> ,LATERAL(SELECT $unnest AS u FROM unnest(x.r))
> /* WHERE ... */) t2
> FROM
> (SELECT 'unnest is useful' AS t) sub
> {code}
>
> _Please note that in 1.18 the error for these requests was:_
> {code:java}
> Error: PLAN ERROR: Cannot convert RexNode to equivalent Drill expression.
> RexNode Class: org.apache.calcite.rex.RexCorrelVariable, RexNode Digest: $cor0
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Comment Edited] (CALCITE-3244) Decorrelating sub-queries - Adding Implicit RowID support
[
https://issues.apache.org/jira/browse/CALCITE-3244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16906038#comment-16906038
]
Volodymyr Vysotskyi edited comment on CALCITE-3244 at 8/13/19 10:10 AM:
Thanks, [~benj641] for reporting this issue and [~julianhyde] for taking an
initial look.
Looks like the root cause why it fails in Drill is connected with the way how
Calcite decorrelates sub-queries of such type.
For the following test:
{code:java}
@Test public void testDecorrelateScalarSubQuery() throws Exception {
final String sql = "select \n"
+ "(select count(t2.id) \n"
+ "from (values(1), (2)) t2(id) where t2.id = t1.id)\n"
+ "from (values(1), (2)) t1(id)";
checkSubQuery(sql).withLateDecorrelation(true).check();
}
{code}
Calcite will generate the following plan with {{LogicalCorrelate}} operator:
{noformat}
LogicalProject(EXPR$0=[$1])
LogicalCorrelate(correlation=[$cor0], joinType=[left], requiredColumns=[{0}])
LogicalValues(tuples=[[{ 1 }, { 2 }]])
LogicalAggregate(group=[{}], EXPR$0=[COUNT()])
LogicalFilter(condition=[=($0, $cor0.ID)])
LogicalValues(tuples=[[{ 1 }, { 2 }]])
{noformat}
But Drill does not support Correlate operator except the case when it
corresponds to lateral unnest so it was expected that RelDecorrelator will
rewrite the plan to remove correlate operator.
was (Author: vvysotskyi):
Thanks, [~benj641] for reporting this issue and [~julianhyde] for taking an
initial look.
Looks like the root cause why it fails in Drill is connected with the way how
Calcite decorrelates sub-queries of such type.
For the following test:
{code:java}
@Test public void testDecorrelateScalarSubQuery() throws Exception {
final String sql =
"select \n"
+ "(select count(t2.id) \n"
+ "from (values(1), (2)) t2(id) where t2.id = t1.id)\n"
+ "from (values(1), (2)) t1(id)";
checkSubQuery(sql).withLateDecorrelation(true).check();
}
{code}
Calcite will generate the following plan with {{LogicalCorrelate}} operator:
{noformat}
LogicalProject(EXPR$0=[$1])
LogicalCorrelate(correlation=[$cor0], joinType=[left], requiredColumns=[{0}])
LogicalValues(tuples=[[{ 1 }, { 2 }]])
LogicalAggregate(group=[{}], EXPR$0=[COUNT()])
LogicalFilter(condition=[=($0, $cor0.ID)])
LogicalValues(tuples=[[{ 1 }, { 2 }]])
{noformat}
But Drill does not support Correlate operator except the case when it
corresponds to lateral unnest so it was expected that RelDecorrelator will
rewrite the plan to remove correlate operator.
> Decorrelating sub-queries - Adding Implicit RowID support
> -
>
> Key: CALCITE-3244
> URL: https://issues.apache.org/jira/browse/CALCITE-3244
> Project: Calcite
> Issue Type: Improvement
>Affects Versions: 1.20.0
>Reporter: benj
>Priority: Major
>
> Some very useful type of requests currently failed with:
> {code:java}
> SYSTEM ERROR: UnsupportedOperationException: Adding Implicit RowID column is
> not supported for ValuesPrel operator
> {code}
> Examples from DRILL-7050:
> {code:sql}
> select t1.id,
> (select count(t2.id)
> from (
> select 1 as id
> union all
> select 2 as id
> ) t2 where t2.id = t1.id)
> from (
> select 1 as id
> union all
> select 2 as id
> ) t1
> {code}
> {code:sql}
> SELECT t,
> (SELECT count(*) FROM
> (SELECT split(r,' ') AS r FROM
> (SELECT sub.t AS r)) AS x
> ,LATERAL(SELECT $unnest AS u FROM unnest(x.r))
> /* WHERE ... */) t2
> FROM
> (SELECT 'unnest is useful' AS t) sub
> {code}
>
> _Please note that in 1.18 the error for these requests was:_
> {code:java}
> Error: PLAN ERROR: Cannot convert RexNode to equivalent Drill expression.
> RexNode Class: org.apache.calcite.rex.RexCorrelVariable, RexNode Digest: $cor0
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (CALCITE-3121) VolcanoPlanner hangs due to removing ORDER BY from sub-query
[
https://issues.apache.org/jira/browse/CALCITE-3121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16863367#comment-16863367
]
Volodymyr Vysotskyi commented on CALCITE-3121:
--
[~julianhyde], can I merge this PR to include it into the release?
> VolcanoPlanner hangs due to removing ORDER BY from sub-query
>
>
> Key: CALCITE-3121
> URL: https://issues.apache.org/jira/browse/CALCITE-3121
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.19.0
>Reporter: Bohdan Kazydub
>Assignee: Danny Chan
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> After the fix for CALCITE-2798 some queries hang during planning in
> VolcanoPlanner (similar issue was reported in CALCITE-2223).
> Here is a test case which should be added to the {{RelOptRulesTest}} class:
> {code:java}
> @Test public void testSubQueryWithOrderByHang() {
> String sql = "select n.n_regionkey from ( select * from "
> + "( select * from sales.customer) t order by t.n_regionkey) n where
> n.n_nationkey >1 ";
> VolcanoPlanner planner = new VolcanoPlanner(null, null);
> planner.addRelTraitDef(ConventionTraitDef.INSTANCE);
> Tester dynamicTester = createDynamicTester().withDecorrelation(true)
> .withClusterFactory(
> relOptCluster -> RelOptCluster.create(planner,
> relOptCluster.getRexBuilder()));
> RelRoot root = dynamicTester.convertSqlToRel(sql);
> String planBefore = NL + RelOptUtil.toString(root.rel);
> getDiffRepos().assertEquals("planBefore", "${planBefore}", planBefore);
> PushProjector.ExprCondition exprCondition = expr -> {
> if (expr instanceof RexCall) {
> RexCall call = (RexCall) expr;
> return
> "item".equals(call.getOperator().getName().toLowerCase(Locale.ROOT));
> }
> return false;
> };
> RuleSet ruleSet =
> RuleSets.ofList(
> FilterProjectTransposeRule.INSTANCE,
> FilterMergeRule.INSTANCE,
> ProjectMergeRule.INSTANCE,
> new ProjectFilterTransposeRule(Project.class, Filter .class,
> RelFactories.LOGICAL_BUILDER, exprCondition),
> EnumerableRules.ENUMERABLE_PROJECT_RULE,
> EnumerableRules.ENUMERABLE_FILTER_RULE,
> EnumerableRules.ENUMERABLE_SORT_RULE,
> EnumerableRules.ENUMERABLE_LIMIT_RULE,
> EnumerableRules.ENUMERABLE_TABLE_SCAN_RULE);
> Program program = Programs.of(ruleSet);
> RelTraitSet toTraits =
> root.rel.getCluster().traitSet()
> .replace(0, EnumerableConvention.INSTANCE);
> RelNode relAfter = program.run(planner, root.rel, toTraits,
> Collections.emptyList(), Collections.emptyList());
> String planAfter = NL + RelOptUtil.toString(relAfter);
> getDiffRepos().assertEquals("planAfter", "${planAfter}", planAfter);
> }
> {code}
> Please note that if {{LIMIT }} is added to the sub-query with order by
> (so it is not removed due to the fix for CALCITE-2798) the test succeeds.
> Though the issue with hanging is more general, I think that if it wouldn't be
> fixed, the fix for CALCITE-2798 should be reverted to reduce cases when
> planner may hang.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-3121) VolcanoPlanner hangs due to removing ORDER BY from sub-query
[
https://issues.apache.org/jira/browse/CALCITE-3121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16861962#comment-16861962
]
Volodymyr Vysotskyi commented on CALCITE-3121:
--
The aim of CALCITE-2798 was to optimize queries by removing unnecessary ORDER
BY clauses, but as was mentioned above, it caused the hanging of some queries
(due to another, more general issue).
>From the functionality perspective, this commit caused a regression, and UT
>attached to the Jira proves that, so the behavior without the fix for
>CALCITE-2798 was better - volcano planner was able to plan some queries which
>hang now, I think it is more important than the possibility to produce
>optimization.
The ideal solution is to fix the original issue which causes hanging with or
without ORDER BY clauses in sub-queries (perhaps it is the same as
CALCITE-2223), but taking into account upcoming release and its timings, revert
is an acceptable solution from my point of view.
> VolcanoPlanner hangs due to removing ORDER BY from sub-query
>
>
> Key: CALCITE-3121
> URL: https://issues.apache.org/jira/browse/CALCITE-3121
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.19.0
>Reporter: Bohdan Kazydub
>Assignee: Danny Chan
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> After the fix for CALCITE-2798 some queries hang during planning in
> VolcanoPlanner (similar issue was reported in CALCITE-2223).
> Here is a test case which should be added to the {{RelOptRulesTest}} class:
> {code:java}
> @Test public void testSubQueryWithOrderByHang() {
> String sql = "select n.n_regionkey from ( select * from "
> + "( select * from sales.customer) t order by t.n_regionkey) n where
> n.n_nationkey >1 ";
> VolcanoPlanner planner = new VolcanoPlanner(null, null);
> planner.addRelTraitDef(ConventionTraitDef.INSTANCE);
> Tester dynamicTester = createDynamicTester().withDecorrelation(true)
> .withClusterFactory(
> relOptCluster -> RelOptCluster.create(planner,
> relOptCluster.getRexBuilder()));
> RelRoot root = dynamicTester.convertSqlToRel(sql);
> String planBefore = NL + RelOptUtil.toString(root.rel);
> getDiffRepos().assertEquals("planBefore", "${planBefore}", planBefore);
> PushProjector.ExprCondition exprCondition = expr -> {
> if (expr instanceof RexCall) {
> RexCall call = (RexCall) expr;
> return
> "item".equals(call.getOperator().getName().toLowerCase(Locale.ROOT));
> }
> return false;
> };
> RuleSet ruleSet =
> RuleSets.ofList(
> FilterProjectTransposeRule.INSTANCE,
> FilterMergeRule.INSTANCE,
> ProjectMergeRule.INSTANCE,
> new ProjectFilterTransposeRule(Project.class, Filter .class,
> RelFactories.LOGICAL_BUILDER, exprCondition),
> EnumerableRules.ENUMERABLE_PROJECT_RULE,
> EnumerableRules.ENUMERABLE_FILTER_RULE,
> EnumerableRules.ENUMERABLE_SORT_RULE,
> EnumerableRules.ENUMERABLE_LIMIT_RULE,
> EnumerableRules.ENUMERABLE_TABLE_SCAN_RULE);
> Program program = Programs.of(ruleSet);
> RelTraitSet toTraits =
> root.rel.getCluster().traitSet()
> .replace(0, EnumerableConvention.INSTANCE);
> RelNode relAfter = program.run(planner, root.rel, toTraits,
> Collections.emptyList(), Collections.emptyList());
> String planAfter = NL + RelOptUtil.toString(relAfter);
> getDiffRepos().assertEquals("planAfter", "${planAfter}", planAfter);
> }
> {code}
> Please note that if {{LIMIT }} is added to the sub-query with order by
> (so it is not removed due to the fix for CALCITE-2798) the test succeeds.
> Though the issue with hanging is more general, I think that if it wouldn't be
> fixed, the fix for CALCITE-2798 should be reverted to reduce cases when
> planner may hang.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-3121) VolcanoPlanner hangs due to removing ORDER BY from sub-query
[
https://issues.apache.org/jira/browse/CALCITE-3121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16861249#comment-16861249
]
Volodymyr Vysotskyi commented on CALCITE-3121:
--
If there are no objections, I'll merge the PR soon.
> VolcanoPlanner hangs due to removing ORDER BY from sub-query
>
>
> Key: CALCITE-3121
> URL: https://issues.apache.org/jira/browse/CALCITE-3121
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.19.0
>Reporter: Bohdan Kazydub
>Assignee: Danny Chan
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> After the fix for CALCITE-2798 some queries hang during planning in
> VolcanoPlanner (similar issue was reported in CALCITE-2223).
> Here is a test case which should be added to the {{RelOptRulesTest}} class:
> {code:java}
> @Test public void testSubQueryWithOrderByHang() {
> String sql = "select n.n_regionkey from ( select * from "
> + "( select * from sales.customer) t order by t.n_regionkey) n where
> n.n_nationkey >1 ";
> VolcanoPlanner planner = new VolcanoPlanner(null, null);
> planner.addRelTraitDef(ConventionTraitDef.INSTANCE);
> Tester dynamicTester = createDynamicTester().withDecorrelation(true)
> .withClusterFactory(
> relOptCluster -> RelOptCluster.create(planner,
> relOptCluster.getRexBuilder()));
> RelRoot root = dynamicTester.convertSqlToRel(sql);
> String planBefore = NL + RelOptUtil.toString(root.rel);
> getDiffRepos().assertEquals("planBefore", "${planBefore}", planBefore);
> PushProjector.ExprCondition exprCondition = expr -> {
> if (expr instanceof RexCall) {
> RexCall call = (RexCall) expr;
> return
> "item".equals(call.getOperator().getName().toLowerCase(Locale.ROOT));
> }
> return false;
> };
> RuleSet ruleSet =
> RuleSets.ofList(
> FilterProjectTransposeRule.INSTANCE,
> FilterMergeRule.INSTANCE,
> ProjectMergeRule.INSTANCE,
> new ProjectFilterTransposeRule(Project.class, Filter .class,
> RelFactories.LOGICAL_BUILDER, exprCondition),
> EnumerableRules.ENUMERABLE_PROJECT_RULE,
> EnumerableRules.ENUMERABLE_FILTER_RULE,
> EnumerableRules.ENUMERABLE_SORT_RULE,
> EnumerableRules.ENUMERABLE_LIMIT_RULE,
> EnumerableRules.ENUMERABLE_TABLE_SCAN_RULE);
> Program program = Programs.of(ruleSet);
> RelTraitSet toTraits =
> root.rel.getCluster().traitSet()
> .replace(0, EnumerableConvention.INSTANCE);
> RelNode relAfter = program.run(planner, root.rel, toTraits,
> Collections.emptyList(), Collections.emptyList());
> String planAfter = NL + RelOptUtil.toString(relAfter);
> getDiffRepos().assertEquals("planAfter", "${planAfter}", planAfter);
> }
> {code}
> Please note that if {{LIMIT }} is added to the sub-query with order by
> (so it is not removed due to the fix for CALCITE-2798) the test succeeds.
> Though the issue with hanging is more general, I think that if it wouldn't be
> fixed, the fix for CALCITE-2798 should be reverted to reduce cases when
> planner may hang.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (CALCITE-3061) Query with WITH clause fails when alias is the same as the table with rolled up column
[
https://issues.apache.org/jira/browse/CALCITE-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi resolved CALCITE-3061.
--
Resolution: Fixed
Fixed in
[d0a06f7|https://github.com/apache/calcite/commit/d0a06f78644906c2ca73e6fb5ea08f7d17f55dab].
Thanks [~julianhyde] and [~danny0405] for the review!
> Query with WITH clause fails when alias is the same as the table with rolled
> up column
> --
>
> Key: CALCITE-3061
> URL: https://issues.apache.org/jira/browse/CALCITE-3061
> Project: Calcite
> Issue Type: Bug
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.20.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> {{SqlValidatorImpl}} incorrectly resolves aliases for CTE when checking
> rolled up columns when a table with the same name is present.
> It may be observed with the following case: we have table {{emp_r}} which
> contains rolled up column {{slackingmin}}. The following query:
> {code:sql}
> with emp_r as (select 1 as slackingmin) select slackingmin from emp_r
> {code}
> should pass since it uses CTE, not table.
> Error for this case:
> {noformat}
> org.apache.calcite.sql.validate.SqlValidatorException: Rolled up column
> 'SLACKINGMIN' is not allowed in SELECT
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:463)
> at org.apache.calcite.runtime.Resources$ExInst.ex(Resources.java:572)
> at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:787)
> at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:772)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError(SqlValidatorImpl.java:4825)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.checkRollUp(SqlValidatorImpl.java:3438)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.checkRollUp(SqlValidatorImpl.java:3447)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.checkRollUpInSelectList(SqlValidatorImpl.java:3373)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3361)
> at
> org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:994)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:954)
> at
> org.apache.calcite.sql.validate.WithNamespace.validateImpl(WithNamespace.java:57)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:994)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateWith(SqlValidatorImpl.java:3787)
> at org.apache.calcite.sql.SqlWith.validate(SqlWith.java:71)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:929)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:633)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-3061) Query with WITH clause fails when alias is the same as the table with rolled up column
[
https://issues.apache.org/jira/browse/CALCITE-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16838043#comment-16838043
]
Volodymyr Vysotskyi commented on CALCITE-3061:
--
Thanks for the review, I have made requested changes. Agree that with the
inlined {{findTable}} method code looks clearer.
If the code looks ok, I'll merge it soon.
> Query with WITH clause fails when alias is the same as the table with rolled
> up column
> --
>
> Key: CALCITE-3061
> URL: https://issues.apache.org/jira/browse/CALCITE-3061
> Project: Calcite
> Issue Type: Bug
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.20.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{SqlValidatorImpl}} incorrectly resolves aliases for CTE when checking
> rolled up columns when a table with the same name is present.
> It may be observed with the following case: we have table {{emp_r}} which
> contains rolled up column {{slackingmin}}. The following query:
> {code:sql}
> with emp_r as (select 1 as slackingmin) select slackingmin from emp_r
> {code}
> should pass since it uses CTE, not table.
> Error for this case:
> {noformat}
> org.apache.calcite.sql.validate.SqlValidatorException: Rolled up column
> 'SLACKINGMIN' is not allowed in SELECT
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:463)
> at org.apache.calcite.runtime.Resources$ExInst.ex(Resources.java:572)
> at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:787)
> at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:772)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError(SqlValidatorImpl.java:4825)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.checkRollUp(SqlValidatorImpl.java:3438)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.checkRollUp(SqlValidatorImpl.java:3447)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.checkRollUpInSelectList(SqlValidatorImpl.java:3373)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3361)
> at
> org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:994)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:954)
> at
> org.apache.calcite.sql.validate.WithNamespace.validateImpl(WithNamespace.java:57)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:994)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateWith(SqlValidatorImpl.java:3787)
> at org.apache.calcite.sql.SqlWith.validate(SqlWith.java:71)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:929)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:633)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-3061) Query with WITH clause fails when alias is the same as the table with rolled up column
[
https://issues.apache.org/jira/browse/CALCITE-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837452#comment-16837452
]
Volodymyr Vysotskyi commented on CALCITE-3061:
--
I have created PR with the fix: [https://github.com/apache/calcite/pull/1206]
could someone please take a look?
> Query with WITH clause fails when alias is the same as the table with rolled
> up column
> --
>
> Key: CALCITE-3061
> URL: https://issues.apache.org/jira/browse/CALCITE-3061
> Project: Calcite
> Issue Type: Bug
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.20.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{SqlValidatorImpl}} incorrectly resolves aliases for CTE when checking
> rolled up columns when a table with the same name is present.
> It may be observed with the following case: we have table {{emp_r}} which
> contains rolled up column {{slackingmin}}. The following query:
> {code:sql}
> with emp_r as (select 1 as slackingmin) select slackingmin from emp_r
> {code}
> should pass since it uses CTE, not table.
> Error for this case:
> {noformat}
> org.apache.calcite.sql.validate.SqlValidatorException: Rolled up column
> 'SLACKINGMIN' is not allowed in SELECT
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:463)
> at org.apache.calcite.runtime.Resources$ExInst.ex(Resources.java:572)
> at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:787)
> at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:772)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError(SqlValidatorImpl.java:4825)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.checkRollUp(SqlValidatorImpl.java:3438)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.checkRollUp(SqlValidatorImpl.java:3447)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.checkRollUpInSelectList(SqlValidatorImpl.java:3373)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3361)
> at
> org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:994)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:954)
> at
> org.apache.calcite.sql.validate.WithNamespace.validateImpl(WithNamespace.java:57)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:994)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateWith(SqlValidatorImpl.java:3787)
> at org.apache.calcite.sql.SqlWith.validate(SqlWith.java:71)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:929)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:633)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CALCITE-3061) Query with WITH clause fails when alias is the same as the table with rolled up column
Volodymyr Vysotskyi created CALCITE-3061:
Summary: Query with WITH clause fails when alias is the same as
the table with rolled up column
Key: CALCITE-3061
URL: https://issues.apache.org/jira/browse/CALCITE-3061
Project: Calcite
Issue Type: Bug
Reporter: Volodymyr Vysotskyi
Assignee: Volodymyr Vysotskyi
Fix For: 1.20.0
{{SqlValidatorImpl}} incorrectly resolves aliases for CTE when checking rolled
up columns when a table with the same name is present.
It may be observed with the following case: we have table {{emp_r}} which
contains rolled up column {{slackingmin}}. The following query:
{code:sql}
with emp_r as (select 1 as slackingmin) select slackingmin from emp_r
{code}
should pass since it uses CTE, not table.
Error for this case:
{noformat}
org.apache.calcite.sql.validate.SqlValidatorException: Rolled up column
'SLACKINGMIN' is not allowed in SELECT
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:463)
at org.apache.calcite.runtime.Resources$ExInst.ex(Resources.java:572)
at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:787)
at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:772)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError(SqlValidatorImpl.java:4825)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.checkRollUp(SqlValidatorImpl.java:3438)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.checkRollUp(SqlValidatorImpl.java:3447)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.checkRollUpInSelectList(SqlValidatorImpl.java:3373)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3361)
at
org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60)
at
org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:994)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:954)
at
org.apache.calcite.sql.validate.WithNamespace.validateImpl(WithNamespace.java:57)
at
org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:994)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateWith(SqlValidatorImpl.java:3787)
at org.apache.calcite.sql.SqlWith.validate(SqlWith.java:71)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:929)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:633)
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (CALCITE-3055) TableScan type mismatch in VolcanoPlanner
[
https://issues.apache.org/jira/browse/CALCITE-3055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi updated CALCITE-3055:
-
Fix Version/s: 1.20.0
> TableScan type mismatch in VolcanoPlanner
> -
>
> Key: CALCITE-3055
> URL: https://issues.apache.org/jira/browse/CALCITE-3055
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.19.0
>Reporter: Bohdan Kazydub
>Assignee: Danny Chan
>Priority: Blocker
> Fix For: 1.20.0
>
>
> In Drill with Calcite updated to 1.19.0 we observe following problem:
> the following query
> {code}
> CREATE VIEW nation_view_testunionall_expandable_star AS select n_name,
> n_nationkey FROM cp.`tpch/nation.parquet`;
> (SELECT n_comment, n_regionkey FROM cp.`tpch/nation.parquet` LIMIT 5) UNION
> ALL (SELECT * FROM nation_view_testunionall_expandable_star LIMIT 5)
> {code}
> fails with
> {code}
> 15:46:06.005 [2333c7f1-89f3-0b80-8369-68e24895729e:foreman] ERROR
> o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR: AssertionError: Type
> mismatch:
> left:
> RecordType(DYNAMIC_STAR **, ANY n_comment, ANY n_regionkey) NOT NULL
> right:
> RecordType(DYNAMIC_STAR **, ANY n_comment, ANY n_regionkey, ANY n_name, ANY
> n_nationkey) NOT NULL
> {code}
> when Volcano planner is registering rel nodes. The initial plan is the
> following:
> {code}
> LogicalUnion(all=[true]): rowcount = 10.0, cumulative cost = \{520.0 rows,
> 940.755032994728 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 74
> LogicalSort(fetch=[5]): rowcount = 5.0, cumulative cost = \{205.0 rows,
> 365.377516497364 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 68
> LogicalProject(n_comment=[$1], n_regionkey=[$2]): rowcount = 100.0,
> cumulative cost = \{200.0 rows, 301.0 cpu, 0.0 io, 0.0 network, 0.0 memory},
> id = 67
> EnumerableTableScan(table=[[cp, tpch/nation.parquet]]): rowcount =
> 100.0, cumulative cost = \{100.0 rows, 101.0 cpu, 0.0 io, 0.0 network, 0.0
> memory}, id = 29
> LogicalSort(fetch=[5]): rowcount = 5.0, cumulative cost = \{305.0 rows,
> 565.377516497364 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 73
> LogicalProject(n_name=[$0], n_nationkey=[$1]): rowcount = 100.0,
> cumulative cost = \{300.0 rows, 501.0 cpu, 0.0 io, 0.0 network, 0.0 memory},
> id = 72
> LogicalProject(n_name=[$3], n_nationkey=[$4]): rowcount = 100.0,
> cumulative cost = \{200.0 rows, 301.0 cpu, 0.0 io, 0.0 network, 0.0 memory},
> id = 71
> EnumerableTableScan(table=[[cp, tpch/nation.parquet]]): rowcount =
> 100.0, cumulative cost = \{100.0 rows, 101.0 cpu, 0.0 io, 0.0 network, 0.0
> memory}, id = 34
> {code}
> It contains two EnumerableTableScan operators with different row types
> specified in the error message above. In the scope of CALCITE-2454 was
> removed rel data type from the key in the \{{VolcanoPlanner.mapDigestToRel}}
> map.
> To fix this, `{{TableScan` should include rowType}} item in
> {{AbstractRelNode#explainTerms(RelWriter)}} method because in case of dynamic
> tables there is the same digest for different operators. The same thing was
> done for Values operator and literals in conditions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (CALCITE-3055) TableScan type mismatch in VolcanoPlanner
[
https://issues.apache.org/jira/browse/CALCITE-3055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi updated CALCITE-3055:
-
Priority: Blocker (was: Major)
> TableScan type mismatch in VolcanoPlanner
> -
>
> Key: CALCITE-3055
> URL: https://issues.apache.org/jira/browse/CALCITE-3055
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.19.0
>Reporter: Bohdan Kazydub
>Assignee: Danny Chan
>Priority: Blocker
>
> In Drill with Calcite updated to 1.19.0 we observe following problem:
> the following query
> {code}
> CREATE VIEW nation_view_testunionall_expandable_star AS select n_name,
> n_nationkey FROM cp.`tpch/nation.parquet`;
> (SELECT n_comment, n_regionkey FROM cp.`tpch/nation.parquet` LIMIT 5) UNION
> ALL (SELECT * FROM nation_view_testunionall_expandable_star LIMIT 5)
> {code}
> fails with
> {code}
> 15:46:06.005 [2333c7f1-89f3-0b80-8369-68e24895729e:foreman] ERROR
> o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR: AssertionError: Type
> mismatch:
> left:
> RecordType(DYNAMIC_STAR **, ANY n_comment, ANY n_regionkey) NOT NULL
> right:
> RecordType(DYNAMIC_STAR **, ANY n_comment, ANY n_regionkey, ANY n_name, ANY
> n_nationkey) NOT NULL
> {code}
> when Volcano planner is registering rel nodes. The initial plan is the
> following:
> {code}
> LogicalUnion(all=[true]): rowcount = 10.0, cumulative cost = \{520.0 rows,
> 940.755032994728 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 74
> LogicalSort(fetch=[5]): rowcount = 5.0, cumulative cost = \{205.0 rows,
> 365.377516497364 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 68
> LogicalProject(n_comment=[$1], n_regionkey=[$2]): rowcount = 100.0,
> cumulative cost = \{200.0 rows, 301.0 cpu, 0.0 io, 0.0 network, 0.0 memory},
> id = 67
> EnumerableTableScan(table=[[cp, tpch/nation.parquet]]): rowcount =
> 100.0, cumulative cost = \{100.0 rows, 101.0 cpu, 0.0 io, 0.0 network, 0.0
> memory}, id = 29
> LogicalSort(fetch=[5]): rowcount = 5.0, cumulative cost = \{305.0 rows,
> 565.377516497364 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 73
> LogicalProject(n_name=[$0], n_nationkey=[$1]): rowcount = 100.0,
> cumulative cost = \{300.0 rows, 501.0 cpu, 0.0 io, 0.0 network, 0.0 memory},
> id = 72
> LogicalProject(n_name=[$3], n_nationkey=[$4]): rowcount = 100.0,
> cumulative cost = \{200.0 rows, 301.0 cpu, 0.0 io, 0.0 network, 0.0 memory},
> id = 71
> EnumerableTableScan(table=[[cp, tpch/nation.parquet]]): rowcount =
> 100.0, cumulative cost = \{100.0 rows, 101.0 cpu, 0.0 io, 0.0 network, 0.0
> memory}, id = 34
> {code}
> It contains two EnumerableTableScan operators with different row types
> specified in the error message above. In the scope of CALCITE-2454 was
> removed rel data type from the key in the \{{VolcanoPlanner.mapDigestToRel}}
> map.
> To fix this, `{{TableScan` should include rowType}} item in
> {{AbstractRelNode#explainTerms(RelWriter)}} method because in case of dynamic
> tables there is the same digest for different operators. The same thing was
> done for Values operator and literals in conditions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-3055) TableScan type mismatch in VolcanoPlanner
[
https://issues.apache.org/jira/browse/CALCITE-3055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836665#comment-16836665
]
Volodymyr Vysotskyi commented on CALCITE-3055:
--
[~KazydubB], I was able to reproduce this issue in Calcite only. Here is the
branch with the test: [https://github.com/vvysotskyi/calcite/tree/CALCITE-3055].
Except for the test case itself, was added testing code for expanding views
and modified testing code connected with dynamic tables to reflect real use
case (dynamic table cannot be {{RelOptTable}} since the last one is used during
re nodes creation, but at that time it is forbidden to update row type
dynamically).
[~julianhyde], that's absolutely correct that Drill depends deeply on Calcite
internals, and it is the root cause why it is sometimes is not tested on the
release candidates. AFAIK [~KazydubB] started updating Calcite before the
release, but due to some breaking Calcite changes, a lot of unit tests in Drill
failed, and he investigated whether the reason was in Drill or Calcite.
Regarding the proposed fix, the discussion was started in CALCITE-3020, and
there were provided arguments in favor of adding row type to the description.
Except for this change, another possible way for resolving this issue is
reverting a part of the fix for CALCITE-2454 where {{Pair.of(digest,
rel.getRowType())}} usage was replaced with {{digest}} only.
Do you have other proposals on how to resolve this issue?
Besides Drill, this change (CALCITE-2454) also breaks other projects (perhaps
Flink in CALCITE-3020), so I think that we should treat this Jira and
CALCITE-3020 as blockers for the upcoming release.
> TableScan type mismatch in VolcanoPlanner
> -
>
> Key: CALCITE-3055
> URL: https://issues.apache.org/jira/browse/CALCITE-3055
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.19.0
>Reporter: Bohdan Kazydub
>Assignee: Danny Chan
>Priority: Major
>
> In Drill with Calcite updated to 1.19.0 we observe following problem:
> the following query
> {code}
> CREATE VIEW nation_view_testunionall_expandable_star AS select n_name,
> n_nationkey FROM cp.`tpch/nation.parquet`;
> (SELECT n_comment, n_regionkey FROM cp.`tpch/nation.parquet` LIMIT 5) UNION
> ALL (SELECT * FROM nation_view_testunionall_expandable_star LIMIT 5)
> {code}
> fails with
> {code}
> 15:46:06.005 [2333c7f1-89f3-0b80-8369-68e24895729e:foreman] ERROR
> o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR: AssertionError: Type
> mismatch:
> left:
> RecordType(DYNAMIC_STAR **, ANY n_comment, ANY n_regionkey) NOT NULL
> right:
> RecordType(DYNAMIC_STAR **, ANY n_comment, ANY n_regionkey, ANY n_name, ANY
> n_nationkey) NOT NULL
> {code}
> when Volcano planner is registering rel nodes. The initial plan is the
> following:
> {code}
> LogicalUnion(all=[true]): rowcount = 10.0, cumulative cost = \{520.0 rows,
> 940.755032994728 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 74
> LogicalSort(fetch=[5]): rowcount = 5.0, cumulative cost = \{205.0 rows,
> 365.377516497364 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 68
> LogicalProject(n_comment=[$1], n_regionkey=[$2]): rowcount = 100.0,
> cumulative cost = \{200.0 rows, 301.0 cpu, 0.0 io, 0.0 network, 0.0 memory},
> id = 67
> EnumerableTableScan(table=[[cp, tpch/nation.parquet]]): rowcount =
> 100.0, cumulative cost = \{100.0 rows, 101.0 cpu, 0.0 io, 0.0 network, 0.0
> memory}, id = 29
> LogicalSort(fetch=[5]): rowcount = 5.0, cumulative cost = \{305.0 rows,
> 565.377516497364 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 73
> LogicalProject(n_name=[$0], n_nationkey=[$1]): rowcount = 100.0,
> cumulative cost = \{300.0 rows, 501.0 cpu, 0.0 io, 0.0 network, 0.0 memory},
> id = 72
> LogicalProject(n_name=[$3], n_nationkey=[$4]): rowcount = 100.0,
> cumulative cost = \{200.0 rows, 301.0 cpu, 0.0 io, 0.0 network, 0.0 memory},
> id = 71
> EnumerableTableScan(table=[[cp, tpch/nation.parquet]]): rowcount =
> 100.0, cumulative cost = \{100.0 rows, 101.0 cpu, 0.0 io, 0.0 network, 0.0
> memory}, id = 34
> {code}
> It contains two EnumerableTableScan operators with different row types
> specified in the error message above. In the scope of CALCITE-2454 was
> removed rel data type from the key in the \{{VolcanoPlanner.mapDigestToRel}}
> map.
> To fix this, `{{TableScan` should include rowType}} item in
> {{AbstractRelNode#explainTerms(RelWriter)}} method because in case of dynamic
> tables there is the same digest for different operators. The same thing was
> done for Values operator and literals in conditions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-2948) SqlToRelConverter generates complicated logical plan for in subquery with non-equi condition
[
https://issues.apache.org/jira/browse/CALCITE-2948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819972#comment-16819972
]
Volodymyr Vysotskyi commented on CALCITE-2948:
--
[~danny0405], the aggregation for these cases guarantees that the resulting
rows number will be not greater than the row number of the outer table, since
without aggregate it may happen for the case when joining columns with repeated
values. Could you please check whether this case works correctly with your fix?
> SqlToRelConverter generates complicated logical plan for in subquery with
> non-equi condition
>
>
> Key: CALCITE-2948
> URL: https://issues.apache.org/jira/browse/CALCITE-2948
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Haisheng Yuan
>Assignee: Danny Chan
>Priority: Major
> Labels: pull-request-available, sub-query
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Repro:
> Add the following test to SqlToRelConverterTest.java.
> {code:java}
> @Test public void testSubQueryIN() {
> final String sql = "select deptno\n"
> + "from EMP e\n"
> + "where deptno in (select deptno\n"
> + "from EMP where empno=e.empno+1)";
> sql(sql).ok();
> }
> {code}
> Plan:
> {code:java}
> LogicalProject(DEPTNO=[$7])
> LogicalJoin(condition=[AND(=($0, $10), =($7, $9))], joinType=[inner])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> LogicalAggregate(group=[{0, 1}])
> LogicalProject(DEPTNO=[$7], EMPNO0=[$9])
> LogicalJoin(condition=[=($0, +($9, 1))], joinType=[inner])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> LogicalProject(EMPNO=[$0])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {code}
> One join would suffice.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-2018) Queries failed with AssertionError: rel has lower cost than best cost of subset
[
https://issues.apache.org/jira/browse/CALCITE-2018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818872#comment-16818872
]
Volodymyr Vysotskyi commented on CALCITE-2018:
--
[~danny0405], thanks for moving this issue forward. This issue partially solves
the problem with propagating best cost including the fixes for preserving only
actual `RelMetadataQuery`.
But it is blocked by a more serious problem - CALCITE-2166 where was shown that
the cumulative cost of best rel node actually is not always the best cost, the
problem there with the way how the best cost is calculated.
PS. I have rebased my PR onto the master and resolved all the conflicts.
Also, I don't mind fixing this Jira either with the fix I proposed or any
alternative fix, the main thing is to resolve not fewer problematic cases
compared to the original PR.
> Queries failed with AssertionError: rel has lower cost than best cost of
> subset
> ---
>
> Key: CALCITE-2018
> URL: https://issues.apache.org/jira/browse/CALCITE-2018
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.13.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Danny Chan
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> *Problem description*
> When rootLogger level is DEBUG, unit tests
> * MaterializationTest.testMaterializationSubstitution2
> * MaterializationTest.testJoinMaterializationUKFK8
> * MaterializationTest.testJoinMaterializationUKFK6
> * JdbcTest.testWhereNot
> unit tests are failed with error AssertionError: rel has lower cost than best
> cost of subset.
> Full stack trace for test
> {{MaterializationTest.testMaterializationSubstitution2}}:
> {noformat}
> java.lang.AssertionError: rel
> [rel#245:EnumerableUnion.ENUMERABLE.[](input#0=rel#246:Subset#5.ENUMERABLE.[],input#1=rel#239:Subset#6.ENUMERABLE.[0],all=true)]
> has lower cost {14.0 rows, 19.0 cpu, 0.0 io} than best cost {15.0 rows, 20.0
> cpu, 0.0 io} of subset [rel#243:Subset#7.ENUMERABLE.[]]
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.validate(VolcanoPlanner.java:906)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:866)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:883)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:101)
> at
> org.apache.calcite.rel.AbstractRelNode.onRegister(AbstractRelNode.java:336)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1495)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:863)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:883)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1766)
> at
> org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:135)
> at
> org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:234)
> at
> org.apache.calcite.rel.rules.FilterProjectTransposeRule.onMatch(FilterProjectTransposeRule.java:143)
> at
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:212)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:650)
> at org.apache.calcite.tools.Programs$5.run(Programs.java:326)
> at
> org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:387)
> at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:187)
> at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:318)
> at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:229)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:786)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:640)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:610)
> at org.apache.calcite.schema.Schemas.prepare(Schemas.java:346)
> at
> org.apache.calcite.materialize.MaterializationService$DefaultTableFactory.createTable(MaterializationService.java:374)
> at
> org.apache.calcite.materialize.MaterializationService.defineMaterialization(MaterializationService.java:137)
> at
> org.apache.calcite.materialize.MaterializationService.defineMaterialization(MaterializationService.java:99)
> at
> org.apache.calcite.schema.impl.MaterializedViewTable$MaterializedViewTableMacro.(MaterializedViewTable.java:110)
> at
>
[jira] [Commented] (CALCITE-2223) ProjectMergeRule is infinitely matched when is applied after ProjectReduceExpressionsRule
[
https://issues.apache.org/jira/browse/CALCITE-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16759377#comment-16759377
]
Volodymyr Vysotskyi commented on CALCITE-2223:
--
Yes, that's correct that usual cartesian join cannot be planned in Drill, but
there is relaxation for this limitation: it is allowed for the case when one of
the join inputs returns a single value (or nothing).
For the case of this query, {{SINGLE_VALUE}} aggregate function is applied to
the result of the subquery in the filter condition. Later, joins are reordered
somehow, and received resulting plan:
{noformat}
00-00Screen : rowType = RecordType(ANY last_name, ANY n_name): rowcount =
1.0, cumulative cost = {1455.1 rows, 191524.1 cpu, 474685.0 io, 0.0 network,
440.06 memory}, id = 1165
00-01 Project(last_name=[$0], n_name=[$2]) : rowType = RecordType(ANY
last_name, ANY n_name): rowcount = 1.0, cumulative cost = {1455.0 rows,
191524.0 cpu, 474685.0 io, 0.0 network, 440.06 memory}, id = 1164
00-02Project(last_name=[$0], n_nationkey=[$2], n_name=[$3], $f0=[$1]) :
rowType = RecordType(ANY last_name, ANY n_nationkey, ANY n_name, ANY $f0):
rowcount = 1.0, cumulative cost = {1454.0 rows, 191522.0 cpu, 474685.0 io, 0.0
network, 440.06 memory}, id = 1163
00-03 SelectionVectorRemover : rowType = RecordType(ANY last_name, ANY
$f0, ANY n_nationkey, ANY n_name): rowcount = 1.0, cumulative cost = {1453.0
rows, 191518.0 cpu, 474685.0 io, 0.0 network, 440.06 memory}, id =
1162
00-04Limit(fetch=[1]) : rowType = RecordType(ANY last_name, ANY
$f0, ANY n_nationkey, ANY n_name): rowcount = 1.0, cumulative cost = {1452.0
rows, 191517.0 cpu, 474685.0 io, 0.0 network, 440.06 memory}, id =
1161
00-05 Limit(fetch=[1]) : rowType = RecordType(ANY last_name, ANY
$f0, ANY n_nationkey, ANY n_name): rowcount = 1.0, cumulative cost = {1451.0
rows, 191513.0 cpu, 474685.0 io, 0.0 network, 440.06 memory}, id =
1160
00-06HashJoin(condition=[=($2, $1)], joinType=[inner],
semi-join: =[false]) : rowType = RecordType(ANY last_name, ANY $f0, ANY
n_nationkey, ANY n_name): rowcount = 463.0, cumulative cost = {1450.0 rows,
191509.0 cpu, 474685.0 io, 0.0 network, 440.06 memory}, id = 1159
00-08 NestedLoopJoin(condition=[true], joinType=[inner]) :
rowType = RecordType(ANY last_name, ANY $f0): rowcount = 463.0, cumulative cost
= {937.0 rows, 185703.0 cpu, 474635.0 io, 0.0 network, 0.0 memory}, id = 1157
00-10Scan(table=[[cp, employee.json]],
groupscan=[EasyGroupScan [selectionRoot=classpath:/employee.json, numFiles=1,
columns=[`last_name`], files=[classpath:/employee.json]]]) : rowType =
RecordType(ANY last_name): rowcount = 463.0, cumulative cost = {463.0 rows,
463.0 cpu, 474630.0 io, 0.0 network, 0.0 memory}, id = 1153
00-09StreamAgg(group=[{}], agg#0=[SINGLE_VALUE($0)]) :
rowType = RecordType(ANY $f0): rowcount = 1.0, cumulative cost = {11.0 rows,
40.0 cpu, 5.0 io, 0.0 network, 0.0 memory}, id = 1156
00-11 Filter(condition=[=($0, 1)]) : rowType =
RecordType(ANY r_regionkey): rowcount = 1.0, cumulative cost = {10.0 rows, 28.0
cpu, 5.0 io, 0.0 network, 0.0 memory}, id = 1155
00-12Scan(table=[[cp, tpch/region.parquet]],
groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath
[path=classpath:/tpch/region.parquet]],
selectionRoot=classpath:/tpch/region.parquet, numFiles=1, numRowGroups=1,
usedMetadataFile=false, columns=[`r_regionkey`]]]) : rowType = RecordType(ANY
r_regionkey): rowcount = 5.0, cumulative cost = {5.0 rows, 5.0 cpu, 5.0 io, 0.0
network, 0.0 memory}, id = 1154
00-07 Scan(table=[[cp, tpch/nation.parquet]],
groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath
[path=classpath:/tpch/nation.parquet]],
selectionRoot=classpath:/tpch/nation.parquet, numFiles=1, numRowGroups=1,
usedMetadataFile=false, columns=[`n_nationkey`, `n_name`]]]) : rowType =
RecordType(ANY n_nationkey, ANY n_name): rowcount = 25.0, cumulative cost =
{25.0 rows, 50.0 cpu, 50.0 io, 0.0 network, 0.0 memory}, id = 1158
{noformat}
So one of the inputs of {{NestedLoopJoin}} is {{StreamAgg(group=[{}],
agg#0=[SINGLE_VALUE($0)])}} which returns single row.
Regarding the check for single value, one of the conditions is [aggregate with
empty
groupset|https://github.com/apache/drill/blob/f6c63bf5dbc7bcd14b202249d013cd974a96a68a/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/JoinUtils.java#L275].
> ProjectMergeRule is infinitely matched when is applied after
> ProjectReduceExpressionsRule
> -
>
> Key: CALCITE-2223
> URL: https://issues.apache.org/jira/browse/CALCITE-2223
>
[jira] [Commented] (CALCITE-2223) ProjectMergeRule is infinitely matched when is applied after ProjectReduceExpressionsRule
[
https://issues.apache.org/jira/browse/CALCITE-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16759024#comment-16759024
]
Volodymyr Vysotskyi commented on CALCITE-2223:
--
If I understand the intention of PR-1019 correctly, it will significantly
reduce matching rules by allowing applying rules to the rel nodes with concrete
convention either specified explicitly or deduced from passed RelBuilderFactory.
>From the logs for failing
>{{TestExampleQueries.testMultipleComparisonWithSingleValueSubQuery}} test I
>see a lot of the following entries:
{{Rel {} does not match rule {} since the relation does not match implicit
traits {}}}
Most of the rules accept {{RelFactories.LOGICAL_BUILDER}}, so they will create
logical rel nodes, and at the logical planning stage, everything works fine.
But the same rules which are used for the logical stage are also applied at the
physical stage to allow optimizing the plan after applying physical rules.
Not sure that the case {{LogicalProject(FilterPrel(...))}} is applicable for
the rules, but there are converters, which may convert {{LogicalProject}} to
{{DrillProjectRel}} and then to {{ProjectPrel}}, so the rule may be applied at
least to {{ProjectPrel(FilterPrel(...))}}
But with the changes made in the PR it becomes impossible. I think this is the
reason why these tests fail. In particular, I think the problem here is with
joins reordering. One of the rules (or more) wasn't applied, and perhaps nested
loop join which by default allowed only for scalar subqueries was on top of the
regular loin.
> ProjectMergeRule is infinitely matched when is applied after
> ProjectReduceExpressionsRule
> -
>
> Key: CALCITE-2223
> URL: https://issues.apache.org/jira/browse/CALCITE-2223
> Project: Calcite
> Issue Type: Bug
>Reporter: Volodymyr Vysotskyi
>Assignee: Julian Hyde
>Priority: Critical
> Attachments:
> TestLimitWithExchanges_testPushLimitPastUnionExchange.png, heap_overview.png,
> provenance_contents.png
>
>
> For queries like this:
> {code:sql}
> select t1.f from (select cast(f as int) f, f from (select cast(f as int) f
> from (values('1')) t(f))) as t1
> {code}
> OOM is thrown when {{ProjectMergeRule}} is applied before applying
> {{ProjectReduceExpressionsRule}} in VolcanoPlanner.
> A simple test to reproduce this issue (in {{RelOptRulesTest}}):
> {code:java}
> @Test public void testOomProjectMergeRule() {
> RelBuilder relBuilder =
> RelBuilder.create(RelBuilderTest.config().build());
> RelNode relNode = relBuilder
> .values(new String[]{"f"}, "1")
> .project(
> relBuilder.alias(
> relBuilder.cast(relBuilder.field(0), SqlTypeName.INTEGER),
> "f"))
> .project(
> relBuilder.alias(
> relBuilder.cast(relBuilder.field(0), SqlTypeName.INTEGER),
> "f0"),
> relBuilder.alias(relBuilder.field(0), "f"))
> .project(
> relBuilder.alias(relBuilder.field(0), "f"))
> .build();
> RelOptPlanner planner = relNode.getCluster().getPlanner();
> RuleSet ruleSet =
> RuleSets.ofList(
> ReduceExpressionsRule.PROJECT_INSTANCE,
> new ProjectMergeRuleWithLongerName(),
> EnumerableRules.ENUMERABLE_PROJECT_RULE,
> EnumerableRules.ENUMERABLE_VALUES_RULE);
> Program program = Programs.of(ruleSet);
> RelTraitSet toTraits =
> relNode.getCluster().traitSet()
> .replace(0, EnumerableConvention.INSTANCE);
> RelNode output = program.run(planner, relNode, toTraits,
> ImmutableList.of(),
> ImmutableList.of());
> // check for output
> }
> /**
>* ProjectMergeRule inheritor which has
>* class name greater than ProjectReduceExpressionsRule class name
> (String.compareTo()).
>*
>* It is needed for RuleQueue.popMatch() method
>* to apply this rule before ProjectReduceExpressionsRule.
>*/
> private static class ProjectMergeRuleWithLongerName extends
> ProjectMergeRule {
> public ProjectMergeRuleWithLongerName() {
> super(true, RelFactories.LOGICAL_BUILDER);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-2329) Enhance SubQueryRemoveRule to rewrite IN operator with the constant from the left side more optimally
[
https://issues.apache.org/jira/browse/CALCITE-2329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756664#comment-16756664
]
Volodymyr Vysotskyi commented on CALCITE-2329:
--
[~vgarg], I agree with you that such plan will produce wrong results, since it
may change resulting rows number. I have submitted a similar query for the
current master, and resulting plan contains the required aggregate operator:
{noformat}
select \"empid\" from \"hr\".\"emps\" where 2 in (select \"deptno\" from
\"hr\".\"emps\")
LogicalProject(empid=[$0]): rowcount = 100.0, cumulative cost = {631.0 rows,
617.0 cpu, 0.0 io}, id = 17
LogicalJoin(condition=[true], joinType=[inner]): rowcount = 100.0, cumulative
cost = {531.0 rows, 517.0 cpu, 0.0 io}, id = 16
LogicalProject(empid=[$0]): rowcount = 100.0, cumulative cost = {200.0
rows, 201.0 cpu, 0.0 io}, id = 9
EnumerableTableScan(table=[[hr, emps]]): rowcount = 100.0, cumulative
cost = {100.0 rows, 101.0 cpu, 0.0 io}, id = 0
LogicalAggregate(group=[{0}]): rowcount = 1.0, cumulative cost = {231.0
rows, 316.0 cpu, 0.0 io}, id = 15
LogicalProject(cs=[true]): rowcount = 15.0, cumulative cost = {230.0
rows, 316.0 cpu, 0.0 io}, id = 14
LogicalFilter(condition=[=(2, $0)]): rowcount = 15.0, cumulative cost =
{215.0 rows, 301.0 cpu, 0.0 io}, id = 13
LogicalProject(deptno=[$1]): rowcount = 100.0, cumulative cost =
{200.0 rows, 201.0 cpu, 0.0 io}, id = 2
EnumerableTableScan(table=[[hr, emps]]): rowcount = 100.0,
cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io}, id = 1
{noformat}
> Enhance SubQueryRemoveRule to rewrite IN operator with the constant from the
> left side more optimally
> -
>
> Key: CALCITE-2329
> URL: https://issues.apache.org/jira/browse/CALCITE-2329
> Project: Calcite
> Issue Type: Improvement
>Reporter: Volodymyr Vysotskyi
>Assignee: Julian Hyde
>Priority: Major
> Fix For: 1.17.0
>
>
> Currently, for queries like this:
> {code:sql}
> select sal from emp where 2 IN (select deptno from dept)
> {code}
> SubQueryRemoveRule rules expand query plan in such a way:
> {noformat}
> LogicalProject(SAL=[$5])
> LogicalProject(EMPNO=[$0], ENAME=[$1], JOB=[$2], MGR=[$3], HIREDATE=[$4],
> SAL=[$5], COMM=[$6], DEPTNO=[$7], SLACKER=[$8])
> LogicalJoin(condition=[=(2, $9)], joinType=[inner])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> LogicalProject(DEPTNO=[$0])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> {noformat}
> Since join condition is actually a filter condition, it will be pushed into
> the filter during further planning stages and join with the true condition is
> left.
> But these types of the queries may be rewritten in the same way as EXISTS
> queries:
> {code:sql}
> select sal from emp where EXISTS (select deptno from dept where deptno=2)
> {code}
> with the more optimal plan:
> {noformat}
> LogicalProject(SAL=[$5])
> LogicalProject(EMPNO=[$0], ENAME=[$1], JOB=[$2], MGR=[$3], HIREDATE=[$4],
> SAL=[$5], COMM=[$6], DEPTNO=[$7], SLACKER=[$8])
> LogicalJoin(condition=[true], joinType=[inner])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> LogicalAggregate(group=[{0}])
> LogicalProject(i=[true])
> LogicalFilter(condition=[=($0, 2)])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-2223) ProjectMergeRule is infinitely matched when is applied after ProjectReduceExpressionsRule
[
https://issues.apache.org/jira/browse/CALCITE-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16753953#comment-16753953
]
Volodymyr Vysotskyi commented on CALCITE-2223:
--
I see you have enabled {{FilterSetOpTransposeRule.INSTANCE}} and
{{AggregateUnionTransposeRule.INSTANCE}} rules, but if they are enabled in
Drill
([1|https://github.com/apache/drill/blob/b557b796dc1ca7796b0db956e39df1d52f212f12/exec/java-exec/src/main/java/org/apache/drill/exec/planner/PlannerPhase.java#L328],
[2|https://github.com/apache/drill/blob/b557b796dc1ca7796b0db956e39df1d52f212f12/exec/java-exec/src/main/java/org/apache/drill/exec/planner/PlannerPhase.java#L308]),
some tests (at least {{TestUnionAll.testUnionAllInWith}}) fail with
{{StackOverflowError}} though Drill does not use materialization rules yet.
Does similar scenarios possible for Calcite?
> ProjectMergeRule is infinitely matched when is applied after
> ProjectReduceExpressionsRule
> -
>
> Key: CALCITE-2223
> URL: https://issues.apache.org/jira/browse/CALCITE-2223
> Project: Calcite
> Issue Type: Bug
>Reporter: Volodymyr Vysotskyi
>Assignee: Julian Hyde
>Priority: Critical
> Attachments:
> TestLimitWithExchanges_testPushLimitPastUnionExchange.png, heap_overview.png,
> provenance_contents.png
>
>
> For queries like this:
> {code:sql}
> select t1.f from (select cast(f as int) f, f from (select cast(f as int) f
> from (values('1')) t(f))) as t1
> {code}
> OOM is thrown when {{ProjectMergeRule}} is applied before applying
> {{ProjectReduceExpressionsRule}} in VolcanoPlanner.
> A simple test to reproduce this issue (in {{RelOptRulesTest}}):
> {code:java}
> @Test public void testOomProjectMergeRule() {
> RelBuilder relBuilder =
> RelBuilder.create(RelBuilderTest.config().build());
> RelNode relNode = relBuilder
> .values(new String[]{"f"}, "1")
> .project(
> relBuilder.alias(
> relBuilder.cast(relBuilder.field(0), SqlTypeName.INTEGER),
> "f"))
> .project(
> relBuilder.alias(
> relBuilder.cast(relBuilder.field(0), SqlTypeName.INTEGER),
> "f0"),
> relBuilder.alias(relBuilder.field(0), "f"))
> .project(
> relBuilder.alias(relBuilder.field(0), "f"))
> .build();
> RelOptPlanner planner = relNode.getCluster().getPlanner();
> RuleSet ruleSet =
> RuleSets.ofList(
> ReduceExpressionsRule.PROJECT_INSTANCE,
> new ProjectMergeRuleWithLongerName(),
> EnumerableRules.ENUMERABLE_PROJECT_RULE,
> EnumerableRules.ENUMERABLE_VALUES_RULE);
> Program program = Programs.of(ruleSet);
> RelTraitSet toTraits =
> relNode.getCluster().traitSet()
> .replace(0, EnumerableConvention.INSTANCE);
> RelNode output = program.run(planner, relNode, toTraits,
> ImmutableList.of(),
> ImmutableList.of());
> // check for output
> }
> /**
>* ProjectMergeRule inheritor which has
>* class name greater than ProjectReduceExpressionsRule class name
> (String.compareTo()).
>*
>* It is needed for RuleQueue.popMatch() method
>* to apply this rule before ProjectReduceExpressionsRule.
>*/
> private static class ProjectMergeRuleWithLongerName extends
> ProjectMergeRule {
> public ProjectMergeRuleWithLongerName() {
> super(true, RelFactories.LOGICAL_BUILDER);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-2223) ProjectMergeRule is infinitely matched when is applied after ProjectReduceExpressionsRule
[
https://issues.apache.org/jira/browse/CALCITE-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16753889#comment-16753889
]
Volodymyr Vysotskyi commented on CALCITE-2223:
--
Perhaps rough, but should be a working workaround: run tests with properties
{{-Dmaven.surefire.debug="-Duser.language=en -Duser.region=US
-Duser.country=US" -Dmaven.failsafe.debug="-Duser.language=en -Duser.region=US
-Duser.country=US"}}
Didn't find a better way without modifying the code.
> ProjectMergeRule is infinitely matched when is applied after
> ProjectReduceExpressionsRule
> -
>
> Key: CALCITE-2223
> URL: https://issues.apache.org/jira/browse/CALCITE-2223
> Project: Calcite
> Issue Type: Bug
>Reporter: Volodymyr Vysotskyi
>Assignee: Julian Hyde
>Priority: Critical
> Attachments:
> TestLimitWithExchanges_testPushLimitPastUnionExchange.png, heap_overview.png,
> provenance_contents.png
>
>
> For queries like this:
> {code:sql}
> select t1.f from (select cast(f as int) f, f from (select cast(f as int) f
> from (values('1')) t(f))) as t1
> {code}
> OOM is thrown when {{ProjectMergeRule}} is applied before applying
> {{ProjectReduceExpressionsRule}} in VolcanoPlanner.
> A simple test to reproduce this issue (in {{RelOptRulesTest}}):
> {code:java}
> @Test public void testOomProjectMergeRule() {
> RelBuilder relBuilder =
> RelBuilder.create(RelBuilderTest.config().build());
> RelNode relNode = relBuilder
> .values(new String[]{"f"}, "1")
> .project(
> relBuilder.alias(
> relBuilder.cast(relBuilder.field(0), SqlTypeName.INTEGER),
> "f"))
> .project(
> relBuilder.alias(
> relBuilder.cast(relBuilder.field(0), SqlTypeName.INTEGER),
> "f0"),
> relBuilder.alias(relBuilder.field(0), "f"))
> .project(
> relBuilder.alias(relBuilder.field(0), "f"))
> .build();
> RelOptPlanner planner = relNode.getCluster().getPlanner();
> RuleSet ruleSet =
> RuleSets.ofList(
> ReduceExpressionsRule.PROJECT_INSTANCE,
> new ProjectMergeRuleWithLongerName(),
> EnumerableRules.ENUMERABLE_PROJECT_RULE,
> EnumerableRules.ENUMERABLE_VALUES_RULE);
> Program program = Programs.of(ruleSet);
> RelTraitSet toTraits =
> relNode.getCluster().traitSet()
> .replace(0, EnumerableConvention.INSTANCE);
> RelNode output = program.run(planner, relNode, toTraits,
> ImmutableList.of(),
> ImmutableList.of());
> // check for output
> }
> /**
>* ProjectMergeRule inheritor which has
>* class name greater than ProjectReduceExpressionsRule class name
> (String.compareTo()).
>*
>* It is needed for RuleQueue.popMatch() method
>* to apply this rule before ProjectReduceExpressionsRule.
>*/
> private static class ProjectMergeRuleWithLongerName extends
> ProjectMergeRule {
> public ProjectMergeRuleWithLongerName() {
> super(true, RelFactories.LOGICAL_BUILDER);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-2223) ProjectMergeRule is infinitely matched when is applied after ProjectReduceExpressionsRule
[
https://issues.apache.org/jira/browse/CALCITE-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16753391#comment-16753391
]
Volodymyr Vysotskyi commented on CALCITE-2223:
--
Yes, that's correct, ignored {{UnionExchangePrel}} instance is created using
converter.
> ProjectMergeRule is infinitely matched when is applied after
> ProjectReduceExpressionsRule
> -
>
> Key: CALCITE-2223
> URL: https://issues.apache.org/jira/browse/CALCITE-2223
> Project: Calcite
> Issue Type: Bug
>Reporter: Volodymyr Vysotskyi
>Assignee: Julian Hyde
>Priority: Critical
> Attachments:
> TestLimitWithExchanges_testPushLimitPastUnionExchange.png, heap_overview.png,
> provenance_contents.png
>
>
> For queries like this:
> {code:sql}
> select t1.f from (select cast(f as int) f, f from (select cast(f as int) f
> from (values('1')) t(f))) as t1
> {code}
> OOM is thrown when {{ProjectMergeRule}} is applied before applying
> {{ProjectReduceExpressionsRule}} in VolcanoPlanner.
> A simple test to reproduce this issue (in {{RelOptRulesTest}}):
> {code:java}
> @Test public void testOomProjectMergeRule() {
> RelBuilder relBuilder =
> RelBuilder.create(RelBuilderTest.config().build());
> RelNode relNode = relBuilder
> .values(new String[]{"f"}, "1")
> .project(
> relBuilder.alias(
> relBuilder.cast(relBuilder.field(0), SqlTypeName.INTEGER),
> "f"))
> .project(
> relBuilder.alias(
> relBuilder.cast(relBuilder.field(0), SqlTypeName.INTEGER),
> "f0"),
> relBuilder.alias(relBuilder.field(0), "f"))
> .project(
> relBuilder.alias(relBuilder.field(0), "f"))
> .build();
> RelOptPlanner planner = relNode.getCluster().getPlanner();
> RuleSet ruleSet =
> RuleSets.ofList(
> ReduceExpressionsRule.PROJECT_INSTANCE,
> new ProjectMergeRuleWithLongerName(),
> EnumerableRules.ENUMERABLE_PROJECT_RULE,
> EnumerableRules.ENUMERABLE_VALUES_RULE);
> Program program = Programs.of(ruleSet);
> RelTraitSet toTraits =
> relNode.getCluster().traitSet()
> .replace(0, EnumerableConvention.INSTANCE);
> RelNode output = program.run(planner, relNode, toTraits,
> ImmutableList.of(),
> ImmutableList.of());
> // check for output
> }
> /**
>* ProjectMergeRule inheritor which has
>* class name greater than ProjectReduceExpressionsRule class name
> (String.compareTo()).
>*
>* It is needed for RuleQueue.popMatch() method
>* to apply this rule before ProjectReduceExpressionsRule.
>*/
> private static class ProjectMergeRuleWithLongerName extends
> ProjectMergeRule {
> public ProjectMergeRuleWithLongerName() {
> super(true, RelFactories.LOGICAL_BUILDER);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-2223) ProjectMergeRule is infinitely matched when is applied after ProjectReduceExpressionsRule
[
https://issues.apache.org/jira/browse/CALCITE-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16753370#comment-16753370
]
Volodymyr Vysotskyi commented on CALCITE-2223:
--
These failures aren't expected. Do these tests pass when running separately? If
yes, it is possible that there are not enough resources, so please try to
increase the heap size:
{{export MAVEN_OPTS="-Xms2G -Xmx2G"}}
before executing {{mvn clean install}}. If it does not help, you may try to run
{{mvn clean install -DforkCount=1}}.
If this is env issue, please check {{etc/hosts}} that the hostname of the
machine with its correct IP are specified and associated. I'm not a MacOS user,
so I can't help here significantly.
Also, I have run tests using your branch, and the next test failed:
{noformat}
[ERROR] Failures:
[ERROR]
TestLimitWithExchanges.testPushLimitPastUnionExchange:56->testLimitHelper:142
Did not find expected pattern in plan: (?s)Limit\(offset=\[2\],
fetch=\[1\].*UnionExchange.*Limit\(fetch=\[3\]\).*Scan
00-00Screen
00-01 Project(**=[$0])
00-02SelectionVectorRemover
00-03 Limit(offset=[2], fetch=[1])
00-04UnionExchange
01-01 Scan(table=[[dfs, multilevel/json]],
groupscan=[EasyGroupScan
[selectionRoot=file:/home/user515050/work/review/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.limit.TestLimitWithExchanges/root/multilevel/json,
numFiles=12, columns=[`**`],
files=[file:/home/user515050/work/review/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.limit.TestLimitWithExchanges/root/multilevel/json/1995/Q3/orders_95_q3.json,
file:/home/user515050/work/review/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.limit.TestLimitWithExchanges/root/multilevel/json/1995/Q1/orders_95_q1.json,
file:/home/user515050/work/review/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.limit.TestLimitWithExchanges/root/multilevel/json/1995/Q4/orders_95_q4.json,
file:/home/user515050/work/review/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.limit.TestLimitWithExchanges/root/multilevel/json/1995/Q2/orders_95_q2.json,
file:/home/user515050/work/review/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.limit.TestLimitWithExchanges/root/multilevel/json/1996/Q3/orders_96_q3.json,
file:/home/user515050/work/review/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.limit.TestLimitWithExchanges/root/multilevel/json/1996/Q1/orders_96_q1.json,
file:/home/user515050/work/review/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.limit.TestLimitWithExchanges/root/multilevel/json/1996/Q4/orders_96_q4.json,
file:/home/user515050/work/review/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.limit.TestLimitWithExchanges/root/multilevel/json/1996/Q2/orders_96_q2.json,
file:/home/user515050/work/review/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.limit.TestLimitWithExchanges/root/multilevel/json/1994/Q3/orders_94_q3.json,
file:/home/user515050/work/review/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.limit.TestLimitWithExchanges/root/multilevel/json/1994/Q1/orders_94_q1.json,
file:/home/user515050/work/review/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.limit.TestLimitWithExchanges/root/multilevel/json/1994/Q4/orders_94_q4.json,
file:/home/user515050/work/review/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.limit.TestLimitWithExchanges/root/multilevel/json/1994/Q2/orders_94_q2.json]]])
[INFO]
[ERROR] Tests run: 3739, Failures: 1, Errors: 0, Skipped: 153
{noformat}
With this change, {{LimitExchangeTransposeRule}} cannot be applied because of
ignoring {{UnionExchangePrel}} rel node as a part of the cycle.
Drill has a lot of interesting cases, in particular, tests with category
{{PlannerTest}}, but as it is expected, these tests mostly check the
correctness of drill-specific rules and use cases.
> ProjectMergeRule is infinitely matched when is applied after
> ProjectReduceExpressionsRule
> -
>
> Key: CALCITE-2223
> URL: https://issues.apache.org/jira/browse/CALCITE-2223
> Project: Calcite
> Issue Type: Bug
>Reporter: Volodymyr Vysotskyi
>Assignee: Julian Hyde
>Priority: Critical
> Attachments: heap_overview.png, provenance_contents.png
>
>
> For queries like this:
> {code:sql}
> select t1.f from (select cast(f as int) f, f from (select cast(f as int) f
> from (values('1')) t(f))) as t1
> {code}
> OOM is thrown when {{ProjectMergeRule}} is applied before applying
> {{ProjectReduceExpressionsRule}} in VolcanoPlanner.
> A simple test to reproduce this issue (in {{RelOptRulesTest}}):
> {code:java}
> @Test
[jira] [Commented] (CALCITE-2223) ProjectMergeRule is infinitely matched when is applied after ProjectReduceExpressionsRule
[
https://issues.apache.org/jira/browse/CALCITE-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16753049#comment-16753049
]
Volodymyr Vysotskyi commented on CALCITE-2223:
--
[~vladimirsitnikov] , thanks a lot for debugging this issue from drill side.
I have run Drill tests for the changes from the PR, and it causes failures for
many drill tests:
{noformat}
[ERROR] Errors:
[ERROR] TestAltSortQueries.testJoinWithLimit » UserRemote SYSTEM ERROR:
CannotPlanExce...
[ERROR] TestBugFixes.DRILL883 » UserRemote SYSTEM ERROR: CannotPlanException:
Node [re...
[ERROR]
TestBugFixes.testDRILL2361_JoinColumnAliasWithDots:161->BaseTestQuery.testRunAndReturn:341
» Rpc
[ERROR] TestBugFixes.testDRILL5269 » UserRemote UNSUPPORTED_OPERATION ERROR:
This quer...
[ERROR] TestCaseSensitivity.testCaseSenWhenQueryTwoDiffCols:56 »
org.apache.drill.com...
[ERROR]
TestCorrelation.testSeveralExistsCorrelateSubquery:105->BaseTestQuery.testRunAndReturn:341
» Rpc
[ERROR] TestExampleQueries.testDRILL_811ViewJoin » UserRemote SYSTEM ERROR:
CannotPlan...
[ERROR] TestExampleQueries.testJoinExpOn » UserRemote SYSTEM ERROR:
CannotPlanExceptio...
[ERROR] TestExampleQueries.testNestedTypesPastJoinReportsValidResult:677 »
org.apache...
[ERROR] TestExampleQueries.testPushExpInJoinConditionRightJoin » UserRemote
SYSTEM ERR...
[ERROR]
TestExampleQueries.testRepeatedListProjectionPastJoin:1012->BaseTestQuery.testRunAndReturn:341
» Rpc
[ERROR] TestExampleQueries.testTextJoin » UserRemote SYSTEM ERROR:
CannotPlanException...
[ERROR] TestExampleQueries.testViewFileName » UserRemote VALIDATION ERROR: A
view with...
[ERROR]
TestExampleQueries.testWindowFunAndStarCol:1070->BaseTestQuery.testRunAndReturn:341
» Rpc
[ERROR]
TestJoinNullable.testFullJoinUsingUntypedNullColumn:572->BaseTestQuery.testRunAndReturn:341
» Rpc
[ERROR] TestJoinNullable.testHashFOJOnNullableColumns » UserRemote SYSTEM
ERROR: Canno...
[ERROR] TestJoinNullable.testHashInnerJoinOnNullableColumns » UserRemote
SYSTEM ERROR:...
[ERROR] TestJoinNullable.testHashLOJOnNullableColumns » UserRemote SYSTEM
ERROR: Canno...
[ERROR] TestJoinNullable.testHashROJOnNullableColumns » UserRemote SYSTEM
ERROR: Canno...
[ERROR] TestJoinNullable.testMergeInnerJoinOnNullableColumns » UserRemote
SYSTEM ERROR...
[ERROR] TestJoinNullable.testMergeLOJNullable » UserRemote SYSTEM ERROR:
CannotPlanExc...
[ERROR] TestJoinNullable.testMergeLOJNullableNoOrderedInputs » UserRemote
SYSTEM ERROR...
[ERROR] TestJoinNullable.testMergeLOJNullableOneOrderedInputAscNullsFirst »
UserRemote
[ERROR] TestJoinNullable.testMergeLOJNullableOneOrderedInputAscNullsLast »
UserRemote ...
[ERROR] TestJoinNullable.testMergeLOJNullableOneOrderedInputDescNullsFirst »
UserRemote
[ERROR] TestJoinNullable.testMergeLOJNullableOneOrderedInputDescNullsLast »
UserRemote
[ERROR] TestJoinNullable.testMergeROJOnNullableColumns » UserRemote SYSTEM
ERROR: Cann...
[ERROR]
TestJoinNullable.testMixedEqualAndEqualOrHashJoin:537->nullMixedComparatorEqualJoinHelper:587->BaseTestQuery.testRunAndReturn:341
» Rpc
[ERROR]
TestJoinNullable.testMixedEqualAndEqualOrMergeJoin:553->nullMixedComparatorEqualJoinHelper:587->BaseTestQuery.testRunAndReturn:341
» Rpc
[ERROR]
TestJoinNullable.testMixedEqualAndIsNotDistinctFilterHashJoin:505->nullMixedComparatorEqualJoinHelper:587->BaseTestQuery.testRunAndReturn:341
» Rpc
[ERROR]
TestJoinNullable.testMixedEqualAndIsNotDistinctFilterMergeJoin:521->nullMixedComparatorEqualJoinHelper:587->BaseTestQuery.testRunAndReturn:341
» Rpc
[ERROR]
TestJoinNullable.testMixedEqualAndIsNotDistinctHashJoin:475->nullMixedComparatorEqualJoinHelper:587->BaseTestQuery.testRunAndReturn:341
» Rpc
[ERROR]
TestJoinNullable.testMixedEqualAndIsNotDistinctMergeJoin:489->nullMixedComparatorEqualJoinHelper:587->BaseTestQuery.testRunAndReturn:341
» Rpc
[ERROR]
TestJoinNullable.testNullEqualAdditionFilter:464->BaseTestQuery.testRunAndReturn:341
» Rpc
[ERROR]
TestJoinNullable.testNullEqualHashJoin:417->nullEqualJoinHelper:447->BaseTestQuery.testRunAndReturn:341
» Rpc
[ERROR]
TestJoinNullable.testNullEqualInWhereConditionHashJoin:394->nullEqualJoinHelper:447->BaseTestQuery.testRunAndReturn:341
» Rpc
[ERROR]
TestJoinNullable.testNullEqualInWhereConditionMergeJoin:405->nullEqualJoinHelper:447->BaseTestQuery.testRunAndReturn:341
» Rpc
[ERROR]
TestJoinNullable.testNullEqualMergeJoin:428->nullEqualJoinHelper:447->BaseTestQuery.testRunAndReturn:341
» Rpc
[ERROR]
TestMergeFilterPlan.testDRILL_FilterMerge:48->PlanTestBase.testPlanSubstrPatterns:190->PlanTestBase.getPlanInString:369->BaseTestQuery.testSqlWithResults:322->BaseTestQuery.testRunAndReturn:341
» Rpc
[ERROR] TestOperatorDump.testScanBatchChecked » Unexpected exception,
[jira] [Comment Edited] (CALCITE-2799) Allow alias in having clause for aggregate functions
[
https://issues.apache.org/jira/browse/CALCITE-2799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16750961#comment-16750961
]
Volodymyr Vysotskyi edited comment on CALCITE-2799 at 1/24/19 10:03 AM:
So since aliases priority is another issue, I think we can create a separate
Jira to address it, but merge this PR with ignored test case proposed by
[~vladimirsitnikov] and modified plan since this fix makes things more
consistent.
[~julianhyde], [~vladimirsitnikov] what do you think about this?
\* Under "makes things more consistent" I meant such cases:
{code:sql}
select x+10 as x
from (values(1),(2)) as t(x)
group by x
having max(x)>11;
select x+10 as x
from (values(1),(2)) as t(x)
group by x
having x>11;
{code}
was (Author: vvysotskyi):
So since aliases priority is another issue, I think we can create a separate
Jira to address it, but merge this PR with ignored test case proposed by
[~vladimirsitnikov] and modified plan since this fix makes things more
consistent.
[~julianhyde], [~vladimirsitnikov] what do you think about this?
** Under "makes things more consistent" I meant such cases:
{code:sql}
select x+10 as x
from (values(1),(2)) as t(x)
group by x
having max(x)>11;
select x+10 as x
from (values(1),(2)) as t(x)
group by x
having x>11;
{code}
> Allow alias in having clause for aggregate functions
>
>
> Key: CALCITE-2799
> URL: https://issues.apache.org/jira/browse/CALCITE-2799
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.18.0
>Reporter: Arina Ielchiieva
>Assignee: Julian Hyde
>Priority: Major
> Fix For: 1.19.0
>
>
> Currently alias is not allowed in having for aggregate functions.
> MySql supports such cases and taking into account that alias in having clause
> is allowed only for the following conformance levels: MYSQL_5, LENIENT,
> BABEL, it makes sense to allow alias in having for aggregate functions.
> {noformat}
> /**
>* Whether to allow aliases from the {@code SELECT} clause to be used as
>* column names in the {@code HAVING} clause.
>*
>* Among the built-in conformance levels, true in
>* {@link SqlConformanceEnum#BABEL},
>* {@link SqlConformanceEnum#LENIENT},
>* {@link SqlConformanceEnum#MYSQL_5};
>* false otherwise.
>*/
> boolean isHavingAlias();
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-2799) Allow alias in having clause for aggregate functions
[
https://issues.apache.org/jira/browse/CALCITE-2799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16750961#comment-16750961
]
Volodymyr Vysotskyi commented on CALCITE-2799:
--
So since aliases priority is another issue, I think we can create a separate
Jira to address it, but merge this PR with ignored test case proposed by
[~vladimirsitnikov] and modified plan since this fix makes things more
consistent.
[~julianhyde], [~vladimirsitnikov] what do you think about this?
** Under "makes things more consistent" I meant such cases:
{code:sql}
select x+10 as x
from (values(1),(2)) as t(x)
group by x
having max(x)>11;
select x+10 as x
from (values(1),(2)) as t(x)
group by x
having x>11;
{code}
> Allow alias in having clause for aggregate functions
>
>
> Key: CALCITE-2799
> URL: https://issues.apache.org/jira/browse/CALCITE-2799
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.18.0
>Reporter: Arina Ielchiieva
>Assignee: Julian Hyde
>Priority: Major
> Fix For: 1.19.0
>
>
> Currently alias is not allowed in having for aggregate functions.
> MySql supports such cases and taking into account that alias in having clause
> is allowed only for the following conformance levels: MYSQL_5, LENIENT,
> BABEL, it makes sense to allow alias in having for aggregate functions.
> {noformat}
> /**
>* Whether to allow aliases from the {@code SELECT} clause to be used as
>* column names in the {@code HAVING} clause.
>*
>* Among the built-in conformance levels, true in
>* {@link SqlConformanceEnum#BABEL},
>* {@link SqlConformanceEnum#LENIENT},
>* {@link SqlConformanceEnum#MYSQL_5};
>* false otherwise.
>*/
> boolean isHavingAlias();
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-2799) Allow alias in having clause for aggregate functions
[
https://issues.apache.org/jira/browse/CALCITE-2799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16750547#comment-16750547
]
Volodymyr Vysotskyi commented on CALCITE-2799:
--
I agree with [~vladimirsitnikov] that the problem here is in the priorities of
alias and column for both having and group by clauses, and it is observed for a
more simple case:
{noformat}
drop table t;
\\
create table t (i int, j int);
\\
insert into t values (1, 1), (2, 3);
\\
select j * 0 as j
from t
group by j;
j
===
0
0
{noformat}
But regarding using aliases inside aggregate functions in having clauses, MySQL
really allows it:
{noformat}
drop table t;
\\
create table t (i int, j int);
\\
insert into t values (1, 1), (2, 3);
\\
select i, j + 10 as j1
from t
group by i
having max(j1) < 13;
i j1
=== ===
1 11
{noformat}
One more thing about the problem with aliases. MySQL uses either column or
alias if such column wasn't found. In Calcite for regular tables I suppose it
can be also implemented, but for dynamic tables, we don't know whether a table
has such a column.
> Allow alias in having clause for aggregate functions
>
>
> Key: CALCITE-2799
> URL: https://issues.apache.org/jira/browse/CALCITE-2799
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.18.0
>Reporter: Arina Ielchiieva
>Assignee: Julian Hyde
>Priority: Major
> Fix For: 1.19.0
>
>
> Currently alias is not allowed in having for aggregate functions.
> MySql supports such cases and taking into account that alias in having clause
> is allowed only for the following conformance levels: MYSQL_5, LENIENT,
> BABEL, it makes sense to allow alias in having for aggregate functions.
> {noformat}
> /**
>* Whether to allow aliases from the {@code SELECT} clause to be used as
>* column names in the {@code HAVING} clause.
>*
>* Among the built-in conformance levels, true in
>* {@link SqlConformanceEnum#BABEL},
>* {@link SqlConformanceEnum#LENIENT},
>* {@link SqlConformanceEnum#MYSQL_5};
>* false otherwise.
>*/
> boolean isHavingAlias();
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-2223) ProjectMergeRule is infinitely matched when is applied after ProjectReduceExpressionsRule
[
https://issues.apache.org/jira/browse/CALCITE-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16747562#comment-16747562
]
Volodymyr Vysotskyi commented on CALCITE-2223:
--
[~vladimirsitnikov], to reproduce this issue in Drill, Calcite should be built
from this branch: [https://github.com/vvysotskyi/calcite/tree/CALCITE-2223-18]
(it contains changes from your PR) and then Drill should be built from this
branch: [https://github.com/vvysotskyi/drill/tree/CALCITE-2223_merge] (it
contains commit which reverts the fix for DRILL-6212).
With these changes, {{TestProjectWithFunctions.testCastFunctions()}} test will
fail with OOM.
> ProjectMergeRule is infinitely matched when is applied after
> ProjectReduceExpressionsRule
> -
>
> Key: CALCITE-2223
> URL: https://issues.apache.org/jira/browse/CALCITE-2223
> Project: Calcite
> Issue Type: Bug
>Reporter: Volodymyr Vysotskyi
>Assignee: Julian Hyde
>Priority: Critical
> Attachments: heap_overview.png, provenance_contents.png
>
>
> For queries like this:
> {code:sql}
> select t1.f from (select cast(f as int) f, f from (select cast(f as int) f
> from (values('1')) t(f))) as t1
> {code}
> OOM is thrown when {{ProjectMergeRule}} is applied before applying
> {{ProjectReduceExpressionsRule}} in VolcanoPlanner.
> A simple test to reproduce this issue (in {{RelOptRulesTest}}):
> {code:java}
> @Test public void testOomProjectMergeRule() {
> RelBuilder relBuilder =
> RelBuilder.create(RelBuilderTest.config().build());
> RelNode relNode = relBuilder
> .values(new String[]{"f"}, "1")
> .project(
> relBuilder.alias(
> relBuilder.cast(relBuilder.field(0), SqlTypeName.INTEGER),
> "f"))
> .project(
> relBuilder.alias(
> relBuilder.cast(relBuilder.field(0), SqlTypeName.INTEGER),
> "f0"),
> relBuilder.alias(relBuilder.field(0), "f"))
> .project(
> relBuilder.alias(relBuilder.field(0), "f"))
> .build();
> RelOptPlanner planner = relNode.getCluster().getPlanner();
> RuleSet ruleSet =
> RuleSets.ofList(
> ReduceExpressionsRule.PROJECT_INSTANCE,
> new ProjectMergeRuleWithLongerName(),
> EnumerableRules.ENUMERABLE_PROJECT_RULE,
> EnumerableRules.ENUMERABLE_VALUES_RULE);
> Program program = Programs.of(ruleSet);
> RelTraitSet toTraits =
> relNode.getCluster().traitSet()
> .replace(0, EnumerableConvention.INSTANCE);
> RelNode output = program.run(planner, relNode, toTraits,
> ImmutableList.of(),
> ImmutableList.of());
> // check for output
> }
> /**
>* ProjectMergeRule inheritor which has
>* class name greater than ProjectReduceExpressionsRule class name
> (String.compareTo()).
>*
>* It is needed for RuleQueue.popMatch() method
>* to apply this rule before ProjectReduceExpressionsRule.
>*/
> private static class ProjectMergeRuleWithLongerName extends
> ProjectMergeRule {
> public ProjectMergeRuleWithLongerName() {
> super(true, RelFactories.LOGICAL_BUILDER);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-2223) ProjectMergeRule is infinitely matched when is applied after ProjectReduceExpressionsRule
[
https://issues.apache.org/jira/browse/CALCITE-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737141#comment-16737141
]
Volodymyr Vysotskyi commented on CALCITE-2223:
--
[~vladimirsitnikov], sorry, I thought that the case you have specified with
{{MaterializationTest}} reproduces the similar problem, therefore I didn't try
to reproduce it.
With this fix and reverted fix for DRILL-6212, a query from Jira description
still fails with OOM, but now it is reproduced with Drills
[{{DrillPushFilterPastProjectRule}}|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillPushFilterPastProjectRule.java]
rule, so it is possible that the problem is in the rule.
> ProjectMergeRule is infinitely matched when is applied after
> ProjectReduceExpressionsRule
> -
>
> Key: CALCITE-2223
> URL: https://issues.apache.org/jira/browse/CALCITE-2223
> Project: Calcite
> Issue Type: Bug
>Reporter: Volodymyr Vysotskyi
>Assignee: Julian Hyde
>Priority: Critical
> Attachments: heap_overview.png, provenance_contents.png
>
>
> For queries like this:
> {code:sql}
> select t1.f from (select cast(f as int) f, f from (select cast(f as int) f
> from (values('1')) t(f))) as t1
> {code}
> OOM is thrown when {{ProjectMergeRule}} is applied before applying
> {{ProjectReduceExpressionsRule}} in VolcanoPlanner.
> A simple test to reproduce this issue (in {{RelOptRulesTest}}):
> {code:java}
> @Test public void testOomProjectMergeRule() {
> RelBuilder relBuilder =
> RelBuilder.create(RelBuilderTest.config().build());
> RelNode relNode = relBuilder
> .values(new String[]{"f"}, "1")
> .project(
> relBuilder.alias(
> relBuilder.cast(relBuilder.field(0), SqlTypeName.INTEGER),
> "f"))
> .project(
> relBuilder.alias(
> relBuilder.cast(relBuilder.field(0), SqlTypeName.INTEGER),
> "f0"),
> relBuilder.alias(relBuilder.field(0), "f"))
> .project(
> relBuilder.alias(relBuilder.field(0), "f"))
> .build();
> RelOptPlanner planner = relNode.getCluster().getPlanner();
> RuleSet ruleSet =
> RuleSets.ofList(
> ReduceExpressionsRule.PROJECT_INSTANCE,
> new ProjectMergeRuleWithLongerName(),
> EnumerableRules.ENUMERABLE_PROJECT_RULE,
> EnumerableRules.ENUMERABLE_VALUES_RULE);
> Program program = Programs.of(ruleSet);
> RelTraitSet toTraits =
> relNode.getCluster().traitSet()
> .replace(0, EnumerableConvention.INSTANCE);
> RelNode output = program.run(planner, relNode, toTraits,
> ImmutableList.of(),
> ImmutableList.of());
> // check for output
> }
> /**
>* ProjectMergeRule inheritor which has
>* class name greater than ProjectReduceExpressionsRule class name
> (String.compareTo()).
>*
>* It is needed for RuleQueue.popMatch() method
>* to apply this rule before ProjectReduceExpressionsRule.
>*/
> private static class ProjectMergeRuleWithLongerName extends
> ProjectMergeRule {
> public ProjectMergeRuleWithLongerName() {
> super(true, RelFactories.LOGICAL_BUILDER);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (CALCITE-2463) Silence ERROR logs from CalciteException, SqlValidatorException
[ https://issues.apache.org/jira/browse/CALCITE-2463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi resolved CALCITE-2463. -- Resolution: Fixed Fixed in [a3711d7|https://github.com/apache/calcite/commit/a3711d78d450b7183a65e0ce695527ca92218bd8]. Thanks for the fix, [~vladimirsitnikov]. > Silence ERROR logs from CalciteException, SqlValidatorException > --- > > Key: CALCITE-2463 > URL: https://issues.apache.org/jira/browse/CALCITE-2463 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.17.0 >Reporter: Vladimir Sitnikov >Assignee: Julian Hyde >Priority: Major > Fix For: 1.19.0 > > > CalciteException and SqlValidatorException log exceptions at ERROR level. > This is not appropriate since > 1) the exception might be expected by the caller > 2) exception might contain sensitive information > 3) ERROR level is not appropriate, since error in validation does not imply > business-level error. > I suggest to remove `LOGGER.error` calls from Calcite code. > Logging exceptions at TRACE level is fine from my point of view, so client > can enable trace logging if required for debug purposes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CALCITE-2463) Silence ERROR logs from CalciteException, SqlValidatorException
[ https://issues.apache.org/jira/browse/CALCITE-2463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi updated CALCITE-2463: - Fix Version/s: 1.19.0 > Silence ERROR logs from CalciteException, SqlValidatorException > --- > > Key: CALCITE-2463 > URL: https://issues.apache.org/jira/browse/CALCITE-2463 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.17.0 >Reporter: Vladimir Sitnikov >Assignee: Julian Hyde >Priority: Major > Fix For: 1.19.0 > > > CalciteException and SqlValidatorException log exceptions at ERROR level. > This is not appropriate since > 1) the exception might be expected by the caller > 2) exception might contain sensitive information > 3) ERROR level is not appropriate, since error in validation does not imply > business-level error. > I suggest to remove `LOGGER.error` calls from Calcite code. > Logging exceptions at TRACE level is fine from my point of view, so client > can enable trace logging if required for debug purposes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (CALCITE-2745) RexSimplify fails with ClassCastException when the expression contains comparisons of operands with different types
[
https://issues.apache.org/jira/browse/CALCITE-2745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723361#comment-16723361
]
Volodymyr Vysotskyi edited comment on CALCITE-2745 at 12/17/18 9:11 PM:
Thanks for looking into this.
I agree that {{SqlToRelConverter}} should be responsible for adding casts,
where it is required. But for the case of dynamic record type, it is impossible
to determine whether cast should be added or not.
Is it acceptable to modify {{RexSimplify}} do not attempt to simplify only
expression parts which cannot be compared without casts? So it will allow
postponing the responsibility for the handling of expressions to the executors.
was (Author: vvysotskyi):
Thanks for looking into this.
I agree that {{SqlToRelConverter}} should be responsible for adding casts,
where it is required. But for the case of dynamic record type, it is impossible
to determine whether cast should be added or not.
Is it acceptable to modify {{RexSimplify}} do not attempt to simplify only
expression parts which be compared without casts? So it will allow postponing
the responsibility for the handling of expressions to the executors.
> RexSimplify fails with ClassCastException when the expression contains
> comparisons of operands with different types
> ---
>
> Key: CALCITE-2745
> URL: https://issues.apache.org/jira/browse/CALCITE-2745
> Project: Calcite
> Issue Type: Bug
>Reporter: Volodymyr Vysotskyi
>Assignee: Julian Hyde
>Priority: Major
>
> {{RexSimplify}} fails with {{ClassCastException}} when the expression
> contains predicates with comparisons of operands with different types.
> Test to reproduce this issue (placed into {{JdbcAdapterTest}}):
> {code:sql}
> @Test public void testSeveralTypesComparison() {
> CalciteAssert.model(JdbcTest.FOODMART_MODEL)
> .query("SELECT \"full_name\" FROM \"employee\" WHERE "
> + "\"employee_id\" = '1' and \"employee_id\" > 0")
> .returns("full_name=Sheri Nowmer\n");
> }
> {code}
> It fails with the error:
> {noformat}
> Caused by: java.lang.ClassCastException: org.apache.calcite.util.NlsString
> cannot be cast to java.math.BigDecimal
> at java.math.BigDecimal.compareTo(BigDecimal.java:220)
> at
> org.apache.calcite.rex.RexSimplify.processRange(RexSimplify.java:1780)
> at
> org.apache.calcite.rex.RexSimplify.simplifyAnd2ForUnknownAsFalse(RexSimplify.java:1242)
> at
> org.apache.calcite.rex.RexSimplify.simplifyAnd2ForUnknownAsFalse(RexSimplify.java:1109)
> at org.apache.calcite.rex.RexSimplify.simplifyAnds(RexSimplify.java:386)
> at
> org.apache.calcite.rex.RexSimplify.simplifyFilterPredicates(RexSimplify.java:2028)
> at org.apache.calcite.tools.RelBuilder.filter(RelBuilder.java:1048)
> at org.apache.calcite.tools.RelBuilder.filter(RelBuilder.java:1037)
> at
> org.apache.calcite.rel.rules.PushProjector.convertProject(PushProjector.java:382)
> at
> org.apache.calcite.rel.rules.ProjectFilterTransposeRule.onMatch(ProjectFilterTransposeRule.java:104)
> at
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:212)
> ... 41 more
> {noformat}
> When the query has any of the {{AND}} operands, or when
> {{ProjectFilterTransposeRule}} is not applied it is passed.
> Regarding the comparison of different data types, SQL spec says the following:
> {quote}8.2 Syntax Rules
> 3) The declared types of the corresponding fields of the two predicand>s shall be comparable.
> {quote}
> But it also allows implicit casts from numeric types to chars (6.13 specification>).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-2745) RexSimplify fails with ClassCastException when the expression contains comparisons of operands with different types
[
https://issues.apache.org/jira/browse/CALCITE-2745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723361#comment-16723361
]
Volodymyr Vysotskyi commented on CALCITE-2745:
--
Thanks for looking into this.
I agree that {{SqlToRelConverter}} should be responsible for adding casts,
where it is required. But for the case of dynamic record type, it is impossible
to determine whether cast should be added or not.
Is it acceptable to modify {{RexSimplify}} do not attempt to simplify only
expression parts which be compared without casts? So it will allow postponing
the responsibility for the handling of expressions to the executors.
> RexSimplify fails with ClassCastException when the expression contains
> comparisons of operands with different types
> ---
>
> Key: CALCITE-2745
> URL: https://issues.apache.org/jira/browse/CALCITE-2745
> Project: Calcite
> Issue Type: Bug
>Reporter: Volodymyr Vysotskyi
>Assignee: Julian Hyde
>Priority: Major
>
> {{RexSimplify}} fails with {{ClassCastException}} when the expression
> contains predicates with comparisons of operands with different types.
> Test to reproduce this issue (placed into {{JdbcAdapterTest}}):
> {code:sql}
> @Test public void testSeveralTypesComparison() {
> CalciteAssert.model(JdbcTest.FOODMART_MODEL)
> .query("SELECT \"full_name\" FROM \"employee\" WHERE "
> + "\"employee_id\" = '1' and \"employee_id\" > 0")
> .returns("full_name=Sheri Nowmer\n");
> }
> {code}
> It fails with the error:
> {noformat}
> Caused by: java.lang.ClassCastException: org.apache.calcite.util.NlsString
> cannot be cast to java.math.BigDecimal
> at java.math.BigDecimal.compareTo(BigDecimal.java:220)
> at
> org.apache.calcite.rex.RexSimplify.processRange(RexSimplify.java:1780)
> at
> org.apache.calcite.rex.RexSimplify.simplifyAnd2ForUnknownAsFalse(RexSimplify.java:1242)
> at
> org.apache.calcite.rex.RexSimplify.simplifyAnd2ForUnknownAsFalse(RexSimplify.java:1109)
> at org.apache.calcite.rex.RexSimplify.simplifyAnds(RexSimplify.java:386)
> at
> org.apache.calcite.rex.RexSimplify.simplifyFilterPredicates(RexSimplify.java:2028)
> at org.apache.calcite.tools.RelBuilder.filter(RelBuilder.java:1048)
> at org.apache.calcite.tools.RelBuilder.filter(RelBuilder.java:1037)
> at
> org.apache.calcite.rel.rules.PushProjector.convertProject(PushProjector.java:382)
> at
> org.apache.calcite.rel.rules.ProjectFilterTransposeRule.onMatch(ProjectFilterTransposeRule.java:104)
> at
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:212)
> ... 41 more
> {noformat}
> When the query has any of the {{AND}} operands, or when
> {{ProjectFilterTransposeRule}} is not applied it is passed.
> Regarding the comparison of different data types, SQL spec says the following:
> {quote}8.2 Syntax Rules
> 3) The declared types of the corresponding fields of the two predicand>s shall be comparable.
> {quote}
> But it also allows implicit casts from numeric types to chars (6.13 specification>).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (CALCITE-2745) RexSimplify fails with ClassCastException when the expression contains comparisons of operands with different types
[
https://issues.apache.org/jira/browse/CALCITE-2745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723226#comment-16723226
]
Volodymyr Vysotskyi edited comment on CALCITE-2745 at 12/17/18 6:23 PM:
Similar problem, but with another method of {{RexSimplify}}:
This test
{code:java}
checkSimplifyUnchanged(and(gt(vInt(1), literal(10)), eq(vInt(1),
literal("2";
{code}
fails with
{noformat}
java.lang.ClassCastException: java.math.BigDecimal cannot be cast to
org.apache.calcite.util.NlsString
at org.apache.calcite.util.NlsString.compareTo(NlsString.java:45)
at com.google.common.collect.Range.compareOrThrow(Range.java:672)
at com.google.common.collect.Cut.compareTo(Cut.java:79)
at com.google.common.collect.Range.encloses(Range.java:497)
at org.apache.calcite.rex.RexSimplify.residue(RexSimplify.java:1413)
at
org.apache.calcite.rex.RexSimplify.simplifyUsingPredicates(RexSimplify.java:1349)
at
org.apache.calcite.rex.RexSimplify.simplifyComparison(RexSimplify.java:366)
at
org.apache.calcite.rex.RexSimplify.simplifyComparison(RexSimplify.java:294)
at org.apache.calcite.rex.RexSimplify.simplify(RexSimplify.java:285)
at
org.apache.calcite.rex.RexSimplify.simplifyAndTerms(RexSimplify.java:406)
at org.apache.calcite.rex.RexSimplify.simplifyAnd(RexSimplify.java:1031)
at org.apache.calcite.rex.RexSimplify.simplify(RexSimplify.java:257)
at
org.apache.calcite.rex.RexSimplify.lambda$simplifyUnknownAs$0(RexSimplify.java:232)
at org.apache.calcite.rex.RexSimplify.verify(RexSimplify.java:1486)
at
org.apache.calcite.rex.RexSimplify.simplifyUnknownAs(RexSimplify.java:231)
at
org.apache.calcite.test.RexProgramTest.checkSimplify3_(RexProgramTest.java:182)
at
org.apache.calcite.test.RexProgramTest.checkSimplify2(RexProgramTest.java:157)
at
org.apache.calcite.test.RexProgramTest.testSimplifyAndNot(RexProgramTest.java:2415)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at
com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
at
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
{noformat}
was (Author: vvysotskyi):
Similar problem, but with another method of {{RexSimplify}}:
This test
{code:java}
checkSimplify2(and(gt(vInt(1), literal(1)), eq(vInt(1), literal("2"))),
"AND(>(?0.int1, 1), =(?0.int1, '2'))",
"=(?0.int1, '2')");
{code}
fails with
{noformat}
java.lang.ClassCastException: java.math.BigDecimal cannot be cast to
org.apache.calcite.util.NlsString
at org.apache.calcite.util.NlsString.compareTo(NlsString.java:45)
at com.google.common.collect.Range.compareOrThrow(Range.java:672)
at com.google.common.collect.Cut.compareTo(Cut.java:79)
at com.google.common.collect.Range.encloses(Range.java:497)
at org.apache.calcite.rex.RexSimplify.residue(RexSimplify.java:1413)
at
org.apache.calcite.rex.RexSimplify.simplifyUsingPredicates(RexSimplify.java:1349)
at
org.apache.calcite.rex.RexSimplify.simplifyComparison(RexSimplify.java:366)
[jira] [Commented] (CALCITE-2745) RexSimplify fails with ClassCastException when the expression contains comparisons of operands with different types
[
https://issues.apache.org/jira/browse/CALCITE-2745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723226#comment-16723226
]
Volodymyr Vysotskyi commented on CALCITE-2745:
--
Similar problem, but with another method of {{RexSimplify}}:
This test
{code:java}
checkSimplify2(and(gt(vInt(1), literal(1)), eq(vInt(1), literal("2"))),
"AND(>(?0.int1, 1), =(?0.int1, '2'))",
"=(?0.int1, '2')");
{code}
fails with
{noformat}
java.lang.ClassCastException: java.math.BigDecimal cannot be cast to
org.apache.calcite.util.NlsString
at org.apache.calcite.util.NlsString.compareTo(NlsString.java:45)
at com.google.common.collect.Range.compareOrThrow(Range.java:672)
at com.google.common.collect.Cut.compareTo(Cut.java:79)
at com.google.common.collect.Range.encloses(Range.java:497)
at org.apache.calcite.rex.RexSimplify.residue(RexSimplify.java:1413)
at
org.apache.calcite.rex.RexSimplify.simplifyUsingPredicates(RexSimplify.java:1349)
at
org.apache.calcite.rex.RexSimplify.simplifyComparison(RexSimplify.java:366)
at
org.apache.calcite.rex.RexSimplify.simplifyComparison(RexSimplify.java:294)
at org.apache.calcite.rex.RexSimplify.simplify(RexSimplify.java:285)
at
org.apache.calcite.rex.RexSimplify.simplifyAndTerms(RexSimplify.java:406)
at org.apache.calcite.rex.RexSimplify.simplifyAnd(RexSimplify.java:1031)
at org.apache.calcite.rex.RexSimplify.simplify(RexSimplify.java:257)
at
org.apache.calcite.rex.RexSimplify.lambda$simplifyUnknownAs$0(RexSimplify.java:232)
at org.apache.calcite.rex.RexSimplify.verify(RexSimplify.java:1486)
at
org.apache.calcite.rex.RexSimplify.simplifyUnknownAs(RexSimplify.java:231)
at
org.apache.calcite.test.RexProgramTest.checkSimplify3_(RexProgramTest.java:182)
at
org.apache.calcite.test.RexProgramTest.checkSimplify2(RexProgramTest.java:157)
at
org.apache.calcite.test.RexProgramTest.testSimplifyAndNot(RexProgramTest.java:2415)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at
com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
at
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
{noformat}
> RexSimplify fails with ClassCastException when the expression contains
> comparisons of operands with different types
> ---
>
> Key: CALCITE-2745
> URL: https://issues.apache.org/jira/browse/CALCITE-2745
> Project: Calcite
> Issue Type: Bug
>Reporter: Volodymyr Vysotskyi
>Assignee: Julian Hyde
>Priority: Major
>
> {{RexSimplify}} fails with {{ClassCastException}} when the expression
> contains predicates with comparisons of operands with different types.
> Test to reproduce this issue (placed into {{JdbcAdapterTest}}):
> {code:sql}
> @Test public void testSeveralTypesComparison() {
> CalciteAssert.model(JdbcTest.FOODMART_MODEL)
> .query("SELECT \"full_name\" FROM \"employee\" WHERE "
>
[jira] [Created] (CALCITE-2745) RexSimplify fails with ClassCastException when the expression contains comparisons of operands with different types
Volodymyr Vysotskyi created CALCITE-2745:
Summary: RexSimplify fails with ClassCastException when the
expression contains comparisons of operands with different types
Key: CALCITE-2745
URL: https://issues.apache.org/jira/browse/CALCITE-2745
Project: Calcite
Issue Type: Bug
Reporter: Volodymyr Vysotskyi
Assignee: Julian Hyde
{{RexSimplify}} fails with {{ClassCastException}} when the expression contains
predicates with comparisons of operands with different types.
Test to reproduce this issue (placed into {{JdbcAdapterTest}}):
{code:sql}
@Test public void testSeveralTypesComparison() {
CalciteAssert.model(JdbcTest.FOODMART_MODEL)
.query("SELECT \"full_name\" FROM \"employee\" WHERE "
+ "\"employee_id\" = '1' and \"employee_id\" > 0")
.returns("full_name=Sheri Nowmer\n");
}
{code}
It fails with the error:
{noformat}
Caused by: java.lang.ClassCastException: org.apache.calcite.util.NlsString
cannot be cast to java.math.BigDecimal
at java.math.BigDecimal.compareTo(BigDecimal.java:220)
at
org.apache.calcite.rex.RexSimplify.processRange(RexSimplify.java:1780)
at
org.apache.calcite.rex.RexSimplify.simplifyAnd2ForUnknownAsFalse(RexSimplify.java:1242)
at
org.apache.calcite.rex.RexSimplify.simplifyAnd2ForUnknownAsFalse(RexSimplify.java:1109)
at org.apache.calcite.rex.RexSimplify.simplifyAnds(RexSimplify.java:386)
at
org.apache.calcite.rex.RexSimplify.simplifyFilterPredicates(RexSimplify.java:2028)
at org.apache.calcite.tools.RelBuilder.filter(RelBuilder.java:1048)
at org.apache.calcite.tools.RelBuilder.filter(RelBuilder.java:1037)
at
org.apache.calcite.rel.rules.PushProjector.convertProject(PushProjector.java:382)
at
org.apache.calcite.rel.rules.ProjectFilterTransposeRule.onMatch(ProjectFilterTransposeRule.java:104)
at
org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:212)
... 41 more
{noformat}
When the query has any of the {{AND}} operands, or when
{{ProjectFilterTransposeRule}} is not applied it is passed.
Regarding the comparison of different data types, SQL spec says the following:
{quote}8.2 Syntax Rules
3) The declared types of the corresponding fields of the two s shall be comparable.
{quote}
But it also allows implicit casts from numeric types to chars (6.13 ).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-2507) Aliasing a star in select list should throw a parsing error.
[
https://issues.apache.org/jira/browse/CALCITE-2507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1586#comment-1586
]
Volodymyr Vysotskyi commented on CALCITE-2507:
--
[~michaelmior], it was just an example of a query with several levels of schema
nesting to show that Calcite allows it. And the same query, but with the alias
will fail at the validation stage, not at the parsing one even with the
proposed fix.
> Aliasing a star in select list should throw a parsing error.
> -
>
> Key: CALCITE-2507
> URL: https://issues.apache.org/jira/browse/CALCITE-2507
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.17.0
>Reporter: Hanumath Rao Maduri
>Assignee: Julian Hyde
>Priority: Minor
> Fix For: 1.18.0
>
>
> Currently aliasing a Star column is not throwing an exception in parser.
> There is a TODO for a simple test case (in SqlParserTest.java) to throw an
> error in parser phase itself.
> Change the parser grammar so as to report an error in parsing phase itself.
> {code:java}
> // TODO: should fail in parser
> @Test public void testStarAsFails() {
> sql("select * as x from emp")
> .ok("SELECT * AS `X`\n"
> + "FROM `EMP`");
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (CALCITE-2158) SubQuery with EXISTS clause creates redundant aggregate call
[
https://issues.apache.org/jira/browse/CALCITE-2158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi resolved CALCITE-2158.
--
Resolution: Won't Fix
Fix Version/s: 1.18.0
> SubQuery with EXISTS clause creates redundant aggregate call
>
>
> Key: CALCITE-2158
> URL: https://issues.apache.org/jira/browse/CALCITE-2158
> Project: Calcite
> Issue Type: Bug
>Reporter: Volodymyr Vysotskyi
>Assignee: Julian Hyde
>Priority: Major
> Fix For: 1.18.0
>
>
> When {{SqlToRelConverter.Config.isExpand()}} returns true, subqueries are
> expanded in {{SqlToRelConverter}}.
> Then for the queries, like this:
> {code:sql}
> SELECT cs1.sal
> FROM emp cs1
> WHERE EXISTS
> (SELECT *
> FROM emp cs2
> WHERE cs1.sal = cs2.sal
>AND cs1.deptno <> cs2.deptno)
> {code}
> Calcite returns logical plan with excessive aggregate calls:
> {noformat}
> LogicalProject(SAL=[$5])
> LogicalFilter(condition=[IS NOT NULL($9)])
> LogicalCorrelate(correlation=[$cor0], joinType=[left],
> requiredColumns=[{5, 7}])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> LogicalAggregate(group=[{}], agg#0=[MIN($0)])
> LogicalProject($f0=[true])
> LogicalProject(EMPNO=[$0], ENAME=[$1], JOB=[$2], MGR=[$3],
> HIREDATE=[$4], SAL=[$5], COMM=[$6], DEPTNO=[$7], SLACKER=[$8])
> LogicalFilter(condition=[AND(=($cor0.SAL, $5), <>($cor0.DEPTNO,
> $7))])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {noformat}
> But when {{SqlToRelConverter.Config.isExpand()}} returns false and
> SubQueryRemoveRule rules are applied to the logical plan with RexSubQuery,
> the resulting logical plan is correct and does not contain excessive
> aggregate calls:
> {noformat}
> LogicalProject(SAL=[$5])
> LogicalProject(EMPNO=[$0], ENAME=[$1], JOB=[$2], MGR=[$3], HIREDATE=[$4],
> SAL=[$5], COMM=[$6], DEPTNO=[$7], SLACKER=[$8])
> LogicalCorrelate(correlation=[$cor0], joinType=[inner],
> requiredColumns=[{5, 7}])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> LogicalAggregate(group=[{0}])
> LogicalProject(i=[true])
> LogicalFilter(condition=[AND(=($cor0.SAL, $5), <>($cor0.DEPTNO,
> $7))])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {noformat}
> These cases may be observed using this unit test:
> {code:java}
> @Test public void testExistsExpand() {
> final HepProgram preProgram = HepProgram.builder()
> .addRuleInstance(SubQueryRemoveRule.FILTER)
> .addRuleInstance(SubQueryRemoveRule.PROJECT)
> .addRuleInstance(SubQueryRemoveRule.JOIN)
> .build();
> final HepProgram program = HepProgram.builder()
> .build();
> final String sql = "SELECT cs1.sal\n"
> + "FROM emp cs1 \n"
> + "WHEREEXISTS\n"
> + "(SELECT *\n"
> + "FROM emp cs2\n"
> + "WHERE cs1.sal = cs2.sal\n"
> + "ANDcs1.deptno <> cs2.deptno)";
> sql(sql)
> .withDecorrelation(false)
> .withTrim(false)
> .expand(true) // change to false
> .withPre(preProgram)
> .with(program)
> .checkUnchanged();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (CALCITE-2639) FilterReduceExpressionsRule causes ArithmeticException at execution time
[
https://issues.apache.org/jira/browse/CALCITE-2639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi updated CALCITE-2639:
-
Priority: Blocker (was: Major)
> FilterReduceExpressionsRule causes ArithmeticException at execution time
>
>
> Key: CALCITE-2639
> URL: https://issues.apache.org/jira/browse/CALCITE-2639
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.17.0
>Reporter: Igor Guzenko
>Assignee: Julian Hyde
>Priority: Blocker
>
> Consider logical plan generated for test case(put this in
> {{RelOptRuleTest}}):
>
> {code:java}
> @Test public void testOversimplifiedCaseStatement() {
> HepProgram program = new HepProgramBuilder()
> .addRuleInstance(ReduceExpressionsRule.FILTER_INSTANCE)
> .build();
> String sql = "select * from emp "
> + "where MGR > 0 and "
> + "case when MGR > 0 then deptno / MGR else null end > 1";
> checkPlanning(program, sql);
> }
> {code}
> Before applying ReduceExpressionsRule.FILTER_INSTANCE rule, query plan is
>
> {code:java}
> LogicalProject(EMPNO=[$0], ENAME=[$1], JOB=[$2], MGR=[$3], HIREDATE=[$4],
> SAL=[$5], COMM=[$6], DEPTNO=[$7], SLACKER=[$8])
> LogicalFilter(condition=[AND(>($3, 0), >(CASE(>($3, 0), /($7, $3), null),
> 1))])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {code}
> Plan after applying the rule:
> {code:java}
> LogicalProject(EMPNO=[$0], ENAME=[$1], JOB=[$2], MGR=[$3], HIREDATE=[$4],
> SAL=[$5], COMM=[$6], DEPTNO=[$7], SLACKER=[$8])
> LogicalFilter(condition=[AND(>($3, 0), CASE(IS NOT NULL($3), >(/($7, $3),
> 1), false))])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {code}
> Here {{LogicalFilter}} has condition {{AND(>($3, 0), CASE(IS NOT NULL($3),
> >(/($7, $3), 1), false))}} where {{CASE}} condition was replaced by {{IS NOT
> NULL($3)}} condition. Since {{AND}} allows permutations for operands, the
> first operand may become {{CASE}}, so query may fail with
> {{ArithmeticException: / by zero}} error at execution stage.
> The regression was caused by
> [CALCITE-1413|https://github.com/apache/calcite/commit/b470a0cd4572c9f6c4c0e9b51926b97c5af58d3f].
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-2639) FilterReduceExpressionsRule causes ArithmeticException at execution time
[
https://issues.apache.org/jira/browse/CALCITE-2639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662060#comment-16662060
]
Volodymyr Vysotskyi commented on CALCITE-2639:
--
[~kgyrtkirk], since this bug was caused by your changes, could you please take
a look?
> FilterReduceExpressionsRule causes ArithmeticException at execution time
>
>
> Key: CALCITE-2639
> URL: https://issues.apache.org/jira/browse/CALCITE-2639
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.17.0
>Reporter: Igor Guzenko
>Assignee: Julian Hyde
>Priority: Major
>
> Consider logical plan generated for test case(put this in
> {{RelOptRuleTest}}):
>
> {code:java}
> @Test public void testOversimplifiedCaseStatement() {
> HepProgram program = new HepProgramBuilder()
> .addRuleInstance(ReduceExpressionsRule.FILTER_INSTANCE)
> .build();
> String sql = "select * from emp "
> + "where MGR > 0 and "
> + "case when MGR > 0 then deptno / MGR else null end > 1";
> checkPlanning(program, sql);
> }
> {code}
> Before applying ReduceExpressionsRule.FILTER_INSTANCE rule, query plan is
>
> {code:java}
> LogicalProject(EMPNO=[$0], ENAME=[$1], JOB=[$2], MGR=[$3], HIREDATE=[$4],
> SAL=[$5], COMM=[$6], DEPTNO=[$7], SLACKER=[$8])
> LogicalFilter(condition=[AND(>($3, 0), >(CASE(>($3, 0), /($7, $3), null),
> 1))])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {code}
> Plan after applying the rule:
> {code:java}
> LogicalProject(EMPNO=[$0], ENAME=[$1], JOB=[$2], MGR=[$3], HIREDATE=[$4],
> SAL=[$5], COMM=[$6], DEPTNO=[$7], SLACKER=[$8])
> LogicalFilter(condition=[AND(>($3, 0), CASE(IS NOT NULL($3), >(/($7, $3),
> 1), false))])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {code}
> Here {{LogicalFilter}} has condition {{AND(>($3, 0), CASE(IS NOT NULL($3),
> >(/($7, $3), 1), false))}} where {{CASE}} condition was replaced by {{IS NOT
> NULL($3)}} condition. Since {{AND}} allows permutations for operands, the
> first operand may become {{CASE}}, so query may fail with
> {{ArithmeticException: / by zero}} error at execution stage.
> The regression was caused by
> [CALCITE-1413|https://github.com/apache/calcite/commit/b470a0cd4572c9f6c4c0e9b51926b97c5af58d3f].
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-2507) Aliasing a star in select list should throw a parsing error.
[
https://issues.apache.org/jira/browse/CALCITE-2507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630295#comment-16630295
]
Volodymyr Vysotskyi commented on CALCITE-2507:
--
This PR does not handle a case when column name is fully specified (including
schema and table names) with several levels of schema nesting.
Calcite allows such queries, for example, the next test put into
{{SqlValidatorTest.java}} passes
{code:java}
@Test
public void testFullyQualified() {
sql("select catalog.sales.nest.emp_r.* from catalog.sales.nest.emp_r")
.ok();
}
{code}
> Aliasing a star in select list should throw a parsing error.
> -
>
> Key: CALCITE-2507
> URL: https://issues.apache.org/jira/browse/CALCITE-2507
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.17.0
>Reporter: Hanumath Rao Maduri
>Assignee: Julian Hyde
>Priority: Minor
> Fix For: 1.18.0
>
>
> Currently aliasing a Star column is not throwing an exception in parser.
> There is a TODO for a simple test case (in SqlParserTest.java) to throw an
> error in parser phase itself.
> Change the parser grammar so as to report an error in parsing phase itself.
> {code:java}
> // TODO: should fail in parser
> @Test public void testStarAsFails() {
> sql("select * as x from emp")
> .ok("SELECT * AS `X`\n"
> + "FROM `EMP`");
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (CALCITE-2223) ProjectMergeRule is infinitely matched when is applied after ProjectReduceExpressionsRule
[
https://issues.apache.org/jira/browse/CALCITE-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16625226#comment-16625226
]
Volodymyr Vysotskyi edited comment on CALCITE-2223 at 9/23/18 7:49 PM:
---
[~vladimirsitnikov], thanks for looking into this issue. I have checked your
fix and it solves the problem for {{ProjectMergeRule}} rule for the specified
plan. But the similar problem with {{ProjectMergeRule}} rule is observed for
other types of the queries (for example similar to the query from the
DRILL-6212 description). Also, rule {{FilterSetOpTransposeRule}} is also
affected by the same bug.
Will try to specify the test cases for this.
was (Author: vvysotskyi):
[~vladimirsitnikov], thanks for looking into this issue. I have checked your
fix and it solves the problem for {{ProjectMergeRule}} rule. But the similar
problem is observed for {{FilterSetOpTransposeRule}} even with this fix. Will
try to specify the test case for this.
> ProjectMergeRule is infinitely matched when is applied after
> ProjectReduceExpressionsRule
> -
>
> Key: CALCITE-2223
> URL: https://issues.apache.org/jira/browse/CALCITE-2223
> Project: Calcite
> Issue Type: Bug
>Reporter: Volodymyr Vysotskyi
>Assignee: Julian Hyde
>Priority: Critical
>
> For queries like this:
> {code:sql}
> select t1.f from (select cast(f as int) f, f from (select cast(f as int) f
> from (values('1')) t(f))) as t1
> {code}
> OOM is thrown when {{ProjectMergeRule}} is applied before applying
> {{ProjectReduceExpressionsRule}} in VolcanoPlanner.
> A simple test to reproduce this issue (in {{RelOptRulesTest}}):
> {code:java}
> @Test public void testOomProjectMergeRule() {
> RelBuilder relBuilder =
> RelBuilder.create(RelBuilderTest.config().build());
> RelNode relNode = relBuilder
> .values(new String[]{"f"}, "1")
> .project(
> relBuilder.alias(
> relBuilder.cast(relBuilder.field(0), SqlTypeName.INTEGER),
> "f"))
> .project(
> relBuilder.alias(
> relBuilder.cast(relBuilder.field(0), SqlTypeName.INTEGER),
> "f0"),
> relBuilder.alias(relBuilder.field(0), "f"))
> .project(
> relBuilder.alias(relBuilder.field(0), "f"))
> .build();
> RelOptPlanner planner = relNode.getCluster().getPlanner();
> RuleSet ruleSet =
> RuleSets.ofList(
> ReduceExpressionsRule.PROJECT_INSTANCE,
> new ProjectMergeRuleWithLongerName(),
> EnumerableRules.ENUMERABLE_PROJECT_RULE,
> EnumerableRules.ENUMERABLE_VALUES_RULE);
> Program program = Programs.of(ruleSet);
> RelTraitSet toTraits =
> relNode.getCluster().traitSet()
> .replace(0, EnumerableConvention.INSTANCE);
> RelNode output = program.run(planner, relNode, toTraits,
> ImmutableList.of(),
> ImmutableList.of());
> // check for output
> }
> /**
>* ProjectMergeRule inheritor which has
>* class name greater than ProjectReduceExpressionsRule class name
> (String.compareTo()).
>*
>* It is needed for RuleQueue.popMatch() method
>* to apply this rule before ProjectReduceExpressionsRule.
>*/
> private static class ProjectMergeRuleWithLongerName extends
> ProjectMergeRule {
> public ProjectMergeRuleWithLongerName() {
> super(true, RelFactories.LOGICAL_BUILDER);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-2223) ProjectMergeRule is infinitely matched when is applied after ProjectReduceExpressionsRule
[
https://issues.apache.org/jira/browse/CALCITE-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16625226#comment-16625226
]
Volodymyr Vysotskyi commented on CALCITE-2223:
--
[~vladimirsitnikov], thanks for looking into this issue. I have checked your
fix and it solves the problem for {{ProjectMergeRule}} rule. But the similar
problem is observed for {{FilterSetOpTransposeRule}} even with this fix. Will
try to specify the test case for this.
> ProjectMergeRule is infinitely matched when is applied after
> ProjectReduceExpressionsRule
> -
>
> Key: CALCITE-2223
> URL: https://issues.apache.org/jira/browse/CALCITE-2223
> Project: Calcite
> Issue Type: Bug
>Reporter: Volodymyr Vysotskyi
>Assignee: Julian Hyde
>Priority: Critical
>
> For queries like this:
> {code:sql}
> select t1.f from (select cast(f as int) f, f from (select cast(f as int) f
> from (values('1')) t(f))) as t1
> {code}
> OOM is thrown when {{ProjectMergeRule}} is applied before applying
> {{ProjectReduceExpressionsRule}} in VolcanoPlanner.
> A simple test to reproduce this issue (in {{RelOptRulesTest}}):
> {code:java}
> @Test public void testOomProjectMergeRule() {
> RelBuilder relBuilder =
> RelBuilder.create(RelBuilderTest.config().build());
> RelNode relNode = relBuilder
> .values(new String[]{"f"}, "1")
> .project(
> relBuilder.alias(
> relBuilder.cast(relBuilder.field(0), SqlTypeName.INTEGER),
> "f"))
> .project(
> relBuilder.alias(
> relBuilder.cast(relBuilder.field(0), SqlTypeName.INTEGER),
> "f0"),
> relBuilder.alias(relBuilder.field(0), "f"))
> .project(
> relBuilder.alias(relBuilder.field(0), "f"))
> .build();
> RelOptPlanner planner = relNode.getCluster().getPlanner();
> RuleSet ruleSet =
> RuleSets.ofList(
> ReduceExpressionsRule.PROJECT_INSTANCE,
> new ProjectMergeRuleWithLongerName(),
> EnumerableRules.ENUMERABLE_PROJECT_RULE,
> EnumerableRules.ENUMERABLE_VALUES_RULE);
> Program program = Programs.of(ruleSet);
> RelTraitSet toTraits =
> relNode.getCluster().traitSet()
> .replace(0, EnumerableConvention.INSTANCE);
> RelNode output = program.run(planner, relNode, toTraits,
> ImmutableList.of(),
> ImmutableList.of());
> // check for output
> }
> /**
>* ProjectMergeRule inheritor which has
>* class name greater than ProjectReduceExpressionsRule class name
> (String.compareTo()).
>*
>* It is needed for RuleQueue.popMatch() method
>* to apply this rule before ProjectReduceExpressionsRule.
>*/
> private static class ProjectMergeRuleWithLongerName extends
> ProjectMergeRule {
> public ProjectMergeRuleWithLongerName() {
> super(true, RelFactories.LOGICAL_BUILDER);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (CALCITE-2514) Add SqlIdentifier conversion to ITEM operator for dynamic tables in ExtendedExpander
[
https://issues.apache.org/jira/browse/CALCITE-2514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi resolved CALCITE-2514.
--
Resolution: Fixed
Fixed in
[d125495|https://github.com/apache/calcite/commit/d12549530c0aeeedbd6b03f5473c15029d93566f].
Thanks, [~arina] for fixing this issue!
> Add SqlIdentifier conversion to ITEM operator for dynamic tables in
> ExtendedExpander
>
>
> Key: CALCITE-2514
> URL: https://issues.apache.org/jira/browse/CALCITE-2514
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.17.0
>Reporter: Arina Ielchiieva
>Assignee: Julian Hyde
>Priority: Major
> Fix For: 1.18.0
>
>
> {{SqlIdentifier}} is not converted to ITEM operator for dynamic tables in
> {{ExtendedExpander}} and thus query returns wrong results. Happens when
> {{isGroupByAlias}} is set to true in SQL conformance.
> Query example:
> {noformat}
> select n_regionkey, max(min(n_nationkey)) over(partition by n_regionkey)
> from (select * from SALES.NATION)
> group by n_regionkey
> {noformat}
> Incorrect logical plan:
> {noformat}
> LogicalProject(N_REGIONKEY=[ITEM($0, 'N_REGIONKEY')], EXPR$1=[MAX($1) OVER
> (PARTITION BY ITEM($0, 'N_REGIONKEY') RANGE BETWEEN UNBOUNDED PRECEDING AND
> UNBOUNDED FOLLOWING)])
> LogicalAggregate(group=[{0}], agg#0=[MIN($1)])
> LogicalProject(**=[$0], $f1=[ITEM($0, 'N_NATIONKEY')])
> LogicalProject(**=[$0])
> LogicalTableScan(table=[[CATALOG, SALES, NATION]])
> {noformat}
> Correct logical plan:
> {noformat}
> LogicalProject(N_REGIONKEY=[$0], EXPR$1=[MAX($1) OVER (PARTITION BY $0 RANGE
> BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING)])
> LogicalAggregate(group=[{0}], agg#0=[MIN($1)])
> LogicalProject(N_REGIONKEY=[ITEM($0, 'N_REGIONKEY')], $f1=[ITEM($0,
> 'N_NATIONKEY')])
> LogicalProject(**=[$0])
> LogicalTableScan(table=[[CATALOG, SALES, NATION]])
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (CALCITE-2497) Update Janino version to 3.0.9
[ https://issues.apache.org/jira/browse/CALCITE-2497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi resolved CALCITE-2497. -- Resolution: Fixed Fixed in [02794e2|https://github.com/apache/calcite/commit/02794e2b9d55b2ff91e78e21eff105791921ea43] > Update Janino version to 3.0.9 > -- > > Key: CALCITE-2497 > URL: https://issues.apache.org/jira/browse/CALCITE-2497 > Project: Calcite > Issue Type: Task >Affects Versions: 1.17.0 >Reporter: Volodymyr Vysotskyi >Assignee: Volodymyr Vysotskyi >Priority: Minor > Fix For: 1.18.0 > > > Update Janino version to 3.0.9 where was fixed > [https://github.com/janino-compiler/janino/issues/47], and remove workaround > made in CALCITE-2261 because of this issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2497) Update Janino version to 3.0.9
Volodymyr Vysotskyi created CALCITE-2497: Summary: Update Janino version to 3.0.9 Key: CALCITE-2497 URL: https://issues.apache.org/jira/browse/CALCITE-2497 Project: Calcite Issue Type: Task Affects Versions: 1.17.0 Reporter: Volodymyr Vysotskyi Assignee: Volodymyr Vysotskyi Fix For: 1.18.0 Update Janino version to 3.0.9 where was fixed [https://github.com/janino-compiler/janino/issues/47], and remove workaround made in CALCITE-2261 because of this issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (CALCITE-2470) RelBuilder.project should combine expressions if underlying node is a Project
[
https://issues.apache.org/jira/browse/CALCITE-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594236#comment-16594236
]
Volodymyr Vysotskyi edited comment on CALCITE-2470 at 8/27/18 9:26 PM:
---
It would be good to add an ability somehow to disable this feature for
downstream projects. For example, add one more property to the {{RelBuilder}}
class, or use a method to detect that merging is allowed, so the underlying
project can override it in custom {{RelBuilder}}.
was (Author: vvysotskyi):
It would be good to add an ability somehow to disable this feature for
downstream projects. For example, add one more property to the {{RelBuilder}}
class, or use a method to detect that merging is allowed, so the underlying
method can override it.
> RelBuilder.project should combine expressions if underlying node is a Project
> -
>
> Key: CALCITE-2470
> URL: https://issues.apache.org/jira/browse/CALCITE-2470
> Project: Calcite
> Issue Type: Bug
>Reporter: Julian Hyde
>Assignee: Julian Hyde
>Priority: Major
> Fix For: 1.18.0
>
>
> The {{RelBuilder.project}} method should combine expressions if underlying
> node is a {{Project}}.
> For example, if {{r}} is a {{RelBuilder}} and its stack initially contains
> {code:java}
> Project(Scan(T), a, b, a + b as c){code}
> and we call (in pseudo-code)
> {code:java}
> r.project(a + c as d){code}
> then the result should be
> {code:java}
> Project(Scan(T), a + (a + b) as d){code}
> Today the result is
> {code:java}
> Project(Project(Scan(T), a, a + b as c), a + c as d){code}
> In some circumstances the result will be larger (i.e. contain more
> {{RexNode}} instances) but much more frequently the result will be smaller
> and simpler.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-2470) RelBuilder.project should combine expressions if underlying node is a Project
[
https://issues.apache.org/jira/browse/CALCITE-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594236#comment-16594236
]
Volodymyr Vysotskyi commented on CALCITE-2470:
--
It would be good to add an ability somehow to disable this feature for
downstream projects. For example, add one more property to the {{RelBuilder}}
class, or use a method to detect that merging is allowed, so the underlying
method can override it.
> RelBuilder.project should combine expressions if underlying node is a Project
> -
>
> Key: CALCITE-2470
> URL: https://issues.apache.org/jira/browse/CALCITE-2470
> Project: Calcite
> Issue Type: Bug
>Reporter: Julian Hyde
>Assignee: Julian Hyde
>Priority: Major
> Fix For: 1.18.0
>
>
> The {{RelBuilder.project}} method should combine expressions if underlying
> node is a {{Project}}.
> For example, if {{r}} is a {{RelBuilder}} and its stack initially contains
> {code:java}
> Project(Scan(T), a, b, a + b as c){code}
> and we call (in pseudo-code)
> {code:java}
> r.project(a + c as d){code}
> then the result should be
> {code:java}
> Project(Scan(T), a + (a + b) as d){code}
> Today the result is
> {code:java}
> Project(Project(Scan(T), a, a + b as c), a + c as d){code}
> In some circumstances the result will be larger (i.e. contain more
> {{RexNode}} instances) but much more frequently the result will be smaller
> and simpler.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-2470) RelBuilder.project should combine expressions if underlying node is a Project
[
https://issues.apache.org/jira/browse/CALCITE-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593732#comment-16593732
]
Volodymyr Vysotskyi commented on CALCITE-2470:
--
[~julianhyde], thanks for pointing to these tests. I have reviewed these
changes, all looks good, but I have a single point - the behavior of
{{RelBuilder.project}} method does not depend on the value of {{force}}
argument - for both cases if underlying rel node is a project, it will be
replaced with the new project.
I have checked these changes with Drill, and there are no issues with dynamic
tables, but exists more complex one: some tests with queries where the argument
of one function was another one, failed if the inner function returned complex
output (i.e map or array).
For example this query:
{code:sql}
select mycol['x'] as COMPLEX_COL from (select convert_from('{ x : [1,2], y :
100 }', 'JSON') as mycol) as foo(mycol)
{code}
cannot be executed in Drill with these changes because of merging two projects
with their function calls into a single one.
> RelBuilder.project should combine expressions if underlying node is a Project
> -
>
> Key: CALCITE-2470
> URL: https://issues.apache.org/jira/browse/CALCITE-2470
> Project: Calcite
> Issue Type: Bug
>Reporter: Julian Hyde
>Assignee: Julian Hyde
>Priority: Major
> Fix For: 1.18.0
>
>
> The {{RelBuilder.project}} method should combine expressions if underlying
> node is a {{Project}}.
> For example, if {{r}} is a {{RelBuilder}} and its stack initially contains
> {code:java}
> Project(Scan(T), a, b, a + b as c){code}
> and we call (in pseudo-code)
> {code:java}
> r.project(a + c as d){code}
> then the result should be
> {code:java}
> Project(Scan(T), a + (a + b) as d){code}
> Today the result is
> {code:java}
> Project(Project(Scan(T), a, a + b as c), a + c as d){code}
> In some circumstances the result will be larger (i.e. contain more
> {{RexNode}} instances) but much more frequently the result will be smaller
> and simpler.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-2440) SQL with UNNEST fail in volcano planner in some cases
[
https://issues.apache.org/jira/browse/CALCITE-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16585616#comment-16585616
]
Volodymyr Vysotskyi commented on CALCITE-2440:
--
I think the problem is that one of the rules which convert rel nodes to
required implementation with required trait set cannot be applied. I would
recommend revising all rules which create rel nodes that are available in the
resulting plan (before Calcite upgrade) and check if one of the rules is not
applied.
I assume you have a rule which expects {{Correlate -> Uncollect -> Project}}
section of the tree, but tree contains {{Project}} after {{Correlate}} and
before {{Uncollect}}.
> SQL with UNNEST fail in volcano planner in some cases
> -
>
> Key: CALCITE-2440
> URL: https://issues.apache.org/jira/browse/CALCITE-2440
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.17.0
>Reporter: Shuyi Chen
>Assignee: Julian Hyde
>Priority: Major
>
> After [CALCITE-2291|https://issues.apache.org/jira/browse/CALCITE-2291],
> specifically, [this
> line|https://github.com/apache/calcite/blob/7088dc7261d294b7c6d5c9f4463435e95f0b9c15/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L1952]
> causes volcano planner to fail to find the best plan for query
> {code:java}
> SELECT a, s FROM T, UNNEST(T.c) as A (s)
> {code}
> . I'll try to write a unittest in Calcite to reproduce. [~cshi] [~vvysotskyi]
> [~HanumathRao], can you provide some context? Thanks a lot.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-2440) SQL with UNNEST fail in volcano planner in some cases
[
https://issues.apache.org/jira/browse/CALCITE-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16584750#comment-16584750
]
Volodymyr Vysotskyi commented on CALCITE-2440:
--
[~suez1224], thanks for the clarification. I'm wondering, will this query pass
if you change the alias for unnest result:
{code:sql}
SELECT a, s FROM T, UNNEST(T.c) as AAA(s)
SELECT T.a, AAA.s FROM T, UNNEST(T.c) as AAA(s)
{code}
There is no direct unit test for this problem, but the side effect of this
change may be observed in this line:
[https://github.com/apache/calcite/commit/68ba411e23ba930bb2086bb3eed4c46edfac23eb#diff-337728f1a3a35c634362df7a0a509505R4546].
Regarding unit tests with volcano planner, you may use a test from the
description of CALCITE-2223 as an example. You just should specify rules which
should be applied, and perhaps more {{EnumerableRules}}, since the initial
query was much simpler. Also, there is built rel nodes tree manually, but you
can use {{SqlToRelConverter}} to specify a query only.
> SQL with UNNEST fail in volcano planner in some cases
> -
>
> Key: CALCITE-2440
> URL: https://issues.apache.org/jira/browse/CALCITE-2440
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.17.0
>Reporter: Shuyi Chen
>Assignee: Julian Hyde
>Priority: Major
>
> After [CALCITE-2291|https://issues.apache.org/jira/browse/CALCITE-2291],
> specifically, [this
> line|https://github.com/apache/calcite/blob/7088dc7261d294b7c6d5c9f4463435e95f0b9c15/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L1952]
> causes volcano planner to fail to find the best plan for query
> {code:java}
> SELECT a, s FROM T, UNNEST(T.c) as A (s)
> {code}
> . I'll try to write a unittest in Calcite to reproduce. [~cshi] [~vvysotskyi]
> [~HanumathRao], can you provide some context? Thanks a lot.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-2455) simplifyCoalesce of constant should match nullability
[
https://issues.apache.org/jira/browse/CALCITE-2455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574560#comment-16574560
]
Volodymyr Vysotskyi commented on CALCITE-2455:
--
Thanks for the explanation.
> simplifyCoalesce of constant should match nullability
> -
>
> Key: CALCITE-2455
> URL: https://issues.apache.org/jira/browse/CALCITE-2455
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.17.0
>Reporter: Yuzhao Chen
>Assignee: Julian Hyde
>Priority: Major
> Fix For: 1.18.0
>
>
> After CALCITE-2227, we will not replace Coalesce to Case When for natural
> join, when we use ReduceExpressionRule for our plan for Coalesce with
> constant, the nullability will chage( from nullable to notnull), this will
> cause VolcanoPlanner to throw error.
> Should i fix this? or we should not use this rule in VolcanoPlanner?
> This is the error thrown sql :
> {code:java}
> select * from lateral (select * from scott_emp) as e
> join (table scott_dept) using (deptno)
> where e.deptno = 10 {code}
> The plan before ReduceExpressionRule:
> {code:java}
> LogicalProject(deptno=[COALESCE($7, $8)], empno=[$0], ename=[$1], job=[$2],
> mgr=[$3], hiredate=[$4], sal=[$5], comm=[$6], dname=[$9], loc=[$10])
> +- LogicalFilter(condition=[=($7, 10)])
> +- LogicalJoin(condition=[=($7, $8)], joinType=[inner])
> :- LogicalProject(empno=[$0], ename=[$1], job=[$2], mgr=[$3], hiredate=[$4],
> sal=[$5], comm=[$6], deptno=[$7])
> : +- LogicalTableScan(table=[[scott_emp]])
> +- LogicalProject(deptno=[$0], dname=[$1], loc=[$2])
> +- LogicalTableScan(table=[[scott_dept]])
> {code}
> The plan after:
> {code:java}
> LogicalProject(deptno=[10], empno=[$0], ename=[$1], job=[$2], mgr=[$3],
> hiredate=[$4], sal=[$5], comm=[$6], dname=[$9], loc=[$10])
> +- LogicalFilter(condition=[=($7, 10)])
> +- LogicalJoin(condition=[=($7, $8)], joinType=[inner])
> :- LogicalProject(empno=[$0], ename=[$1], job=[$2], mgr=[$3], hiredate=[$4],
> sal=[$5], comm=[$6], deptno=[$7])
> : +- LogicalTableScan(table=[[scott_emp]])
> +- LogicalProject(deptno=[$0], dname=[$1], loc=[$2])
> +- LogicalTableScan(table=[[scott_dept]])
> {code}
> We can see that the deptno's nullability changes from nullable to not null.
> And we encounter an type error when using ReduceExpressionRule in
> VolcanoPlanner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-2455) simplifyCoalesce of constant should match nullability
[
https://issues.apache.org/jira/browse/CALCITE-2455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574369#comment-16574369
]
Volodymyr Vysotskyi commented on CALCITE-2455:
--
[~danny0405], I'm wondering why initially Coalesce was nullable, if it
shouldn't be.
> simplifyCoalesce of constant should match nullability
> -
>
> Key: CALCITE-2455
> URL: https://issues.apache.org/jira/browse/CALCITE-2455
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.17.0
>Reporter: Yuzhao Chen
>Assignee: Julian Hyde
>Priority: Major
> Fix For: 1.18.0
>
>
> After CALCITE-2227, we will not replace Coalesce to Case When for natural
> join, when we use ReduceExpressionRule for our plan for Coalesce with
> constant, the nullability will chage( from nullable to notnull), this will
> cause VolcanoPlanner to throw error.
> Should i fix this? or we should not use this rule in VolcanoPlanner?
> This is the error thrown sql :
> {code:java}
> select * from lateral (select * from scott_emp) as e
> join (table scott_dept) using (deptno)
> where e.deptno = 10 {code}
> The plan before ReduceExpressionRule:
> {code:java}
> LogicalProject(deptno=[COALESCE($7, $8)], empno=[$0], ename=[$1], job=[$2],
> mgr=[$3], hiredate=[$4], sal=[$5], comm=[$6], dname=[$9], loc=[$10])
> +- LogicalFilter(condition=[=($7, 10)])
> +- LogicalJoin(condition=[=($7, $8)], joinType=[inner])
> :- LogicalProject(empno=[$0], ename=[$1], job=[$2], mgr=[$3], hiredate=[$4],
> sal=[$5], comm=[$6], deptno=[$7])
> : +- LogicalTableScan(table=[[scott_emp]])
> +- LogicalProject(deptno=[$0], dname=[$1], loc=[$2])
> +- LogicalTableScan(table=[[scott_dept]])
> {code}
> The plan after:
> {code:java}
> LogicalProject(deptno=[10], empno=[$0], ename=[$1], job=[$2], mgr=[$3],
> hiredate=[$4], sal=[$5], comm=[$6], dname=[$9], loc=[$10])
> +- LogicalFilter(condition=[=($7, 10)])
> +- LogicalJoin(condition=[=($7, $8)], joinType=[inner])
> :- LogicalProject(empno=[$0], ename=[$1], job=[$2], mgr=[$3], hiredate=[$4],
> sal=[$5], comm=[$6], deptno=[$7])
> : +- LogicalTableScan(table=[[scott_emp]])
> +- LogicalProject(deptno=[$0], dname=[$1], loc=[$2])
> +- LogicalTableScan(table=[[scott_dept]])
> {code}
> We can see that the deptno's nullability changes from nullable to not null.
> And we encounter an type error when using ReduceExpressionRule in
> VolcanoPlanner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-2440) SQL with UNNEST fail in volcano planner in some cases
[
https://issues.apache.org/jira/browse/CALCITE-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567657#comment-16567657
]
Volodymyr Vysotskyi commented on CALCITE-2440:
--
[~suez1224], the line of code you have provided in the Jira description, helped
to fix a loss of field aliases for the cases of the queries where unnest was
used in a non-root level.
Could you please check a query you have provided? You are using column "a",
does table "T" have such column, or you are trying to reference result of
"unnest"?
> SQL with UNNEST fail in volcano planner in some cases
> -
>
> Key: CALCITE-2440
> URL: https://issues.apache.org/jira/browse/CALCITE-2440
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.17.0
>Reporter: Shuyi Chen
>Assignee: Julian Hyde
>Priority: Major
>
> After [CALCITE-2291|https://issues.apache.org/jira/browse/CALCITE-2291],
> specifically, [this
> line|https://github.com/apache/calcite/blob/7088dc7261d294b7c6d5c9f4463435e95f0b9c15/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L1952]
> causes volcano planner to fail to find the best plan for query
> {code:java}
> SELECT a, s FROM T, UNNEST(T.c) as A (s)
> {code}
> . I'll try to write a unittest in Calcite to reproduce. [~cshi] [~vvysotskyi]
> [~HanumathRao], can you provide some context? Thanks a lot.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (CALCITE-2422) Query with unnest of column from nested subquery fails when dynamic table is used
[
https://issues.apache.org/jira/browse/CALCITE-2422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi resolved CALCITE-2422.
--
Resolution: Fixed
Fixed in
[41a0677|https://github.com/apache/calcite/commit/41a06771876374474d46ef8d3a14886d742c66e9].
Thanks [~Sergey Nuyanzin] for the review!
> Query with unnest of column from nested subquery fails when dynamic table is
> used
> -
>
> Key: CALCITE-2422
> URL: https://issues.apache.org/jira/browse/CALCITE-2422
> Project: Calcite
> Issue Type: Bug
>Reporter: Volodymyr Vysotskyi
>Assignee: Julian Hyde
>Priority: Major
> Fix For: 1.18.0
>
>
> This test:
> {code:java}
> @Test public void testStarDynamicSchemaUnnestNestedSubquery() {
> String sql3 = "select * \n"
> + "from (select * from SALES.CUSTOMER) as t1,\n"
> + "unnest(t1.fake_col) as t2";
> sql(sql3).with(getTesterWithDynamicTable()).ok();
> }
> {code}
> placed in {{SqlToRelConverterTest}} fails with {{AssertionError}}:
> {noformat}
> java.lang.AssertionError
> at
> org.apache.calcite.sql.SqlUnnestOperator.inferReturnType(SqlUnnestOperator.java:81)
> at
> org.apache.calcite.sql.SqlOperator.validateOperands(SqlOperator.java:437)
> at
> org.apache.calcite.sql.validate.UnnestNamespace.validateImpl(UnnestNamespace.java:67)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:968)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:944)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:3032)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:3017)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateJoin(SqlValidatorImpl.java:3069)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:3026)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3275)
> at
> org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:968)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:944)
> at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:226)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:919)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:629)
> at
> org.apache.calcite.test.SqlToRelTestBase$TesterImpl.convertSqlToRel(SqlToRelTestBase.java:642)
> at
> org.apache.calcite.test.SqlToRelTestBase$TesterImpl.assertConvertsTo(SqlToRelTestBase.java:754)
> at
> org.apache.calcite.test.SqlToRelConverterTest$Sql.convertsTo(SqlToRelConverterTest.java:2905)
> at
> org.apache.calcite.test.SqlToRelConverterTest$Sql.ok(SqlToRelConverterTest.java:2897)
> at
> org.apache.calcite.test.SqlToRelConverterTest.testStarDynamicSchemaUnnestNestedSubquery(SqlToRelConverterTest.java:2530)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at
[jira] [Commented] (CALCITE-2422) Query with unnest of column from nested subquery fails when dynamic table is used
[
https://issues.apache.org/jira/browse/CALCITE-2422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558486#comment-16558486
]
Volodymyr Vysotskyi commented on CALCITE-2422:
--
I have created PR with the fix for this issue:
[PR-768|https://github.com/apache/calcite/pull/768]. Could someone please
review it?
In this PR added expansion of every unnest operand before producing the
validation. For the case of the query, specified above with dynamic table, will
be created ITEM call to fetch the desired column from the DYNAMIC_STAR.
> Query with unnest of column from nested subquery fails when dynamic table is
> used
> -
>
> Key: CALCITE-2422
> URL: https://issues.apache.org/jira/browse/CALCITE-2422
> Project: Calcite
> Issue Type: Bug
>Reporter: Volodymyr Vysotskyi
>Assignee: Julian Hyde
>Priority: Major
> Fix For: 1.18.0
>
>
> This test:
> {code:java}
> @Test public void testStarDynamicSchemaUnnestNestedSubquery() {
> String sql3 = "select * \n"
> + "from (select * from SALES.CUSTOMER) as t1,\n"
> + "unnest(t1.fake_col) as t2";
> sql(sql3).with(getTesterWithDynamicTable()).ok();
> }
> {code}
> placed in {{SqlToRelConverterTest}} fails with {{AssertionError}}:
> {noformat}
> java.lang.AssertionError
> at
> org.apache.calcite.sql.SqlUnnestOperator.inferReturnType(SqlUnnestOperator.java:81)
> at
> org.apache.calcite.sql.SqlOperator.validateOperands(SqlOperator.java:437)
> at
> org.apache.calcite.sql.validate.UnnestNamespace.validateImpl(UnnestNamespace.java:67)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:968)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:944)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:3032)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:3017)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateJoin(SqlValidatorImpl.java:3069)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:3026)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3275)
> at
> org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:968)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:944)
> at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:226)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:919)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:629)
> at
> org.apache.calcite.test.SqlToRelTestBase$TesterImpl.convertSqlToRel(SqlToRelTestBase.java:642)
> at
> org.apache.calcite.test.SqlToRelTestBase$TesterImpl.assertConvertsTo(SqlToRelTestBase.java:754)
> at
> org.apache.calcite.test.SqlToRelConverterTest$Sql.convertsTo(SqlToRelConverterTest.java:2905)
> at
> org.apache.calcite.test.SqlToRelConverterTest$Sql.ok(SqlToRelConverterTest.java:2897)
> at
> org.apache.calcite.test.SqlToRelConverterTest.testStarDynamicSchemaUnnestNestedSubquery(SqlToRelConverterTest.java:2530)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at
[jira] [Created] (CALCITE-2422) Query with unnest of column from nested subquery fails when dynamic table is used
Volodymyr Vysotskyi created CALCITE-2422:
Summary: Query with unnest of column from nested subquery fails
when dynamic table is used
Key: CALCITE-2422
URL: https://issues.apache.org/jira/browse/CALCITE-2422
Project: Calcite
Issue Type: Bug
Reporter: Volodymyr Vysotskyi
Assignee: Julian Hyde
This test:
{code:java}
@Test public void testStarDynamicSchemaUnnestNestedSubquery() {
String sql3 = "select * \n"
+ "from (select * from SALES.CUSTOMER) as t1,\n"
+ "unnest(t1.fake_col) as t2";
sql(sql3).with(getTesterWithDynamicTable()).ok();
}
{code}
placed in {{SqlToRelConverterTest}} fails with {{AssertionError}}:
{noformat}
java.lang.AssertionError
at
org.apache.calcite.sql.SqlUnnestOperator.inferReturnType(SqlUnnestOperator.java:81)
at
org.apache.calcite.sql.SqlOperator.validateOperands(SqlOperator.java:437)
at
org.apache.calcite.sql.validate.UnnestNamespace.validateImpl(UnnestNamespace.java:67)
at
org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:968)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:944)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:3032)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:3017)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateJoin(SqlValidatorImpl.java:3069)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:3026)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3275)
at
org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60)
at
org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:968)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:944)
at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:226)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:919)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:629)
at
org.apache.calcite.test.SqlToRelTestBase$TesterImpl.convertSqlToRel(SqlToRelTestBase.java:642)
at
org.apache.calcite.test.SqlToRelTestBase$TesterImpl.assertConvertsTo(SqlToRelTestBase.java:754)
at
org.apache.calcite.test.SqlToRelConverterTest$Sql.convertsTo(SqlToRelConverterTest.java:2905)
at
org.apache.calcite.test.SqlToRelConverterTest$Sql.ok(SqlToRelConverterTest.java:2897)
at
org.apache.calcite.test.SqlToRelConverterTest.testStarDynamicSchemaUnnestNestedSubquery(SqlToRelConverterTest.java:2530)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at
com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
at
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at
[jira] [Closed] (CALCITE-2337) Release Calcite 1.17.0
[ https://issues.apache.org/jira/browse/CALCITE-2337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi closed CALCITE-2337. Resolved in release 1.17.0 (2018-07-20) > Release Calcite 1.17.0 > -- > > Key: CALCITE-2337 > URL: https://issues.apache.org/jira/browse/CALCITE-2337 > Project: Calcite > Issue Type: Task >Reporter: Volodymyr Vysotskyi >Assignee: Volodymyr Vysotskyi >Priority: Major > Fix For: 1.17.0 > > > Release Calcite 1.17.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (CALCITE-2337) Release Calcite 1.17.0
[ https://issues.apache.org/jira/browse/CALCITE-2337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi resolved CALCITE-2337. -- Resolution: Done > Release Calcite 1.17.0 > -- > > Key: CALCITE-2337 > URL: https://issues.apache.org/jira/browse/CALCITE-2337 > Project: Calcite > Issue Type: Task >Reporter: Volodymyr Vysotskyi >Assignee: Volodymyr Vysotskyi >Priority: Major > Fix For: 1.17.0 > > > Release Calcite 1.17.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (CALCITE-2331) Evaluation of predicate "(A or B) and C" fails for Elasticsearch adapter
[
https://issues.apache.org/jira/browse/CALCITE-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi resolved CALCITE-2331.
--
Resolution: Fixed
> Evaluation of predicate "(A or B) and C" fails for Elasticsearch adapter
>
>
> Key: CALCITE-2331
> URL: https://issues.apache.org/jira/browse/CALCITE-2331
> Project: Calcite
> Issue Type: Bug
> Components: elasticsearch-adapter
>Reporter: Andrei Sereda
>Assignee: Michael Mior
>Priority: Critical
> Fix For: 1.17.0
>
>
> h2. Queries that are working
> {code:java}
> select count(*) from "elastic" where _MAP['foo'] in ('1') and true;
> select count(*) from "elastic" where _MAP['foo'] ='1' and true;
> select count(*) from "elastic" where _MAP['foo'] in ('1', '2');
> {code}
> h2. Queries that are failing
> {code:java}
> select count(*) from "elastic" where _MAP['foo'] in ('1', '2') and true;
> select count(*) from "elastic" where true and _MAP['foo'] in ('1', '2');
> select count(*) from "elastic" where (_MAP['foo'] ='1' or _MAP['foo'] = '2')
> and true;
> {code}
> h2. Elastic Adapter Exception
> {quote}java.lang.AssertionError: cannot translate OR(=(ITEM($0, 'foo'), '1'),
> =(ITEM($0, 'foo'), '2'))
> at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$
> Translator.translateMatch2(ElasticsearchFilter.java:234)
> at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$
> Translator.translateAnd(ElasticsearchFilter.java:158)
> at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$
> Translator.translateOr(ElasticsearchFilter.java:115)
> at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$
> Translator.translateMatch(ElasticsearchFilter.java:101)
> at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$
> Translator.access$000(ElasticsearchFilter.java:87)
> at org.apache.calcite.adapter.elasticsearch.
> ElasticsearchFilter.implement(ElasticsearchFilter.java:80)
> at org.apache.calcite.adapter.elasticsearch.
> ElasticsearchRel$Implementor.visitChild(ElasticsearchRel.java:53)
> at org.apache.calcite.adapter.elasticsearch.
> ElasticsearchToEnumerableConverter.implement(ElasticsearchToEnumerableConve
> rter.java:71)
> at org.apache.calcite.adapter.enumerable.EnumerableRelImplementor.
> visitChild(EnumerableRelImplementor.java:98)
> at org.apache.calcite.adapter.enumerable.
> EnumerableAggregate.implement(EnumerableAggregate.java:106)
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-2331) Evaluation of predicate "(A or B) and C" fails for Elasticsearch adapter
[
https://issues.apache.org/jira/browse/CALCITE-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544526#comment-16544526
]
Volodymyr Vysotskyi commented on CALCITE-2331:
--
I have split this Jira into two parts to include fixed one into release notes.
> Evaluation of predicate "(A or B) and C" fails for Elasticsearch adapter
>
>
> Key: CALCITE-2331
> URL: https://issues.apache.org/jira/browse/CALCITE-2331
> Project: Calcite
> Issue Type: Bug
> Components: elasticsearch-adapter
>Reporter: Andrei Sereda
>Assignee: Michael Mior
>Priority: Critical
> Fix For: 1.17.0
>
>
> h2. Queries that are working
> {code:java}
> select count(*) from "elastic" where _MAP['foo'] in ('1') and true;
> select count(*) from "elastic" where _MAP['foo'] ='1' and true;
> select count(*) from "elastic" where _MAP['foo'] in ('1', '2');
> {code}
> h2. Queries that are failing
> {code:java}
> select count(*) from "elastic" where _MAP['foo'] in ('1', '2') and true;
> select count(*) from "elastic" where true and _MAP['foo'] in ('1', '2');
> select count(*) from "elastic" where (_MAP['foo'] ='1' or _MAP['foo'] = '2')
> and true;
> {code}
> h2. Elastic Adapter Exception
> {quote}java.lang.AssertionError: cannot translate OR(=(ITEM($0, 'foo'), '1'),
> =(ITEM($0, 'foo'), '2'))
> at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$
> Translator.translateMatch2(ElasticsearchFilter.java:234)
> at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$
> Translator.translateAnd(ElasticsearchFilter.java:158)
> at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$
> Translator.translateOr(ElasticsearchFilter.java:115)
> at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$
> Translator.translateMatch(ElasticsearchFilter.java:101)
> at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$
> Translator.access$000(ElasticsearchFilter.java:87)
> at org.apache.calcite.adapter.elasticsearch.
> ElasticsearchFilter.implement(ElasticsearchFilter.java:80)
> at org.apache.calcite.adapter.elasticsearch.
> ElasticsearchRel$Implementor.visitChild(ElasticsearchRel.java:53)
> at org.apache.calcite.adapter.elasticsearch.
> ElasticsearchToEnumerableConverter.implement(ElasticsearchToEnumerableConve
> rter.java:71)
> at org.apache.calcite.adapter.enumerable.EnumerableRelImplementor.
> visitChild(EnumerableRelImplementor.java:98)
> at org.apache.calcite.adapter.enumerable.
> EnumerableAggregate.implement(EnumerableAggregate.java:106)
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (CALCITE-2331) Evaluation of predicate "(A or B) and C" fails for Elasticsearch adapter
[
https://issues.apache.org/jira/browse/CALCITE-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi updated CALCITE-2331:
-
Fix Version/s: 1.17.0
> Evaluation of predicate "(A or B) and C" fails for Elasticsearch adapter
>
>
> Key: CALCITE-2331
> URL: https://issues.apache.org/jira/browse/CALCITE-2331
> Project: Calcite
> Issue Type: Bug
> Components: elasticsearch-adapter
>Reporter: Andrei Sereda
>Assignee: Michael Mior
>Priority: Critical
> Fix For: 1.17.0
>
>
> h2. Queries that are working
> {code:java}
> select count(*) from "elastic" where _MAP['foo'] in ('1') and true;
> select count(*) from "elastic" where _MAP['foo'] ='1' and true;
> select count(*) from "elastic" where _MAP['foo'] in ('1', '2');
> {code}
> h2. Queries that are failing
> {code:java}
> select count(*) from "elastic" where _MAP['foo'] in ('1', '2') and true;
> select count(*) from "elastic" where true and _MAP['foo'] in ('1', '2');
> select count(*) from "elastic" where (_MAP['foo'] ='1' or _MAP['foo'] = '2')
> and true;
> {code}
> h2. Elastic Adapter Exception
> {quote}java.lang.AssertionError: cannot translate OR(=(ITEM($0, 'foo'), '1'),
> =(ITEM($0, 'foo'), '2'))
> at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$
> Translator.translateMatch2(ElasticsearchFilter.java:234)
> at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$
> Translator.translateAnd(ElasticsearchFilter.java:158)
> at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$
> Translator.translateOr(ElasticsearchFilter.java:115)
> at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$
> Translator.translateMatch(ElasticsearchFilter.java:101)
> at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$
> Translator.access$000(ElasticsearchFilter.java:87)
> at org.apache.calcite.adapter.elasticsearch.
> ElasticsearchFilter.implement(ElasticsearchFilter.java:80)
> at org.apache.calcite.adapter.elasticsearch.
> ElasticsearchRel$Implementor.visitChild(ElasticsearchRel.java:53)
> at org.apache.calcite.adapter.elasticsearch.
> ElasticsearchToEnumerableConverter.implement(ElasticsearchToEnumerableConve
> rter.java:71)
> at org.apache.calcite.adapter.enumerable.EnumerableRelImplementor.
> visitChild(EnumerableRelImplementor.java:98)
> at org.apache.calcite.adapter.enumerable.
> EnumerableAggregate.implement(EnumerableAggregate.java:106)
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (CALCITE-2331) Evaluation of predicate "(A or B) and C" fails for Elasticsearch adapter
[
https://issues.apache.org/jira/browse/CALCITE-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi updated CALCITE-2331:
-
Description:
h2. Queries that are working
{code:java}
select count(*) from "elastic" where _MAP['foo'] in ('1') and true;
select count(*) from "elastic" where _MAP['foo'] ='1' and true;
select count(*) from "elastic" where _MAP['foo'] in ('1', '2');
{code}
h2. Queries that are failing
{code:java}
select count(*) from "elastic" where _MAP['foo'] in ('1', '2') and true;
select count(*) from "elastic" where true and _MAP['foo'] in ('1', '2');
select count(*) from "elastic" where (_MAP['foo'] ='1' or _MAP['foo'] = '2')
and true;
{code}
h2. Elastic Adapter Exception
{quote}java.lang.AssertionError: cannot translate OR(=(ITEM($0, 'foo'), '1'),
=(ITEM($0, 'foo'), '2'))
at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$
Translator.translateMatch2(ElasticsearchFilter.java:234)
at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$
Translator.translateAnd(ElasticsearchFilter.java:158)
at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$
Translator.translateOr(ElasticsearchFilter.java:115)
at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$
Translator.translateMatch(ElasticsearchFilter.java:101)
at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$
Translator.access$000(ElasticsearchFilter.java:87)
at org.apache.calcite.adapter.elasticsearch.
ElasticsearchFilter.implement(ElasticsearchFilter.java:80)
at org.apache.calcite.adapter.elasticsearch.
ElasticsearchRel$Implementor.visitChild(ElasticsearchRel.java:53)
at org.apache.calcite.adapter.elasticsearch.
ElasticsearchToEnumerableConverter.implement(ElasticsearchToEnumerableConve
rter.java:71)
at org.apache.calcite.adapter.enumerable.EnumerableRelImplementor.
visitChild(EnumerableRelImplementor.java:98)
at org.apache.calcite.adapter.enumerable.
EnumerableAggregate.implement(EnumerableAggregate.java:106)
{quote}
was:
h2. Queries that are working
{code}
select count(*) from "elastic" where _MAP['foo'] in ('1') and true;
select count(*) from "elastic" where _MAP['foo'] ='1' and true;
select count(*) from "elastic" where _MAP['foo'] in ('1', '2');
{code}
h2. Queries that are failing
{code}
select count(*) from "elastic" where _MAP['foo'] in ('1', '2') and true;
select count(*) from "elastic" where true and _MAP['foo'] in ('1', '2');
select count(*) from "elastic" where (_MAP['foo'] ='1' or _MAP['foo'] = '2')
and true;
{code}
h2. Elastic Adapter Exception
{quote}
java.lang.AssertionError: cannot translate OR(=(ITEM($0, 'foo'), '1'),
=(ITEM($0, 'foo'), '2'))
at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$
Translator.translateMatch2(ElasticsearchFilter.java:234)
at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$
Translator.translateAnd(ElasticsearchFilter.java:158)
at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$
Translator.translateOr(ElasticsearchFilter.java:115)
at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$
Translator.translateMatch(ElasticsearchFilter.java:101)
at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$
Translator.access$000(ElasticsearchFilter.java:87)
at org.apache.calcite.adapter.elasticsearch.
ElasticsearchFilter.implement(ElasticsearchFilter.java:80)
at org.apache.calcite.adapter.elasticsearch.
ElasticsearchRel$Implementor.visitChild(ElasticsearchRel.java:53)
at org.apache.calcite.adapter.elasticsearch.
ElasticsearchToEnumerableConverter.implement(ElasticsearchToEnumerableConve
rter.java:71)
at org.apache.calcite.adapter.enumerable.EnumerableRelImplementor.
visitChild(EnumerableRelImplementor.java:98)
at org.apache.calcite.adapter.enumerable.
EnumerableAggregate.implement(EnumerableAggregate.java:106)
{quote}
h2. Mongo Adapter Exception
{quote}
select * from "mongo"."col"
where _MAP['foo'] in ('1', '2') and true;
java.lang.AssertionError: cannot translate OR(=(ITEM($0, 'foo'), '1'),
=(ITEM($0, 'foo'), '2'))
at org.apache.calcite.adapter.mongodb.MongoFilter$
Translator.translateMatch2(MongoFilter.java:181)
at org.apache.calcite.adapter.mongodb.MongoFilter$
Translator.translateAnd(MongoFilter.java:116)
at org.apache.calcite.adapter.mongodb.MongoFilter$
Translator.translateOr(MongoFilter.java:98)
at org.apache.calcite.adapter.mongodb.MongoFilter$
Translator.translateMatch(MongoFilter.java:91)
at org.apache.calcite.adapter.mongodb.MongoFilter$
Translator.access$000(MongoFilter.java:77)
at org.apache.calcite.adapter.mongodb.MongoFilter.implement(
MongoFilter.java:72)
at org.apache.calcite.adapter.mongodb.MongoRel$Implementor.
visitChild(MongoRel.java:51)
at org.apache.calcite.adapter.mongodb.MongoToEnumerableConverter.
[jira] [Updated] (CALCITE-2331) Evaluation of predicate "(A or B) and C" fails for Elasticsearch adapter
[
https://issues.apache.org/jira/browse/CALCITE-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi updated CALCITE-2331:
-
Summary: Evaluation of predicate "(A or B) and C" fails for Elasticsearch
adapter (was: evaluation of predicate (A or B) and C is failing for some
adapters)
> Evaluation of predicate "(A or B) and C" fails for Elasticsearch adapter
>
>
> Key: CALCITE-2331
> URL: https://issues.apache.org/jira/browse/CALCITE-2331
> Project: Calcite
> Issue Type: Bug
> Components: elasticsearch-adapter
>Reporter: Andrei Sereda
>Assignee: Michael Mior
>Priority: Critical
>
> h2. Queries that are working
> {code}
> select count(*) from "elastic" where _MAP['foo'] in ('1') and true;
> select count(*) from "elastic" where _MAP['foo'] ='1' and true;
> select count(*) from "elastic" where _MAP['foo'] in ('1', '2');
> {code}
> h2. Queries that are failing
> {code}
> select count(*) from "elastic" where _MAP['foo'] in ('1', '2') and true;
> select count(*) from "elastic" where true and _MAP['foo'] in ('1', '2');
> select count(*) from "elastic" where (_MAP['foo'] ='1' or _MAP['foo'] = '2')
> and true;
> {code}
> h2. Elastic Adapter Exception
> {quote}
> java.lang.AssertionError: cannot translate OR(=(ITEM($0, 'foo'), '1'),
> =(ITEM($0, 'foo'), '2'))
> at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$
> Translator.translateMatch2(ElasticsearchFilter.java:234)
> at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$
> Translator.translateAnd(ElasticsearchFilter.java:158)
> at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$
> Translator.translateOr(ElasticsearchFilter.java:115)
> at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$
> Translator.translateMatch(ElasticsearchFilter.java:101)
> at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$
> Translator.access$000(ElasticsearchFilter.java:87)
> at org.apache.calcite.adapter.elasticsearch.
> ElasticsearchFilter.implement(ElasticsearchFilter.java:80)
> at org.apache.calcite.adapter.elasticsearch.
> ElasticsearchRel$Implementor.visitChild(ElasticsearchRel.java:53)
> at org.apache.calcite.adapter.elasticsearch.
> ElasticsearchToEnumerableConverter.implement(ElasticsearchToEnumerableConve
> rter.java:71)
> at org.apache.calcite.adapter.enumerable.EnumerableRelImplementor.
> visitChild(EnumerableRelImplementor.java:98)
> at org.apache.calcite.adapter.enumerable.
> EnumerableAggregate.implement(EnumerableAggregate.java:106)
> {quote}
> h2. Mongo Adapter Exception
> {quote}
> select * from "mongo"."col"
> where _MAP['foo'] in ('1', '2') and true;
> java.lang.AssertionError: cannot translate OR(=(ITEM($0, 'foo'), '1'),
> =(ITEM($0, 'foo'), '2'))
> at org.apache.calcite.adapter.mongodb.MongoFilter$
> Translator.translateMatch2(MongoFilter.java:181)
> at org.apache.calcite.adapter.mongodb.MongoFilter$
> Translator.translateAnd(MongoFilter.java:116)
> at org.apache.calcite.adapter.mongodb.MongoFilter$
> Translator.translateOr(MongoFilter.java:98)
> at org.apache.calcite.adapter.mongodb.MongoFilter$
> Translator.translateMatch(MongoFilter.java:91)
> at org.apache.calcite.adapter.mongodb.MongoFilter$
> Translator.access$000(MongoFilter.java:77)
> at org.apache.calcite.adapter.mongodb.MongoFilter.implement(
> MongoFilter.java:72)
> at org.apache.calcite.adapter.mongodb.MongoRel$Implementor.
> visitChild(MongoRel.java:51)
> at org.apache.calcite.adapter.mongodb.MongoToEnumerableConverter.
> implement(MongoToEnumerableConverter.java:84)
> at org.apache.calcite.adapter.enumerable.EnumerableRelImplementor.
> implementRoot(EnumerableRelImplementor.java:103)
> at org.apache.calcite.adapter.enumerable.EnumerableInterpretable.
> toBindable(EnumerableInterpretable.java:92)
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (CALCITE-2331) Evaluation of predicate "(A or B) and C" fails for Elasticsearch adapter
[
https://issues.apache.org/jira/browse/CALCITE-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi updated CALCITE-2331:
-
Component/s: (was: mongodb)
(was: geode)
> Evaluation of predicate "(A or B) and C" fails for Elasticsearch adapter
>
>
> Key: CALCITE-2331
> URL: https://issues.apache.org/jira/browse/CALCITE-2331
> Project: Calcite
> Issue Type: Bug
> Components: elasticsearch-adapter
>Reporter: Andrei Sereda
>Assignee: Michael Mior
>Priority: Critical
>
> h2. Queries that are working
> {code:java}
> select count(*) from "elastic" where _MAP['foo'] in ('1') and true;
> select count(*) from "elastic" where _MAP['foo'] ='1' and true;
> select count(*) from "elastic" where _MAP['foo'] in ('1', '2');
> {code}
> h2. Queries that are failing
> {code:java}
> select count(*) from "elastic" where _MAP['foo'] in ('1', '2') and true;
> select count(*) from "elastic" where true and _MAP['foo'] in ('1', '2');
> select count(*) from "elastic" where (_MAP['foo'] ='1' or _MAP['foo'] = '2')
> and true;
> {code}
> h2. Elastic Adapter Exception
> {quote}java.lang.AssertionError: cannot translate OR(=(ITEM($0, 'foo'), '1'),
> =(ITEM($0, 'foo'), '2'))
> at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$
> Translator.translateMatch2(ElasticsearchFilter.java:234)
> at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$
> Translator.translateAnd(ElasticsearchFilter.java:158)
> at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$
> Translator.translateOr(ElasticsearchFilter.java:115)
> at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$
> Translator.translateMatch(ElasticsearchFilter.java:101)
> at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$
> Translator.access$000(ElasticsearchFilter.java:87)
> at org.apache.calcite.adapter.elasticsearch.
> ElasticsearchFilter.implement(ElasticsearchFilter.java:80)
> at org.apache.calcite.adapter.elasticsearch.
> ElasticsearchRel$Implementor.visitChild(ElasticsearchRel.java:53)
> at org.apache.calcite.adapter.elasticsearch.
> ElasticsearchToEnumerableConverter.implement(ElasticsearchToEnumerableConve
> rter.java:71)
> at org.apache.calcite.adapter.enumerable.EnumerableRelImplementor.
> visitChild(EnumerableRelImplementor.java:98)
> at org.apache.calcite.adapter.enumerable.
> EnumerableAggregate.implement(EnumerableAggregate.java:106)
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (CALCITE-2415) Evaluation of predicate "(A or B) and C" is failing for MongoDB adapter
[
https://issues.apache.org/jira/browse/CALCITE-2415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi updated CALCITE-2415:
-
Component/s: (was: geode)
(was: elasticsearch-adapter)
> Evaluation of predicate "(A or B) and C" is failing for MongoDB adapter
> ---
>
> Key: CALCITE-2415
> URL: https://issues.apache.org/jira/browse/CALCITE-2415
> Project: Calcite
> Issue Type: Bug
> Components: mongodb
>Reporter: Andrei Sereda
>Assignee: Michael Mior
>Priority: Critical
>
> h2. Queries that are working
> {code:java}
> select count(*) from "elastic" where _MAP['foo'] in ('1') and true;
> select count(*) from "elastic" where _MAP['foo'] ='1' and true;
> select count(*) from "elastic" where _MAP['foo'] in ('1', '2');
> {code}
> h2. Queries that are failing
> {code:java}
> select count(*) from "elastic" where _MAP['foo'] in ('1', '2') and true;
> select count(*) from "elastic" where true and _MAP['foo'] in ('1', '2');
> select count(*) from "elastic" where (_MAP['foo'] ='1' or _MAP['foo'] = '2')
> and true;
> {code}
> h2. Mongo Adapter Exception
> {quote}select * from "mongo"."col"
> where _MAP['foo'] in ('1', '2') and true;
> java.lang.AssertionError: cannot translate OR(=(ITEM($0, 'foo'), '1'),
> =(ITEM($0, 'foo'), '2'))
> at org.apache.calcite.adapter.mongodb.MongoFilter$
> Translator.translateMatch2(MongoFilter.java:181)
> at org.apache.calcite.adapter.mongodb.MongoFilter$
> Translator.translateAnd(MongoFilter.java:116)
> at org.apache.calcite.adapter.mongodb.MongoFilter$
> Translator.translateOr(MongoFilter.java:98)
> at org.apache.calcite.adapter.mongodb.MongoFilter$
> Translator.translateMatch(MongoFilter.java:91)
> at org.apache.calcite.adapter.mongodb.MongoFilter$
> Translator.access$000(MongoFilter.java:77)
> at org.apache.calcite.adapter.mongodb.MongoFilter.implement(
> MongoFilter.java:72)
> at org.apache.calcite.adapter.mongodb.MongoRel$Implementor.
> visitChild(MongoRel.java:51)
> at org.apache.calcite.adapter.mongodb.MongoToEnumerableConverter.
> implement(MongoToEnumerableConverter.java:84)
> at org.apache.calcite.adapter.enumerable.EnumerableRelImplementor.
> implementRoot(EnumerableRelImplementor.java:103)
> at org.apache.calcite.adapter.enumerable.EnumerableInterpretable.
> toBindable(EnumerableInterpretable.java:92)
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (CALCITE-2415) Evaluation of predicate "(A or B) and C" is failing for MongoDB adapter
[
https://issues.apache.org/jira/browse/CALCITE-2415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi updated CALCITE-2415:
-
Description:
h2. Queries that are working
{code:java}
select count(*) from "elastic" where _MAP['foo'] in ('1') and true;
select count(*) from "elastic" where _MAP['foo'] ='1' and true;
select count(*) from "elastic" where _MAP['foo'] in ('1', '2');
{code}
h2. Queries that are failing
{code:java}
select count(*) from "elastic" where _MAP['foo'] in ('1', '2') and true;
select count(*) from "elastic" where true and _MAP['foo'] in ('1', '2');
select count(*) from "elastic" where (_MAP['foo'] ='1' or _MAP['foo'] = '2')
and true;
{code}
h2. Mongo Adapter Exception
{quote}select * from "mongo"."col"
where _MAP['foo'] in ('1', '2') and true;
java.lang.AssertionError: cannot translate OR(=(ITEM($0, 'foo'), '1'),
=(ITEM($0, 'foo'), '2'))
at org.apache.calcite.adapter.mongodb.MongoFilter$
Translator.translateMatch2(MongoFilter.java:181)
at org.apache.calcite.adapter.mongodb.MongoFilter$
Translator.translateAnd(MongoFilter.java:116)
at org.apache.calcite.adapter.mongodb.MongoFilter$
Translator.translateOr(MongoFilter.java:98)
at org.apache.calcite.adapter.mongodb.MongoFilter$
Translator.translateMatch(MongoFilter.java:91)
at org.apache.calcite.adapter.mongodb.MongoFilter$
Translator.access$000(MongoFilter.java:77)
at org.apache.calcite.adapter.mongodb.MongoFilter.implement(
MongoFilter.java:72)
at org.apache.calcite.adapter.mongodb.MongoRel$Implementor.
visitChild(MongoRel.java:51)
at org.apache.calcite.adapter.mongodb.MongoToEnumerableConverter.
implement(MongoToEnumerableConverter.java:84)
at org.apache.calcite.adapter.enumerable.EnumerableRelImplementor.
implementRoot(EnumerableRelImplementor.java:103)
at org.apache.calcite.adapter.enumerable.EnumerableInterpretable.
toBindable(EnumerableInterpretable.java:92)
{quote}
was:
h2. Queries that are working
{code}
select count(*) from "elastic" where _MAP['foo'] in ('1') and true;
select count(*) from "elastic" where _MAP['foo'] ='1' and true;
select count(*) from "elastic" where _MAP['foo'] in ('1', '2');
{code}
h2. Queries that are failing
{code}
select count(*) from "elastic" where _MAP['foo'] in ('1', '2') and true;
select count(*) from "elastic" where true and _MAP['foo'] in ('1', '2');
select count(*) from "elastic" where (_MAP['foo'] ='1' or _MAP['foo'] = '2')
and true;
{code}
h2. Elastic Adapter Exception
{quote}
java.lang.AssertionError: cannot translate OR(=(ITEM($0, 'foo'), '1'),
=(ITEM($0, 'foo'), '2'))
at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$
Translator.translateMatch2(ElasticsearchFilter.java:234)
at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$
Translator.translateAnd(ElasticsearchFilter.java:158)
at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$
Translator.translateOr(ElasticsearchFilter.java:115)
at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$
Translator.translateMatch(ElasticsearchFilter.java:101)
at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$
Translator.access$000(ElasticsearchFilter.java:87)
at org.apache.calcite.adapter.elasticsearch.
ElasticsearchFilter.implement(ElasticsearchFilter.java:80)
at org.apache.calcite.adapter.elasticsearch.
ElasticsearchRel$Implementor.visitChild(ElasticsearchRel.java:53)
at org.apache.calcite.adapter.elasticsearch.
ElasticsearchToEnumerableConverter.implement(ElasticsearchToEnumerableConve
rter.java:71)
at org.apache.calcite.adapter.enumerable.EnumerableRelImplementor.
visitChild(EnumerableRelImplementor.java:98)
at org.apache.calcite.adapter.enumerable.
EnumerableAggregate.implement(EnumerableAggregate.java:106)
{quote}
h2. Mongo Adapter Exception
{quote}
select * from "mongo"."col"
where _MAP['foo'] in ('1', '2') and true;
java.lang.AssertionError: cannot translate OR(=(ITEM($0, 'foo'), '1'),
=(ITEM($0, 'foo'), '2'))
at org.apache.calcite.adapter.mongodb.MongoFilter$
Translator.translateMatch2(MongoFilter.java:181)
at org.apache.calcite.adapter.mongodb.MongoFilter$
Translator.translateAnd(MongoFilter.java:116)
at org.apache.calcite.adapter.mongodb.MongoFilter$
Translator.translateOr(MongoFilter.java:98)
at org.apache.calcite.adapter.mongodb.MongoFilter$
Translator.translateMatch(MongoFilter.java:91)
at org.apache.calcite.adapter.mongodb.MongoFilter$
Translator.access$000(MongoFilter.java:77)
at org.apache.calcite.adapter.mongodb.MongoFilter.implement(
MongoFilter.java:72)
at org.apache.calcite.adapter.mongodb.MongoRel$Implementor.
visitChild(MongoRel.java:51)
at org.apache.calcite.adapter.mongodb.MongoToEnumerableConverter.
implement(MongoToEnumerableConverter.java:84)
at
[jira] [Created] (CALCITE-2415) Evaluation of predicate "(A or B) and C" is failing for MongoDB adapter
Volodymyr Vysotskyi created CALCITE-2415:
Summary: Evaluation of predicate "(A or B) and C" is failing for
MongoDB adapter
Key: CALCITE-2415
URL: https://issues.apache.org/jira/browse/CALCITE-2415
Project: Calcite
Issue Type: Bug
Components: elasticsearch-adapter, geode, mongodb
Reporter: Andrei Sereda
Assignee: Michael Mior
h2. Queries that are working
{code}
select count(*) from "elastic" where _MAP['foo'] in ('1') and true;
select count(*) from "elastic" where _MAP['foo'] ='1' and true;
select count(*) from "elastic" where _MAP['foo'] in ('1', '2');
{code}
h2. Queries that are failing
{code}
select count(*) from "elastic" where _MAP['foo'] in ('1', '2') and true;
select count(*) from "elastic" where true and _MAP['foo'] in ('1', '2');
select count(*) from "elastic" where (_MAP['foo'] ='1' or _MAP['foo'] = '2')
and true;
{code}
h2. Elastic Adapter Exception
{quote}
java.lang.AssertionError: cannot translate OR(=(ITEM($0, 'foo'), '1'),
=(ITEM($0, 'foo'), '2'))
at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$
Translator.translateMatch2(ElasticsearchFilter.java:234)
at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$
Translator.translateAnd(ElasticsearchFilter.java:158)
at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$
Translator.translateOr(ElasticsearchFilter.java:115)
at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$
Translator.translateMatch(ElasticsearchFilter.java:101)
at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$
Translator.access$000(ElasticsearchFilter.java:87)
at org.apache.calcite.adapter.elasticsearch.
ElasticsearchFilter.implement(ElasticsearchFilter.java:80)
at org.apache.calcite.adapter.elasticsearch.
ElasticsearchRel$Implementor.visitChild(ElasticsearchRel.java:53)
at org.apache.calcite.adapter.elasticsearch.
ElasticsearchToEnumerableConverter.implement(ElasticsearchToEnumerableConve
rter.java:71)
at org.apache.calcite.adapter.enumerable.EnumerableRelImplementor.
visitChild(EnumerableRelImplementor.java:98)
at org.apache.calcite.adapter.enumerable.
EnumerableAggregate.implement(EnumerableAggregate.java:106)
{quote}
h2. Mongo Adapter Exception
{quote}
select * from "mongo"."col"
where _MAP['foo'] in ('1', '2') and true;
java.lang.AssertionError: cannot translate OR(=(ITEM($0, 'foo'), '1'),
=(ITEM($0, 'foo'), '2'))
at org.apache.calcite.adapter.mongodb.MongoFilter$
Translator.translateMatch2(MongoFilter.java:181)
at org.apache.calcite.adapter.mongodb.MongoFilter$
Translator.translateAnd(MongoFilter.java:116)
at org.apache.calcite.adapter.mongodb.MongoFilter$
Translator.translateOr(MongoFilter.java:98)
at org.apache.calcite.adapter.mongodb.MongoFilter$
Translator.translateMatch(MongoFilter.java:91)
at org.apache.calcite.adapter.mongodb.MongoFilter$
Translator.access$000(MongoFilter.java:77)
at org.apache.calcite.adapter.mongodb.MongoFilter.implement(
MongoFilter.java:72)
at org.apache.calcite.adapter.mongodb.MongoRel$Implementor.
visitChild(MongoRel.java:51)
at org.apache.calcite.adapter.mongodb.MongoToEnumerableConverter.
implement(MongoToEnumerableConverter.java:84)
at org.apache.calcite.adapter.enumerable.EnumerableRelImplementor.
implementRoot(EnumerableRelImplementor.java:103)
at org.apache.calcite.adapter.enumerable.EnumerableInterpretable.
toBindable(EnumerableInterpretable.java:92)
{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (CALCITE-2410) Enable ProjectSetOpTransposeRule for UNION (distinct) SetOp operator
[
https://issues.apache.org/jira/browse/CALCITE-2410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi updated CALCITE-2410:
-
Fix Version/s: 1.17.0
> Enable ProjectSetOpTransposeRule for UNION (distinct) SetOp operator
>
>
> Key: CALCITE-2410
> URL: https://issues.apache.org/jira/browse/CALCITE-2410
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.16.0
>Reporter: Vitalii Diravka
>Assignee: Vitalii Diravka
>Priority: Major
> Fix For: 1.17.0
>
>
> Currently {{ProjectSetOpTransposeRule}} rule can transpose UnionAll SetOp and
> Project RelNodes. It disabled for Union (distinct) SetOp operator.
> {code:java}
> // cannot push project past a distinct
> if (!setOp.all) {
> return;
> }
> {code}
> After removing this check the rule works fine and transposes Union (distinct)
> SetOp and Project RelNodes.
> Test case for reproduce (RelOptRulesTest class):
> {code}
> @Test public void testPushProjectPastUnionDistinct() {
> checkPlanning(ProjectSetOpTransposeRule.INSTANCE,
> "select sal from "
> + "(select * from emp e1 union select * from emp e2)");
> }
>
>
>
>
>
>
>
>
>
>
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (CALCITE-2409) SparkAdapterTest fails on Windows when '/tmp' directory does not exist
[
https://issues.apache.org/jira/browse/CALCITE-2409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi updated CALCITE-2409:
-
Summary: SparkAdapterTest fails on Windows when '/tmp' directory does not
exist (was: SparkAdapterTest fails on Windows)
> SparkAdapterTest fails on Windows when '/tmp' directory does not exist
> --
>
> Key: CALCITE-2409
> URL: https://issues.apache.org/jira/browse/CALCITE-2409
> Project: Calcite
> Issue Type: Bug
> Components: spark
>Reporter: Sergey Nuyanzin
>Assignee: Julian Hyde
>Priority: Major
> Fix For: 1.17.0
>
>
> If there is no {{tmp}} folder on a disk where {{mvn clean install}} started
> it will fail with a trace as below.
> Here it is a proposal to create the folder if it is absent (another option
> could be usage of {{System.getProperty("java.io.tmpdir")}} instead of
> {{"/tmp"}})
> {noformat}
> at org.apache.calcite.avatica.Helper.wrap(Helper.java:37)
> at
> org.apache.calcite.adapter.enumerable.EnumerableInterpretable.toBindable(EnumerableInterpretable.java:108)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl$CalcitePreparingStmt.implement(CalcitePrepareImpl.java:1237)
> at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:331)
> at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:230)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:772)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:636)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:606)
> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:229)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:550)
> at
> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
> ... 28 more
> Caused by: java.lang.RuntimeException: java.io.FileNotFoundException:
> \tmp\CalciteProgram105359.java (The system cannot find the path specified)
> at
> org.apache.calcite.adapter.spark.SparkHandlerImpl.compile(SparkHandlerImpl.java:139)
> at
> org.apache.calcite.adapter.enumerable.EnumerableInterpretable.toBindable(EnumerableInterpretable.java:103)
> ... 38 more
> Caused by: java.io.FileNotFoundException: \tmp\CalciteProgram105359.java (The
> system cannot find the path specified)
> at java.base/java.io.FileOutputStream.open0(Native Method)
> at java.base/java.io.FileOutputStream.open(FileOutputStream.java:299)
> at java.base/java.io.FileOutputStream.(FileOutputStream.java:238)
> at java.base/java.io.FileOutputStream.(FileOutputStream.java:188)
> at org.apache.calcite.util.Util.printWriter(Util.java:2306)
> at
> org.apache.calcite.adapter.spark.SparkHandlerImpl.compile(SparkHandlerImpl.java:112)
> ... 39 more
> java.lang.RuntimeException: With materializationsEnabled=false, limit=0
> at
> org.apache.calcite.test.CalciteAssert.assertQuery(CalciteAssert.java:568)
> at
> org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1328)
> at
> org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1305)
> at
> org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1268)
> at
> org.apache.calcite.test.SparkAdapterTest.testOrderByLimit(SparkAdapterTest.java:447)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:564)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at
[jira] [Resolved] (CALCITE-2409) SparkAdapterTest fails on Windows
[
https://issues.apache.org/jira/browse/CALCITE-2409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi resolved CALCITE-2409.
--
Resolution: Fixed
Fix Version/s: 1.17.0
Thanks, [~Sergey Nuyanzin], for finding this issue and for the pull request!
Fixed in
[008bfd6|https://github.com/apache/calcite/commit/008bfd6e9668f5be01057f73c14334d4b77446e4].
> SparkAdapterTest fails on Windows
> -
>
> Key: CALCITE-2409
> URL: https://issues.apache.org/jira/browse/CALCITE-2409
> Project: Calcite
> Issue Type: Bug
> Components: spark
>Reporter: Sergey Nuyanzin
>Assignee: Julian Hyde
>Priority: Major
> Fix For: 1.17.0
>
>
> If there is no {{tmp}} folder on a disk where {{mvn clean install}} started
> it will fail with a trace as below.
> Here it is a proposal to create the folder if it is absent (another option
> could be usage of {{System.getProperty("java.io.tmpdir")}} instead of
> {{"/tmp"}})
> {noformat}
> at org.apache.calcite.avatica.Helper.wrap(Helper.java:37)
> at
> org.apache.calcite.adapter.enumerable.EnumerableInterpretable.toBindable(EnumerableInterpretable.java:108)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl$CalcitePreparingStmt.implement(CalcitePrepareImpl.java:1237)
> at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:331)
> at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:230)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:772)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:636)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:606)
> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:229)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:550)
> at
> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
> ... 28 more
> Caused by: java.lang.RuntimeException: java.io.FileNotFoundException:
> \tmp\CalciteProgram105359.java (The system cannot find the path specified)
> at
> org.apache.calcite.adapter.spark.SparkHandlerImpl.compile(SparkHandlerImpl.java:139)
> at
> org.apache.calcite.adapter.enumerable.EnumerableInterpretable.toBindable(EnumerableInterpretable.java:103)
> ... 38 more
> Caused by: java.io.FileNotFoundException: \tmp\CalciteProgram105359.java (The
> system cannot find the path specified)
> at java.base/java.io.FileOutputStream.open0(Native Method)
> at java.base/java.io.FileOutputStream.open(FileOutputStream.java:299)
> at java.base/java.io.FileOutputStream.(FileOutputStream.java:238)
> at java.base/java.io.FileOutputStream.(FileOutputStream.java:188)
> at org.apache.calcite.util.Util.printWriter(Util.java:2306)
> at
> org.apache.calcite.adapter.spark.SparkHandlerImpl.compile(SparkHandlerImpl.java:112)
> ... 39 more
> java.lang.RuntimeException: With materializationsEnabled=false, limit=0
> at
> org.apache.calcite.test.CalciteAssert.assertQuery(CalciteAssert.java:568)
> at
> org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1328)
> at
> org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1305)
> at
> org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1268)
> at
> org.apache.calcite.test.SparkAdapterTest.testOrderByLimit(SparkAdapterTest.java:447)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:564)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at
[jira] [Resolved] (CALCITE-2366) Add support for ANY_VALUE function
[
https://issues.apache.org/jira/browse/CALCITE-2366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi resolved CALCITE-2366.
--
Resolution: Fixed
Fixed in
[6a2af65|https://github.com/apache/calcite/commit/6a2af65c945be671123ae35f3c2b9e6f4d9163d1].
Thanks, [~gparai] for the pull request, [~amansinha100] and [~julianhyde] for
the review!
> Add support for ANY_VALUE function
> --
>
> Key: CALCITE-2366
> URL: https://issues.apache.org/jira/browse/CALCITE-2366
> Project: Calcite
> Issue Type: New Feature
>Affects Versions: 1.16.0
>Reporter: Gautam Kumar Parai
>Assignee: Gautam Kumar Parai
>Priority: Major
> Fix For: 1.17.0
>
>
> We had discussions on the Apache
> Calcite([here|https://lists.apache.org/thread.html/f2007a489d3a5741875bcc8a1edd8d5c3715e5114ac45058c3b3a42d@%3Cdev.calcite.apache.org%3E])
> and Apache
> Drill([here|https://lists.apache.org/thread.html/2517eef7410aed4e88b9515f7e4256335215c1ad39a2676a08d21cb9@%3Cdev.drill.apache.org%3E])
> mailing lists regarding an equivalent for {{DISTINCT ON}}. The community
> seems to prefer the {{ANY_VALUE}}. This Jira is a placeholder for
> implementing the {{ANY_VALUE}} aggregate function in Apache Calcite.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (CALCITE-2342) use of assert with side-effect in Schemas
[
https://issues.apache.org/jira/browse/CALCITE-2342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi resolved CALCITE-2342.
--
Resolution: Fixed
Fixed in
[a99f386|https://github.com/apache/calcite/commit/a99f386d5c36d560c832f9e70542f9d058ea22f5]
> use of assert with side-effect in Schemas
> -
>
> Key: CALCITE-2342
> URL: https://issues.apache.org/jira/browse/CALCITE-2342
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Laurent Goujon
>Assignee: Laurent Goujon
>Priority: Major
> Fix For: 1.17.0
>
>
> {{Schemas#path(CalciteSchema,Iterable)}} method has the following
> code:
> {code:java}
> if (!rootSchema.name.isEmpty()) {
> assert rootSchema.name.equals(iterator.next());
> }
> {code}
> Depending if assertions are enabled or not, iterator state might be different
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (CALCITE-2342) use of assert with side-effect in Schemas
[
https://issues.apache.org/jira/browse/CALCITE-2342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi updated CALCITE-2342:
-
Fix Version/s: 1.17.0
> use of assert with side-effect in Schemas
> -
>
> Key: CALCITE-2342
> URL: https://issues.apache.org/jira/browse/CALCITE-2342
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Laurent Goujon
>Assignee: Laurent Goujon
>Priority: Major
> Fix For: 1.17.0
>
>
> {{Schemas#path(CalciteSchema,Iterable)}} method has the following
> code:
> {code:java}
> if (!rootSchema.name.isEmpty()) {
> assert rootSchema.name.equals(iterator.next());
> }
> {code}
> Depending if assertions are enabled or not, iterator state might be different
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (CALCITE-2255) Test against JDK 11
[ https://issues.apache.org/jira/browse/CALCITE-2255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi resolved CALCITE-2255. -- Resolution: Fixed Resolving this Jira since both PRs were merged into Calcite and Avatica. Looks like failures observed on Travis build are caused by [https://bugs.openjdk.java.net/browse/JDK-8200530] > Test against JDK 11 > --- > > Key: CALCITE-2255 > URL: https://issues.apache.org/jira/browse/CALCITE-2255 > Project: Calcite > Issue Type: Improvement >Reporter: Kevin Risden >Assignee: Kevin Risden >Priority: Major > Fix For: 1.17.0, avatica-1.13.0 > > > JDK 11 ([http://jdk.java.net/11/)] is in early release and we can start > testing against it. With the improvements from CALCITE-2063, we should be > able to test against JDK 11 without much effort. > I opened [https://github.com/docker-library/openjdk/pull/186] and > [https://github.com/carlossg/docker-maven/issues/79] to try to get JDK 11 > support for the docker images we are trying to use. > I am working on testing locally as well to see if there is anything broken. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CALCITE-2392) Query with JOIN and USING on dynamic table fails with IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/CALCITE-2392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi updated CALCITE-2392:
-
Fix Version/s: 1.17.0
> Query with JOIN and USING on dynamic table fails with
> IndexOutOfBoundsException
> ---
>
> Key: CALCITE-2392
> URL: https://issues.apache.org/jira/browse/CALCITE-2392
> Project: Calcite
> Issue Type: Bug
>Reporter: Volodymyr Vysotskyi
>Assignee: Julian Hyde
>Priority: Major
> Fix For: 1.17.0
>
>
> Query with join and using fails with {{IndexOutOfBoundsException}} when a
> dynamic table is used. This test:
> {code:sql}
> @Test
> public void testJoinUsingDynamicTable() {
> final String sql = "select * from SALES.NATION t1\n"
> + "join SALES.NATION t2\n"
> + "using (n_nationkey)";
> sql(sql).with(getTesterWithDynamicTable()).ok();
> }
> {code}
> added to {{SqlToRelConverterTest}} class fails:
> {noformat}
> java.lang.IndexOutOfBoundsException: index (2) must be less than size (2)
> at
> com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:310)
> at
> com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:293)
> at
> com.google.common.collect.RegularImmutableList.get(RegularImmutableList.java:67)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$Permute.permute(SqlValidatorImpl.java:6278)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.expandStar(SqlValidatorImpl.java:535)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.expandSelectItem(SqlValidatorImpl.java:423)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelectList(SqlValidatorImpl.java:4021)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3284)
> at
> org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:964)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:940)
> at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:226)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:915)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:625)
> at
> org.apache.calcite.test.SqlToRelTestBase$TesterImpl.convertSqlToRel(SqlToRelTestBase.java:642)
> at
> org.apache.calcite.test.SqlToRelTestBase$TesterImpl.assertConvertsTo(SqlToRelTestBase.java:754)
> at
> org.apache.calcite.test.SqlToRelConverterTest$Sql.convertsTo(SqlToRelConverterTest.java:2884)
> at
> org.apache.calcite.test.SqlToRelConverterTest$Sql.ok(SqlToRelConverterTest.java:2876)
> at
> org.apache.calcite.test.SqlToRelConverterTest.testJoinUsingDynamicTable(SqlToRelConverterTest.java:122)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at
>
[jira] [Resolved] (CALCITE-2392) Query with JOIN and USING on dynamic table fails with IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/CALCITE-2392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi resolved CALCITE-2392.
--
Resolution: Fixed
[~julianhyde], thanks for the review!
Fixed in
[5bbc501|https://github.com/apache/calcite/commit/5bbc501a565494442784f65870a20cd65a5016f4].
> Query with JOIN and USING on dynamic table fails with
> IndexOutOfBoundsException
> ---
>
> Key: CALCITE-2392
> URL: https://issues.apache.org/jira/browse/CALCITE-2392
> Project: Calcite
> Issue Type: Bug
>Reporter: Volodymyr Vysotskyi
>Assignee: Julian Hyde
>Priority: Major
>
> Query with join and using fails with {{IndexOutOfBoundsException}} when a
> dynamic table is used. This test:
> {code:sql}
> @Test
> public void testJoinUsingDynamicTable() {
> final String sql = "select * from SALES.NATION t1\n"
> + "join SALES.NATION t2\n"
> + "using (n_nationkey)";
> sql(sql).with(getTesterWithDynamicTable()).ok();
> }
> {code}
> added to {{SqlToRelConverterTest}} class fails:
> {noformat}
> java.lang.IndexOutOfBoundsException: index (2) must be less than size (2)
> at
> com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:310)
> at
> com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:293)
> at
> com.google.common.collect.RegularImmutableList.get(RegularImmutableList.java:67)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$Permute.permute(SqlValidatorImpl.java:6278)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.expandStar(SqlValidatorImpl.java:535)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.expandSelectItem(SqlValidatorImpl.java:423)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelectList(SqlValidatorImpl.java:4021)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3284)
> at
> org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:964)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:940)
> at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:226)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:915)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:625)
> at
> org.apache.calcite.test.SqlToRelTestBase$TesterImpl.convertSqlToRel(SqlToRelTestBase.java:642)
> at
> org.apache.calcite.test.SqlToRelTestBase$TesterImpl.assertConvertsTo(SqlToRelTestBase.java:754)
> at
> org.apache.calcite.test.SqlToRelConverterTest$Sql.convertsTo(SqlToRelConverterTest.java:2884)
> at
> org.apache.calcite.test.SqlToRelConverterTest$Sql.ok(SqlToRelConverterTest.java:2876)
> at
> org.apache.calcite.test.SqlToRelConverterTest.testJoinUsingDynamicTable(SqlToRelConverterTest.java:122)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
>
[jira] [Commented] (CALCITE-2392) Query with JOIN and USING on dynamic table fails with IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/CALCITE-2392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16532665#comment-16532665
]
Volodymyr Vysotskyi commented on CALCITE-2392:
--
I have created CALCITE-2400 to fix and allow it in future, but for now, I have
created [PR-752|https://github.com/apache/calcite/pull/752] to prevent columns
permutation for dynamic tables.
If there are no objections, I'll merge this PR tomorrow.
> Query with JOIN and USING on dynamic table fails with
> IndexOutOfBoundsException
> ---
>
> Key: CALCITE-2392
> URL: https://issues.apache.org/jira/browse/CALCITE-2392
> Project: Calcite
> Issue Type: Bug
>Reporter: Volodymyr Vysotskyi
>Assignee: Julian Hyde
>Priority: Major
>
> Query with join and using fails with {{IndexOutOfBoundsException}} when a
> dynamic table is used. This test:
> {code:sql}
> @Test
> public void testJoinUsingDynamicTable() {
> final String sql = "select * from SALES.NATION t1\n"
> + "join SALES.NATION t2\n"
> + "using (n_nationkey)";
> sql(sql).with(getTesterWithDynamicTable()).ok();
> }
> {code}
> added to {{SqlToRelConverterTest}} class fails:
> {noformat}
> java.lang.IndexOutOfBoundsException: index (2) must be less than size (2)
> at
> com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:310)
> at
> com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:293)
> at
> com.google.common.collect.RegularImmutableList.get(RegularImmutableList.java:67)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$Permute.permute(SqlValidatorImpl.java:6278)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.expandStar(SqlValidatorImpl.java:535)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.expandSelectItem(SqlValidatorImpl.java:423)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelectList(SqlValidatorImpl.java:4021)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3284)
> at
> org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:964)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:940)
> at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:226)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:915)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:625)
> at
> org.apache.calcite.test.SqlToRelTestBase$TesterImpl.convertSqlToRel(SqlToRelTestBase.java:642)
> at
> org.apache.calcite.test.SqlToRelTestBase$TesterImpl.assertConvertsTo(SqlToRelTestBase.java:754)
> at
> org.apache.calcite.test.SqlToRelConverterTest$Sql.convertsTo(SqlToRelConverterTest.java:2884)
> at
> org.apache.calcite.test.SqlToRelConverterTest$Sql.ok(SqlToRelConverterTest.java:2876)
> at
> org.apache.calcite.test.SqlToRelConverterTest.testJoinUsingDynamicTable(SqlToRelConverterTest.java:122)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
[jira] [Created] (CALCITE-2400) Allow standards-compliant column ordering for NATURAL JOIN and JOIN USING when dynamic tables are used
Volodymyr Vysotskyi created CALCITE-2400: Summary: Allow standards-compliant column ordering for NATURAL JOIN and JOIN USING when dynamic tables are used Key: CALCITE-2400 URL: https://issues.apache.org/jira/browse/CALCITE-2400 Project: Calcite Issue Type: Bug Reporter: Volodymyr Vysotskyi Assignee: Julian Hyde In CALCITE-2227 was allowed standards-compliant column ordering for NATURAL JOIN and JOIN USING for non-dynamic tables. In CALCITE-2392 was prevented permutation of columns for dynamic tables to avoid IOBE. The aim of this Jira is to implement standards-compliant column ordering for NATURAL JOIN and JOIN USING for dynamic tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CALCITE-2392) Query with JOIN and USING on dynamic table fails with IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/CALCITE-2392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531668#comment-16531668
]
Volodymyr Vysotskyi commented on CALCITE-2392:
--
Agree, it is complex to implement it for dynamic tables. So to unblock the
release I think it's enough to disable it and fix after the release.
I'm thinking about implementing a new function for a dynamic star, which
fetches all columns beside specified columns, aka сomplement 'ITEM'. I think it
should help to allow this feature for dynamic tables.
> Query with JOIN and USING on dynamic table fails with
> IndexOutOfBoundsException
> ---
>
> Key: CALCITE-2392
> URL: https://issues.apache.org/jira/browse/CALCITE-2392
> Project: Calcite
> Issue Type: Bug
>Reporter: Volodymyr Vysotskyi
>Assignee: Julian Hyde
>Priority: Major
>
> Query with join and using fails with {{IndexOutOfBoundsException}} when a
> dynamic table is used. This test:
> {code:sql}
> @Test
> public void testJoinUsingDynamicTable() {
> final String sql = "select * from SALES.NATION t1\n"
> + "join SALES.NATION t2\n"
> + "using (n_nationkey)";
> sql(sql).with(getTesterWithDynamicTable()).ok();
> }
> {code}
> added to {{SqlToRelConverterTest}} class fails:
> {noformat}
> java.lang.IndexOutOfBoundsException: index (2) must be less than size (2)
> at
> com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:310)
> at
> com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:293)
> at
> com.google.common.collect.RegularImmutableList.get(RegularImmutableList.java:67)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$Permute.permute(SqlValidatorImpl.java:6278)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.expandStar(SqlValidatorImpl.java:535)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.expandSelectItem(SqlValidatorImpl.java:423)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelectList(SqlValidatorImpl.java:4021)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3284)
> at
> org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:964)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:940)
> at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:226)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:915)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:625)
> at
> org.apache.calcite.test.SqlToRelTestBase$TesterImpl.convertSqlToRel(SqlToRelTestBase.java:642)
> at
> org.apache.calcite.test.SqlToRelTestBase$TesterImpl.assertConvertsTo(SqlToRelTestBase.java:754)
> at
> org.apache.calcite.test.SqlToRelConverterTest$Sql.convertsTo(SqlToRelConverterTest.java:2884)
> at
> org.apache.calcite.test.SqlToRelConverterTest$Sql.ok(SqlToRelConverterTest.java:2876)
> at
> org.apache.calcite.test.SqlToRelConverterTest.testJoinUsingDynamicTable(SqlToRelConverterTest.java:122)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>
[jira] [Commented] (CALCITE-2392) Query with JOIN and USING on dynamic table fails with IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/CALCITE-2392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531613#comment-16531613
]
Volodymyr Vysotskyi commented on CALCITE-2392:
--
I have analyzed the problem and it is a regression after the fix for
CALCITE-2227.
[~julianhyde], since you are the author of the original commit, could you
please take a look at this bug?
> Query with JOIN and USING on dynamic table fails with
> IndexOutOfBoundsException
> ---
>
> Key: CALCITE-2392
> URL: https://issues.apache.org/jira/browse/CALCITE-2392
> Project: Calcite
> Issue Type: Bug
>Reporter: Volodymyr Vysotskyi
>Assignee: Julian Hyde
>Priority: Major
>
> Query with join and using fails with {{IndexOutOfBoundsException}} when a
> dynamic table is used. This test:
> {code:sql}
> @Test
> public void testJoinUsingDynamicTable() {
> final String sql = "select * from SALES.NATION t1\n"
> + "join SALES.NATION t2\n"
> + "using (n_nationkey)";
> sql(sql).with(getTesterWithDynamicTable()).ok();
> }
> {code}
> added to {{SqlToRelConverterTest}} class fails:
> {noformat}
> java.lang.IndexOutOfBoundsException: index (2) must be less than size (2)
> at
> com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:310)
> at
> com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:293)
> at
> com.google.common.collect.RegularImmutableList.get(RegularImmutableList.java:67)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$Permute.permute(SqlValidatorImpl.java:6278)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.expandStar(SqlValidatorImpl.java:535)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.expandSelectItem(SqlValidatorImpl.java:423)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelectList(SqlValidatorImpl.java:4021)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3284)
> at
> org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:964)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:940)
> at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:226)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:915)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:625)
> at
> org.apache.calcite.test.SqlToRelTestBase$TesterImpl.convertSqlToRel(SqlToRelTestBase.java:642)
> at
> org.apache.calcite.test.SqlToRelTestBase$TesterImpl.assertConvertsTo(SqlToRelTestBase.java:754)
> at
> org.apache.calcite.test.SqlToRelConverterTest$Sql.convertsTo(SqlToRelConverterTest.java:2884)
> at
> org.apache.calcite.test.SqlToRelConverterTest$Sql.ok(SqlToRelConverterTest.java:2876)
> at
> org.apache.calcite.test.SqlToRelConverterTest.testJoinUsingDynamicTable(SqlToRelConverterTest.java:122)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at
>
[jira] [Resolved] (CALCITE-2379) CVSS dependency-check-maven fails for calcite-spark module
[
https://issues.apache.org/jira/browse/CALCITE-2379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi resolved CALCITE-2379.
--
Resolution: Fixed
Fixed in
[625edb8|https://github.com/apache/calcite/commit/625edb88e6afd41b37fdae95e15d5d04cc794b7e]
> CVSS dependency-check-maven fails for calcite-spark module
> --
>
> Key: CALCITE-2379
> URL: https://issues.apache.org/jira/browse/CALCITE-2379
> Project: Calcite
> Issue Type: Bug
> Components: spark
>Reporter: Volodymyr Vysotskyi
>Assignee: Julian Hyde
>Priority: Major
> Fix For: 1.17.0
>
>
> Check for vulnerabilities among dependencies fails for {{calcite-spark}}
> module.
> "{{mvn install -Ppedantic -DskipTests=true}}" fails for lib
> {{py4j-0.10.4.jar}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-2391) Aggregate query with UNNEST or LATERAL fails with ClassCastException
[
https://issues.apache.org/jira/browse/CALCITE-2391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530274#comment-16530274
]
Volodymyr Vysotskyi commented on CALCITE-2391:
--
[~julianhyde], thanks for so quick response and fix!
> Aggregate query with UNNEST or LATERAL fails with ClassCastException
>
>
> Key: CALCITE-2391
> URL: https://issues.apache.org/jira/browse/CALCITE-2391
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Volodymyr Vysotskyi
>Assignee: Julian Hyde
>Priority: Major
> Fix For: 1.17.0
>
>
> Query with aggregation of UNNEST column fails with ClassCastException. This
> test:
> {code:java}
> @Test public void testAggUnnestColumn() {
> CalciteAssert.hr()
> .query("select count(d.\"name\") as c\n"
> + "from \"hr\".\"depts\" as d,\n"
> + " UNNEST(d.\"employees\") as e")
> .returnsUnordered(
> "C=3");
> }
> {code}
> added to {{JdbcTest}} class fails with {{ClassCastException}}
> {noformat}
> Caused by: java.lang.ClassCastException:
> org.apache.calcite.sql.validate.AggregatingSelectScope cannot be cast to
> org.apache.calcite.sql.validate.ListScope
> at
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.lookupExp(SqlToRelConverter.java:4293)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:3614)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.access$2300(SqlToRelConverter.java:213)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:4692)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:3990)
> at org.apache.calcite.sql.SqlIdentifier.accept(SqlIdentifier.java:344)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.convertExpression(SqlToRelConverter.java:4554)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2090)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:1957)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2033)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:642)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:623)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:3058)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:559)
> at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:264)
> at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:230)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:783)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:642)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:612)
> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:232)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:609)
> at
> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
> ... 28 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CALCITE-2392) Query with JOIN and USING on dynamic table fails with IndexOutOfBoundsException
Volodymyr Vysotskyi created CALCITE-2392:
Summary: Query with JOIN and USING on dynamic table fails with
IndexOutOfBoundsException
Key: CALCITE-2392
URL: https://issues.apache.org/jira/browse/CALCITE-2392
Project: Calcite
Issue Type: Bug
Reporter: Volodymyr Vysotskyi
Assignee: Julian Hyde
Query with join and using fails with {{IndexOutOfBoundsException}} when a
dynamic table is used. This test:
{code:sql}
@Test
public void testJoinUsingDynamicTable() {
final String sql = "select * from SALES.NATION t1\n"
+ "join SALES.NATION t2\n"
+ "using (n_nationkey)";
sql(sql).with(getTesterWithDynamicTable()).ok();
}
{code}
added to {{SqlToRelConverterTest}} class fails:
{noformat}
java.lang.IndexOutOfBoundsException: index (2) must be less than size (2)
at
com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:310)
at
com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:293)
at
com.google.common.collect.RegularImmutableList.get(RegularImmutableList.java:67)
at
org.apache.calcite.sql.validate.SqlValidatorImpl$Permute.permute(SqlValidatorImpl.java:6278)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.expandStar(SqlValidatorImpl.java:535)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.expandSelectItem(SqlValidatorImpl.java:423)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelectList(SqlValidatorImpl.java:4021)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3284)
at
org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60)
at
org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:964)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:940)
at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:226)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:915)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:625)
at
org.apache.calcite.test.SqlToRelTestBase$TesterImpl.convertSqlToRel(SqlToRelTestBase.java:642)
at
org.apache.calcite.test.SqlToRelTestBase$TesterImpl.assertConvertsTo(SqlToRelTestBase.java:754)
at
org.apache.calcite.test.SqlToRelConverterTest$Sql.convertsTo(SqlToRelConverterTest.java:2884)
at
org.apache.calcite.test.SqlToRelConverterTest$Sql.ok(SqlToRelConverterTest.java:2876)
at
org.apache.calcite.test.SqlToRelConverterTest.testJoinUsingDynamicTable(SqlToRelConverterTest.java:122)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at
com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
at
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (CALCITE-2391) Query with aggregation of UNNEST column fails with ClassCastException
[
https://issues.apache.org/jira/browse/CALCITE-2391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16529233#comment-16529233
]
Volodymyr Vysotskyi edited comment on CALCITE-2391 at 7/1/18 9:30 PM:
--
Yes, it appeared recently (after 1.16 release). I was trying to see the impact
of updating Calcite for Drill and found this issue.
was (Author: vvysotskyi):
Yes, it appeared recently. I was trying to see the impact of updating Calcite
for Drill and found this issue.
> Query with aggregation of UNNEST column fails with ClassCastException
> -
>
> Key: CALCITE-2391
> URL: https://issues.apache.org/jira/browse/CALCITE-2391
> Project: Calcite
> Issue Type: Bug
>Reporter: Volodymyr Vysotskyi
>Assignee: Julian Hyde
>Priority: Major
>
> This test:
> {code:java}
> @Test public void testAggUnnestColumn() {
> CalciteAssert.hr()
> .query("select count(d.\"name\") as c\n"
> + "from \"hr\".\"depts\" as d,\n"
> + " UNNEST(d.\"employees\") as e")
> .returnsUnordered(
> "C=3");
> }
> {code}
> added to {{JdbcTest}} class fails with {{ClassCastException}}
> {noformat}
> Caused by: java.lang.ClassCastException:
> org.apache.calcite.sql.validate.AggregatingSelectScope cannot be cast to
> org.apache.calcite.sql.validate.ListScope
> at
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.lookupExp(SqlToRelConverter.java:4293)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:3614)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.access$2300(SqlToRelConverter.java:213)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:4692)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:3990)
> at org.apache.calcite.sql.SqlIdentifier.accept(SqlIdentifier.java:344)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.convertExpression(SqlToRelConverter.java:4554)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2090)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:1957)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2033)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:642)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:623)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:3058)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:559)
> at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:264)
> at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:230)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:783)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:642)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:612)
> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:232)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:609)
> at
> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
> ... 28 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-2391) Query with aggregation of UNNEST column fails with ClassCastException
[
https://issues.apache.org/jira/browse/CALCITE-2391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16529237#comment-16529237
]
Volodymyr Vysotskyi commented on CALCITE-2391:
--
I have just checked, and your assumption is correct - it appeared after the fix
for CALCITE-531.
> Query with aggregation of UNNEST column fails with ClassCastException
> -
>
> Key: CALCITE-2391
> URL: https://issues.apache.org/jira/browse/CALCITE-2391
> Project: Calcite
> Issue Type: Bug
>Reporter: Volodymyr Vysotskyi
>Assignee: Julian Hyde
>Priority: Major
>
> This test:
> {code:java}
> @Test public void testAggUnnestColumn() {
> CalciteAssert.hr()
> .query("select count(d.\"name\") as c\n"
> + "from \"hr\".\"depts\" as d,\n"
> + " UNNEST(d.\"employees\") as e")
> .returnsUnordered(
> "C=3");
> }
> {code}
> added to {{JdbcTest}} class fails with {{ClassCastException}}
> {noformat}
> Caused by: java.lang.ClassCastException:
> org.apache.calcite.sql.validate.AggregatingSelectScope cannot be cast to
> org.apache.calcite.sql.validate.ListScope
> at
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.lookupExp(SqlToRelConverter.java:4293)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:3614)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.access$2300(SqlToRelConverter.java:213)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:4692)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:3990)
> at org.apache.calcite.sql.SqlIdentifier.accept(SqlIdentifier.java:344)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.convertExpression(SqlToRelConverter.java:4554)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2090)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:1957)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2033)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:642)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:623)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:3058)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:559)
> at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:264)
> at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:230)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:783)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:642)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:612)
> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:232)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:609)
> at
> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
> ... 28 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-2391) Query with aggregation of UNNEST column fails with ClassCastException
[
https://issues.apache.org/jira/browse/CALCITE-2391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16529233#comment-16529233
]
Volodymyr Vysotskyi commented on CALCITE-2391:
--
Yes, it appeared recently. I was trying to see the impact of updating Calcite
for Drill and found this issue.
> Query with aggregation of UNNEST column fails with ClassCastException
> -
>
> Key: CALCITE-2391
> URL: https://issues.apache.org/jira/browse/CALCITE-2391
> Project: Calcite
> Issue Type: Bug
>Reporter: Volodymyr Vysotskyi
>Assignee: Julian Hyde
>Priority: Major
>
> This test:
> {code:java}
> @Test public void testAggUnnestColumn() {
> CalciteAssert.hr()
> .query("select count(d.\"name\") as c\n"
> + "from \"hr\".\"depts\" as d,\n"
> + " UNNEST(d.\"employees\") as e")
> .returnsUnordered(
> "C=3");
> }
> {code}
> added to {{JdbcTest}} class fails with {{ClassCastException}}
> {noformat}
> Caused by: java.lang.ClassCastException:
> org.apache.calcite.sql.validate.AggregatingSelectScope cannot be cast to
> org.apache.calcite.sql.validate.ListScope
> at
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.lookupExp(SqlToRelConverter.java:4293)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:3614)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.access$2300(SqlToRelConverter.java:213)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:4692)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:3990)
> at org.apache.calcite.sql.SqlIdentifier.accept(SqlIdentifier.java:344)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.convertExpression(SqlToRelConverter.java:4554)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2090)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:1957)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2033)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:642)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:623)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:3058)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:559)
> at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:264)
> at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:230)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:783)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:642)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:612)
> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:232)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:609)
> at
> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
> ... 28 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
