date:20190816

[jira] [Updated] (CALCITE-3173) RexNode Code Generation Problem

2019-08-16 Thread Stamatis Zampetakis (JIRA)



 [ 
https://issues.apache.org/jira/browse/CALCITE-3173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stamatis Zampetakis updated CALCITE-3173:
-
Priority: Critical  (was: Major)

> RexNode Code Generation Problem
> ---
>
> Key: CALCITE-3173
> URL: https://issues.apache.org/jira/browse/CALCITE-3173
> Project: Calcite
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 1.20.0
>Reporter: Feng Zhu
>Assignee: Feng Zhu
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 1.21.0
>
> Attachments: code.png, codegen.png
>
>
> *Abstract:* Both RexImpTable and BlockBuilder have codegen issues, but it is 
> interesting that they can work together well for most cases.
>     We can illustrate the problem with a simple test case in JdbcTest, in 
> which the "commission" column is nullable.
> {code:java}
> @Test public void testNullSafeCheck() {
>     CalciteAssert.hr()
>   .query("select \"commission\" + 10 as s from \"hr\".\"emps\"")
>   .returns("S=1010\n"
>+ "S=510\n"
>+ "S=null\n"
>+ "S=260\n");
> }
> {code}
>     This test case can pass as the BlockBuilder is in default optimization 
> mode. However, when we set it into un-optimization mode in _EnumerableCalc_, 
> this test will fail due to NPE. The following picture demonstrates their 
> differences.
> !code.png!
> *1.RexImpTable generates unsafe code*
>      Before translating the RexCall (_*Add*_), RexImpTable firstly translate 
> its operands with (nullAs=*IS_NULL*) [1] to detect whether it is null (i.e., 
> {color:#ff}_inp4_unboxed_{color}). Then RexImpTable sets this operand's 
> nullable in RexToLixTranslator as {color:#FF}false{color} [2]. After 
> that, the operand will be translated again with *NOT_POSSIBLE* [3] to get the 
> value (i.e., {color:#ff}_inp4_0_unboxed_{color}). In the end, the RexCall 
> is implemented by NotNullImplementor.However, it is not safe to conduct 
> operations like unboxing in the second translation phase. 
>  *2.BlockBuiler optimization's semantic issue buries NPE*
>      BlockBuilder.optimize() changes the code semantic in this case. For 
> conditional-like clauses (if...else, ?:, etc), InlineVariableVisitor will 
> wrongly make variables inlined.
>     In general, they can work together for most cases. However, when some 
> special branch is triggered by query, the problem will be exposed. For 
> example, the NewExpression (_new java.math.BigDecimal_) in CALCITE-3143 
> breaks the inline optimization phase.
>  
> *How to fix?*
>      I have digged into this problem a couple of days and tried many 
> approaches to fix it. But in this progress, I found the limitation in current 
> implementation.   The whole recursive framework essentially conducts a 
> sequential codegen beheavior, and may visit a RexNode again and again with 
> different NullAs settings.
>     Due to the limitation, it is difficult to implement null-safe codegen 
> semantics with branching logic. We can also find that there are many branches 
> for special cases in current implementation. Even we can handle potential 
> issues every time, the logic will becomes more and more complex  and 
> unfriendly for maintenance.   
>  
> Therefore, I propose to re-consider this part, including several initial 
> points.
>  (1) {color:#ff}_Visitor Pattern_{color} (RexVisitor). 
> Theoretically, RexNode can be translated into Expression by visiting the node 
> only once. We can implement RexVisitor rather than current recursive 
> translation.
>  (2)The {color:#ff}Result{color} consists of three items (code: 
> BuilderStatement, isNull: ParameterExpression, value: Expression).So it is 
> easy to decide how  to implement a RexNode according to its children.
>  
> Please correct me if I make something wrong. Look forward to suggestions!
>  
> [1][https://github.com/apache/calcite/blob/1748f0503e7b626a8d0165f1698adb8b61bbc31e/core/src/main/java/org/apache/calcite/adapter/enumerable/RexImpTable.java#L1062]
> [2][https://github.com/apache/calcite/blob/1748f0503e7b626a8d0165f1698adb8b61bbc31e/core/src/main/java/org/apache/calcite/adapter/enumerable/RexImpTable.java#L1064]
> [3][https://github.com/apache/calcite/blob/1748f0503e7b626a8d0165f1698adb8b61bbc31e/core/src/main/java/org/apache/calcite/adapter/enumerable/RexImpTable.java#L1113]
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Updated] (CALCITE-2302) Implicit type cast support

2019-08-16 Thread Stamatis Zampetakis (JIRA)



 [ 
https://issues.apache.org/jira/browse/CALCITE-2302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stamatis Zampetakis updated CALCITE-2302:
-
Priority: Critical  (was: Major)

> Implicit type cast support
> --
>
> Key: CALCITE-2302
> URL: https://issues.apache.org/jira/browse/CALCITE-2302
> Project: Calcite
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 1.17.0
>Reporter: Danny Chan
>Assignee: Danny Chan
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 1.21.0
>
>  Time Spent: 6.5h
>  Remaining Estimate: 0h
>
> Now many DBs have support implicit type cast, eg: SqlServer, Oracle, Hive.
> Implicit type cast is an useful function for many cases, So we should support 
> this.
> I checkout Calcite code and found that:
>  # Now we use a validator to validate our operands types[ through kinds of 
> namespaces and scopes ]
>  # Most of the validations will finally goes to
> {code:java}
> SqlOperator.validateOperands
> {code}
>  # which will use validation logic defined in corresponding 
> SqlOperandTypeChecker
> What i'm confused about is where should i put the implicit type cast logic 
> in? I figured out 2 ways:
>  # Supply a tool class/rules to add casts into a parsed SqlNode tree which 
> will then go through the validation logic later on.
>  # Unleash the validation logic in kinds of SqlOperandTypeChecker, then 
> modify the RelNode/RexNodes tree converted from a validated SqlNode tree to 
> add in casts through custom RelOptRules.
> So guys, which of the 2 ways should i go, or if there are better way to do 
> this?
> I need your help.
>  
> Updated 18-05-30:
> Hi guys, i have made a PR in 
> [CALCITE-2302|https://github.com/apache/calcite/pull/706]
> This is design doc: [Calcite Implicit Type Cast 
> Design|https://docs.google.com/document/d/1g2RUnLXyp_LjUlO-wbblKuP5hqEu3a_2Mt2k4dh6RwU/edit?usp=sharing].
> This is the conversion types mapping: [Conversion Types 
> Mapping|https://docs.google.com/spreadsheets/d/1GhleX5h5W8-kJKh7NMJ4vtoE78pwfaZRJl88ULX_MgU/edit?usp=sharing].
> I really appreciate your suggestions, thx.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Commented] (CALCITE-2302) Implicit type cast support

2019-08-16 Thread Stamatis Zampetakis (JIRA)



[ 
https://issues.apache.org/jira/browse/CALCITE-2302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909601#comment-16909601
 ] 

Stamatis Zampetakis commented on CALCITE-2302:
--

I think we agreed that it should be possible to enable/disable all implicit 
casts on demand; if that's the case then in principle we shouldn't break any of 
the existing queries in Postgres and other DBMS if the casts are disabled. If 
we are afraid that there are many cases that this may happen then we could 
choose to have type casts disabled by default for now. 

> Implicit type cast support
> --
>
> Key: CALCITE-2302
> URL: https://issues.apache.org/jira/browse/CALCITE-2302
> Project: Calcite
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 1.17.0
>Reporter: Danny Chan
>Assignee: Danny Chan
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.21.0
>
>  Time Spent: 6.5h
>  Remaining Estimate: 0h
>
> Now many DBs have support implicit type cast, eg: SqlServer, Oracle, Hive.
> Implicit type cast is an useful function for many cases, So we should support 
> this.
> I checkout Calcite code and found that:
>  # Now we use a validator to validate our operands types[ through kinds of 
> namespaces and scopes ]
>  # Most of the validations will finally goes to
> {code:java}
> SqlOperator.validateOperands
> {code}
>  # which will use validation logic defined in corresponding 
> SqlOperandTypeChecker
> What i'm confused about is where should i put the implicit type cast logic 
> in? I figured out 2 ways:
>  # Supply a tool class/rules to add casts into a parsed SqlNode tree which 
> will then go through the validation logic later on.
>  # Unleash the validation logic in kinds of SqlOperandTypeChecker, then 
> modify the RelNode/RexNodes tree converted from a validated SqlNode tree to 
> add in casts through custom RelOptRules.
> So guys, which of the 2 ways should i go, or if there are better way to do 
> this?
> I need your help.
>  
> Updated 18-05-30:
> Hi guys, i have made a PR in 
> [CALCITE-2302|https://github.com/apache/calcite/pull/706]
> This is design doc: [Calcite Implicit Type Cast 
> Design|https://docs.google.com/document/d/1g2RUnLXyp_LjUlO-wbblKuP5hqEu3a_2Mt2k4dh6RwU/edit?usp=sharing].
> This is the conversion types mapping: [Conversion Types 
> Mapping|https://docs.google.com/spreadsheets/d/1GhleX5h5W8-kJKh7NMJ4vtoE78pwfaZRJl88ULX_MgU/edit?usp=sharing].
> I really appreciate your suggestions, thx.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Commented] (CALCITE-3257) Need to clear RelMetaDataQuery cache after dumping RelNodes and Graphviz

2019-08-16 Thread Stamatis Zampetakis (JIRA)



[ 
https://issues.apache.org/jira/browse/CALCITE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909599#comment-16909599
 ] 

Stamatis Zampetakis commented on CALCITE-3257:
--

This is exactly the same problem with CALCITE-2018. It would be better not to 
disperse the discussion into multiple JIRAs; let's close this as duplicate and 
discuss under CALCITE-2018.

> Need to clear RelMetaDataQuery cache after dumping RelNodes and Graphviz
> 
>
> Key: CALCITE-3257
> URL: https://issues.apache.org/jira/browse/CALCITE-3257
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Reporter: Xiening Dai
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> To repro -
> 1. Set Log4J log level to TRACE. So planner will dump rel node info every 
> time at rule match 
> (https://github.com/apache/calcite/blob/3124a85b93ff2f1b79484c7bd4cc41835d4f1920/core/src/main/java/org/apache/calcite/plan/volcano/RuleQueue.java#L435)
> 2. Run JdbcTest.testNotExistsCorrelated. Get below exception -
> java.lang.AssertionError: rel 
> [rel#63:EnumerableAggregate.ENUMERABLE.[](input=RelSubset#62,group={0})] has 
> lower cost {131.0 rows, 216.0 cpu, 0.0 io} than best cost {131.5 rows, 216.0 
> cpu, 0.0 io} of subset [rel#60:Subset#4.ENUMERABLE.[]]
>   at org.apache.calcite.util.Litmus$1.fail(Litmus.java:31)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.isValid(VolcanoPlanner.java:889)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:852)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:869)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1928)
>   at 
> org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:129)
>   at 
> org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:236)
>   at 
> org.apache.calcite.rel.convert.ConverterRule.onMatch(ConverterRule.java:141)
>   at 
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:208)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:631)
>   at 
> org.apache.calcite.tools.Programs.lambda$standard$3(Programs.java:286)
>   at 
> org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:346)
>   at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:189)
>   at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:314)
>   at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:231)
>   at 
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:638)
>   at 
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:502)
>   at 
> org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:472)
>   at 
> org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:231)
>   at 
> org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:550)
>   at 
> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675)
>   at 
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
>   at 
> org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227)
>   at 
> org.apache.calcite.test.CalciteAssert.assertQuery(CalciteAssert.java:522)
>   at 
> org.apache.calcite.test.CalciteAssert$AssertQuery.lambda$returns$1(CalciteAssert.java:1466)
>   at 
> org.apache.calcite.test.CalciteAssert$AssertQuery.withConnection(CalciteAssert.java:1398)
>   at 
> org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1464)
>   at 
> org.apache.calcite.test.CalciteAssert$AssertQuery.explainMatches(CalciteAssert.java:1561)
>   at 
> org.apache.calcite.test.CalciteAssert$AssertQuery.explainContains(CalciteAssert.java:1556)
>   at 
> org.apache.calcite.test.JdbcTest.testNotExistsCorrelated(JdbcTest.java:4562)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>

[jira] [Updated] (CALCITE-3259) Align 'Property' in the serialized xml string of RelXmlWriter.

2019-08-16 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/CALCITE-3259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated CALCITE-3259:

Labels: pull-request-available  (was: )

> Align 'Property' in the serialized xml string of RelXmlWriter.
> --
>
> Key: CALCITE-3259
> URL: https://issues.apache.org/jira/browse/CALCITE-3259
> Project: Calcite
>  Issue Type: Improvement
>Reporter: Wang Yanlin
>Priority: Minor
>  Labels: pull-request-available
>
> In the serialized xml string of  RelXmlWriter, the 'Property' label is 
> aligned.
> For the sql below
> {noformat}
> select 1 + 2, 3 from (values (true))
> {noformat}
> the serialized xml string is like this:
> {noformat}
> 
>   
>   +(1, 2) 
>   
>   3   
>   
>   
>   
>   [{ true }]  
>   
>   
>   
> 
> {noformat}
> It's better to make 'Property' aligned.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (CALCITE-3259) Align 'Property' in the serialized xml string of RelXmlWriter.

2019-08-16 Thread Wang Yanlin (JIRA)

Wang Yanlin created CALCITE-3259:


 Summary: Align 'Property' in the serialized xml string of 
RelXmlWriter.
 Key: CALCITE-3259
 URL: https://issues.apache.org/jira/browse/CALCITE-3259
 Project: Calcite
  Issue Type: Improvement
Reporter: Wang Yanlin


In the serialized xml string of  RelXmlWriter, the 'Property' label is aligned.

For the sql below
{noformat}
select 1 + 2, 3 from (values (true))
{noformat}
the serialized xml string is like this:
{noformat}


+(1, 2) 

3   



[{ true }]  




{noformat}
It's better to make 'Property' aligned.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Assigned] (CALCITE-3226) RelBuilder doesn't keep the alias when scan from an expanded view

2019-08-16 Thread jin xing (JIRA)



 [ 
https://issues.apache.org/jira/browse/CALCITE-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing reassigned CALCITE-3226:
-

Assignee: jin xing

> RelBuilder doesn't keep the alias when scan from an expanded view
> -
>
> Key: CALCITE-3226
> URL: https://issues.apache.org/jira/browse/CALCITE-3226
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Reporter: jin xing
>Assignee: jin xing
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.21.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> In current implementation of {{RelBuilder::scan}} 
> ([https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L1048]),
>  the alias can be derived and recorded into {{Frame}} only when the 
> {{RelNode}} is a {{TableScan}} 
> ([https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L2754]).
>  But when {{RelBuilder::scan}} from an expanded view, the node is not a 
> {{TableScan}} and the alias is not kept.  Below test failed - we cannot 
> reference a field by alias "MYVIEW".
> {code:java}
>   @Test public void testExpandViewShouldKeepAlias() throws SQLException {
> try (Connection connection = 
> DriverManager.getConnection("jdbc:calcite:")) {
>   final Frameworks.ConfigBuilder configBuilder =
>   expandingConfig(connection);
>   final RelOptTable.ViewExpander viewExpander =
>   (RelOptTable.ViewExpander) 
> Frameworks.getPlanner(configBuilder.build());
>   final RelFactories.TableScanFactory tableScanFactory =
>   RelFactories.expandingScanFactory(viewExpander,
>   RelFactories.DEFAULT_TABLE_SCAN_FACTORY);
>   configBuilder.context(Contexts.of(tableScanFactory));
>   final RelBuilder builder = RelBuilder.create(configBuilder.build());
>   RelNode node =
>   builder.scan("MYVIEW")
>   .project(
>   builder.field(1, "MYVIEW", "EMPNO"),   // Exception 
> thrown from here.
>   builder.field(1, "MYVIEW", "ENAME"))
>   .build();
>   String expected =
>   "LogicalProject(EMPNO=[$0], ENAME=[$1])\n"
>   + "  LogicalFilter(condition=[=(1, 1)])\n"
>   + "LogicalTableScan(table=[[scott, EMP]])\n";
>   assertThat(node, hasTree(expected));
> }
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Comment Edited] (CALCITE-3226) RelBuilder doesn't keep the alias when scan from an expanded view

2019-08-16 Thread jin xing (JIRA)



[ 
https://issues.apache.org/jira/browse/CALCITE-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909567#comment-16909567
 ] 

jin xing edited comment on CALCITE-3226 at 8/17/19 4:06 AM:


Thanks a lot for clarify [~rubenql]
I think [~julianhyde] expects to add a test case where the view of “select * 
from t” expands as a Scan, but I tried and turned out that it always expands as 
a Project+Scan.
For a common table scan, the alias would have already been correctly set and I 
added a test case 
testProjectWithAliasFromScan(https://github.com/apache/calcite/pull/1351/files#diff-9f6a8f13d9ad70d7336340bf6b2e7b71R702)

Thanks a lot for review this issue [~julianhyde] [~rubenql] and sorry for my 
late reply


was (Author: jinxing6...@126.com):
Thanks a lot for clarify [~rubenql]
I think [~julianhyde] expects to add a test case where the view of “select * 
from t” expands as a Scan, but I tried and turned out that it always expands as 
a Project+Scan.
For a common table scan, the alias would have already been correctly set and I 
added a test case 
testProjectWithAliasFromScan(https://github.com/apache/calcite/pull/1351/files#diff-9f6a8f13d9ad70d7336340bf6b2e7b71R702)

Thanks a lot for review this issue [~julianhyde][~rubenql] and sorry for my 
late reply

> RelBuilder doesn't keep the alias when scan from an expanded view
> -
>
> Key: CALCITE-3226
> URL: https://issues.apache.org/jira/browse/CALCITE-3226
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Reporter: jin xing
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.21.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> In current implementation of {{RelBuilder::scan}} 
> ([https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L1048]),
>  the alias can be derived and recorded into {{Frame}} only when the 
> {{RelNode}} is a {{TableScan}} 
> ([https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L2754]).
>  But when {{RelBuilder::scan}} from an expanded view, the node is not a 
> {{TableScan}} and the alias is not kept.  Below test failed - we cannot 
> reference a field by alias "MYVIEW".
> {code:java}
>   @Test public void testExpandViewShouldKeepAlias() throws SQLException {
> try (Connection connection = 
> DriverManager.getConnection("jdbc:calcite:")) {
>   final Frameworks.ConfigBuilder configBuilder =
>   expandingConfig(connection);
>   final RelOptTable.ViewExpander viewExpander =
>   (RelOptTable.ViewExpander) 
> Frameworks.getPlanner(configBuilder.build());
>   final RelFactories.TableScanFactory tableScanFactory =
>   RelFactories.expandingScanFactory(viewExpander,
>   RelFactories.DEFAULT_TABLE_SCAN_FACTORY);
>   configBuilder.context(Contexts.of(tableScanFactory));
>   final RelBuilder builder = RelBuilder.create(configBuilder.build());
>   RelNode node =
>   builder.scan("MYVIEW")
>   .project(
>   builder.field(1, "MYVIEW", "EMPNO"),   // Exception 
> thrown from here.
>   builder.field(1, "MYVIEW", "ENAME"))
>   .build();
>   String expected =
>   "LogicalProject(EMPNO=[$0], ENAME=[$1])\n"
>   + "  LogicalFilter(condition=[=(1, 1)])\n"
>   + "LogicalTableScan(table=[[scott, EMP]])\n";
>   assertThat(node, hasTree(expected));
> }
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Commented] (CALCITE-3226) RelBuilder doesn't keep the alias when scan from an expanded view

2019-08-16 Thread jin xing (JIRA)



[ 
https://issues.apache.org/jira/browse/CALCITE-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909567#comment-16909567
 ] 

jin xing commented on CALCITE-3226:
---

Thanks a lot for clarify [~rubenql]
I think [~julianhyde] expects to add a test case where the view of “select * 
from t” expands as a Scan, but I tried and turned out that it always expands as 
a Project+Scan.
For a common table scan, the alias would have already been correctly set and I 
added a test case 
testProjectWithAliasFromScan(https://github.com/apache/calcite/pull/1351/files#diff-9f6a8f13d9ad70d7336340bf6b2e7b71R702)

Thanks a lot for review this issue [~julianhyde][~rubenql] and sorry for my 
late reply

> RelBuilder doesn't keep the alias when scan from an expanded view
> -
>
> Key: CALCITE-3226
> URL: https://issues.apache.org/jira/browse/CALCITE-3226
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Reporter: jin xing
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.21.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> In current implementation of {{RelBuilder::scan}} 
> ([https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L1048]),
>  the alias can be derived and recorded into {{Frame}} only when the 
> {{RelNode}} is a {{TableScan}} 
> ([https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L2754]).
>  But when {{RelBuilder::scan}} from an expanded view, the node is not a 
> {{TableScan}} and the alias is not kept.  Below test failed - we cannot 
> reference a field by alias "MYVIEW".
> {code:java}
>   @Test public void testExpandViewShouldKeepAlias() throws SQLException {
> try (Connection connection = 
> DriverManager.getConnection("jdbc:calcite:")) {
>   final Frameworks.ConfigBuilder configBuilder =
>   expandingConfig(connection);
>   final RelOptTable.ViewExpander viewExpander =
>   (RelOptTable.ViewExpander) 
> Frameworks.getPlanner(configBuilder.build());
>   final RelFactories.TableScanFactory tableScanFactory =
>   RelFactories.expandingScanFactory(viewExpander,
>   RelFactories.DEFAULT_TABLE_SCAN_FACTORY);
>   configBuilder.context(Contexts.of(tableScanFactory));
>   final RelBuilder builder = RelBuilder.create(configBuilder.build());
>   RelNode node =
>   builder.scan("MYVIEW")
>   .project(
>   builder.field(1, "MYVIEW", "EMPNO"),   // Exception 
> thrown from here.
>   builder.field(1, "MYVIEW", "ENAME"))
>   .build();
>   String expected =
>   "LogicalProject(EMPNO=[$0], ENAME=[$1])\n"
>   + "  LogicalFilter(condition=[=(1, 1)])\n"
>   + "LogicalTableScan(table=[[scott, EMP]])\n";
>   assertThat(node, hasTree(expected));
> }
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Commented] (CALCITE-3213) Support complex type expressions for SqlDataTypeSpec

2019-08-16 Thread Danny Chan (JIRA)



[ 
https://issues.apache.org/jira/browse/CALCITE-3213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909552#comment-16909552
 ] 

Danny Chan commented on CALCITE-3213:
-

Let me summarize the broad changes:
 * Support row type
 * Support array type
 * Support nested collection type
 * Support time zone definition for time and timestamp type
 * Support collection type for UDT and fix it's nullability

> Support complex type expressions for  SqlDataTypeSpec
> -
>
> Key: CALCITE-3213
> URL: https://issues.apache.org/jira/browse/CALCITE-3213
> Project: Calcite
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 1.20.0
>Reporter: Danny Chan
>Assignee: Danny Chan
>Priority: Major
> Fix For: 1.21.0
>
>
> We should support nested struct type like:
> {code:sql}
> ROW(
> foo NUMBER(5, 2) NOT NULL,
> rec ROW(b BOOLEAN, i MyUDT NOT NULL))
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Resolved] (CALCITE-3255) CannotPlanException: There are not enough rules to produce a node with desired properties

2019-08-16 Thread Danny Chan (JIRA)



 [ 
https://issues.apache.org/jira/browse/CALCITE-3255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Danny Chan resolved CALCITE-3255.
-
Resolution: Invalid
  Assignee: Danny Chan

> CannotPlanException: There are not enough rules to produce a node with 
> desired properties
> -
>
> Key: CALCITE-3255
> URL: https://issues.apache.org/jira/browse/CALCITE-3255
> Project: Calcite
>  Issue Type: Bug
> Environment: !image-2019-08-16-18-09-57-964.png!
>Reporter: Water Cut Off
>Assignee: Danny Chan
>Priority: Major
> Attachments: image-2019-08-16-18-09-57-964.png
>
>
> I want to use calcite volcano planner to estimate sql cost,but it doesn't 
> work when i try to test a case,The exception is blow:
> {color:#FF}org.apache.calcite.plan.RelOptPlanner$CannotPlanException: 
> There are not enough rules to produce a node with desired properties: 
> convention=NONE, sort=[]. All the inputs have relevant nodes, however the 
> cost is still infinite.{color}
> {color:#FF} Root: rel#40:Subset#4.NONE.[]{color}
> {color:#FF} Original rel:{color}
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Commented] (CALCITE-3255) CannotPlanException: There are not enough rules to produce a node with desired properties

2019-08-16 Thread Danny Chan (JIRA)



[ 
https://issues.apache.org/jira/browse/CALCITE-3255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909539#comment-16909539
 ] 

Danny Chan commented on CALCITE-3255:
-

Thanks, [~Water Cut Off], this is an expected behavior, The relational node 
cost would be infinite if the convention is NONE, you should first set up your 
desired conventions in the target target traits[1], and define you node 
transform converters[2].

You should also supply the metadata in you convention nodes, so that our 
metadata system can compute the cumulative cost correctly.

 

[1] 
[https://github.com/apache/calcite/blob/a68bfc96bacc95b4b8ab5d185bb5cc1991113769/core/src/main/java/org/apache/calcite/tools/Program.java#L38]

[2] 
https://github.com/apache/calcite/blob/a68bfc96bacc95b4b8ab5d185bb5cc1991113769/core/src/main/java/org/apache/calcite/rel/convert/ConverterRule.java#L35

> CannotPlanException: There are not enough rules to produce a node with 
> desired properties
> -
>
> Key: CALCITE-3255
> URL: https://issues.apache.org/jira/browse/CALCITE-3255
> Project: Calcite
>  Issue Type: Bug
> Environment: !image-2019-08-16-18-09-57-964.png!
>Reporter: Water Cut Off
>Priority: Major
> Attachments: image-2019-08-16-18-09-57-964.png
>
>
> I want to use calcite volcano planner to estimate sql cost,but it doesn't 
> work when i try to test a case,The exception is blow:
> {color:#FF}org.apache.calcite.plan.RelOptPlanner$CannotPlanException: 
> There are not enough rules to produce a node with desired properties: 
> convention=NONE, sort=[]. All the inputs have relevant nodes, however the 
> cost is still infinite.{color}
> {color:#FF} Root: rel#40:Subset#4.NONE.[]{color}
> {color:#FF} Original rel:{color}
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Commented] (CALCITE-3250) Support nested collection type for SqlDataTypeSpec

2019-08-16 Thread Danny Chan (JIRA)



[ 
https://issues.apache.org/jira/browse/CALCITE-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909535#comment-16909535
 ] 

Danny Chan commented on CALCITE-3250:
-

Indeed, very sorry about that, i should have made the issue title and commit 
message more accurate.

> Support nested collection type for  SqlDataTypeSpec
> ---
>
> Key: CALCITE-3250
> URL: https://issues.apache.org/jira/browse/CALCITE-3250
> Project: Calcite
>  Issue Type: Sub-task
>  Components: core
>Affects Versions: 1.20.0
>Reporter: Danny Chan
>Assignee: Danny Chan
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.21.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> According to SQL-2011, collection type can be nested:
> {code:sql}
>  ::=
>ARRAY [bracket or trigraph> ]
>  ::=
>   
> {code}
> *Example*:
> {code:sql}
> int array array;
> int multiset array;
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Commented] (CALCITE-3257) Need to clear RelMetaDataQuery cache after dumping RelNodes and Graphviz

2019-08-16 Thread Xiening Dai (JIRA)



[ 
https://issues.apache.org/jira/browse/CALCITE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909486#comment-16909486
 ] 

Xiening Dai commented on CALCITE-3257:
--

I agree with your assumption. Current cost propagation logic wouldn't work if 
the parent node cost is already exists in the cache - in which case there's no 
chance for recomputing the cost based on the change of input cost. I think 
that's why we purge cache every time the rule fires 
(https://github.com/apache/calcite/blob/979a825fab0757c506363f5df6ed7997674e8726/core/src/main/java/org/apache/calcite/plan/volcano/VolcanoRuleCall.java#L130).
 *RuleQueue.dump() calls into dumpGraphviz() which indeed calls getCost() for 
rel node*. At that point, new cache entry is added, causing problem in the next 
rule cost propagation.

Your change might as well fix this particular case in the bug. But I think it's 
still important to keep the behavior consistent regardless the log level, aka 
purging the cache between rules. In the future, we might be able to improve the 
cache so that it can be reused across rules, which would be a huge gain of 
performance. But before that, I think we still need this fix so cache is purged 
after each dump().

> Need to clear RelMetaDataQuery cache after dumping RelNodes and Graphviz
> 
>
> Key: CALCITE-3257
> URL: https://issues.apache.org/jira/browse/CALCITE-3257
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Reporter: Xiening Dai
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> To repro -
> 1. Set Log4J log level to TRACE. So planner will dump rel node info every 
> time at rule match 
> (https://github.com/apache/calcite/blob/3124a85b93ff2f1b79484c7bd4cc41835d4f1920/core/src/main/java/org/apache/calcite/plan/volcano/RuleQueue.java#L435)
> 2. Run JdbcTest.testNotExistsCorrelated. Get below exception -
> java.lang.AssertionError: rel 
> [rel#63:EnumerableAggregate.ENUMERABLE.[](input=RelSubset#62,group={0})] has 
> lower cost {131.0 rows, 216.0 cpu, 0.0 io} than best cost {131.5 rows, 216.0 
> cpu, 0.0 io} of subset [rel#60:Subset#4.ENUMERABLE.[]]
>   at org.apache.calcite.util.Litmus$1.fail(Litmus.java:31)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.isValid(VolcanoPlanner.java:889)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:852)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:869)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1928)
>   at 
> org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:129)
>   at 
> org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:236)
>   at 
> org.apache.calcite.rel.convert.ConverterRule.onMatch(ConverterRule.java:141)
>   at 
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:208)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:631)
>   at 
> org.apache.calcite.tools.Programs.lambda$standard$3(Programs.java:286)
>   at 
> org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:346)
>   at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:189)
>   at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:314)
>   at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:231)
>   at 
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:638)
>   at 
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:502)
>   at 
> org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:472)
>   at 
> org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:231)
>   at 
> org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:550)
>   at 
> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675)
>   at 
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
>   at 
> org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227)
>   at 
> org.apache.calcite.test.CalciteAssert.assertQuery(CalciteAssert.java:522)
>   at 
> org.apache.calcite.test.CalciteAssert$AssertQuery.lambda$returns$1(CalciteAssert.java:1466)
>   at 
> org.apache.calcite.test.CalciteAssert$AssertQuery.withConnection(CalciteAssert.java:1398)
>   at 
> org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1464)
>   at 
>

[jira] [Commented] (CALCITE-3254) AssertionError while deserializing of interval type.

2019-08-16 Thread Wang Yanlin (JIRA)



[ 
https://issues.apache.org/jira/browse/CALCITE-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909481#comment-16909481
 ] 

Wang Yanlin commented on CALCITE-3254:
--

OK, I update the descritpion in pr. Thanks for reviewing.

> AssertionError while deserializing of interval type.
> 
>
> Key: CALCITE-3254
> URL: https://issues.apache.org/jira/browse/CALCITE-3254
> Project: Calcite
>  Issue Type: Bug
>Reporter: Wang Yanlin
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> When deserializing of logical rel haveing interval type, AssertionError 
> occurs.
> The exception stacktrace as follow:
> {code:java}
> java.lang.RuntimeException: java.lang.RuntimeException: 
> java.lang.AssertionError: use createSqlIntervalType() instead
>   at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:181)
>   at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:125)
>   at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:143)
>   at 
> org.apache.calcite.plan.RelWriterTest.testInterval(RelWriterTest.java:639)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
>   at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
>   at 
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
>   at 
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
>   at 
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> Caused by: java.lang.RuntimeException: java.lang.AssertionError: use 
> createSqlIntervalType() instead
>   at 
> org.apache.calcite.rel.externalize.RelJsonReader.readRel(RelJsonReader.java:271)
>   at 
> org.apache.calcite.rel.externalize.RelJsonReader.readRels(RelJsonReader.java:91)
>   at 
> org.apache.calcite.rel.externalize.RelJsonReader.read(RelJsonReader.java:85)
>   at 
> org.apache.calcite.plan.RelWriterTest.lambda$testInterval$8(RelWriterTest.java:644)
>   at 
> org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:130)
>   at 
> org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:915)
>   at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:179)
>   ... 25 more
> Caused by: java.lang.AssertionError: use createSqlIntervalType() instead
>   at 
> org.apache.calcite.sql.type.SqlTypeFactoryImpl.assertBasic(SqlTypeFactoryImpl.java:222)
>   at 
> org.apache.calcite.sql.type.SqlTypeFactoryImpl.createSqlType(SqlTypeFactoryImpl.java:75)
>   at org.apache.calcite.rel.externalize.RelJson.toType(RelJson.java:222)
>   at org.apache.calcite.rel.externalize.RelJson.toRex(RelJson.java:516)
>   at 
> org.apache.calcite.rel.externalize.RelJson.toRexList(RelJson.java:607)
>   at org.apache.calcite.rel.externalize.RelJson.toRex(RelJson.java:447)
>   at 
> org.apache.calcite.rel.externalize.RelJsonReader$2.getExpressionList(RelJsonReader.java:204)
>   at org.apache.calcite.rel.core.Project.(Project.java:100)
>   at 
> org.apache.calcite.rel.logical.LogicalProject.(LogicalProject.java:88)
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
>

[jira] [Commented] (CALCITE-3258) Upgrade jackson-databind from 2.9.9 to 2.9.9.2

2019-08-16 Thread Julian Hyde (JIRA)



[ 
https://issues.apache.org/jira/browse/CALCITE-3258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909460#comment-16909460
 ] 

Julian Hyde commented on CALCITE-3258:
--

I'll try 2.9.9.3.

> Upgrade jackson-databind from 2.9.9 to 2.9.9.2
> --
>
> Key: CALCITE-3258
> URL: https://issues.apache.org/jira/browse/CALCITE-3258
> Project: Calcite
>  Issue Type: Bug
>Reporter: Julian Hyde
>Priority: Major
>
> Upgrade jackson-databind from 2.9.9 to 2.9.9.2.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Commented] (CALCITE-3257) Need to clear RelMetaDataQuery cache after dumping RelNodes and Graphviz

2019-08-16 Thread Volodymyr Vysotskyi (JIRA)



[ 
https://issues.apache.org/jira/browse/CALCITE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909448#comment-16909448
 ] 

Volodymyr Vysotskyi commented on CALCITE-3257:
--

I don't know the exact reason for adding calls to invalidate rel nodes after 
the rule is applied, so I will operate only my assumptions.
I think that it was done to force recalculating rel node cost after the rule is 
applied, so ideally we would have the correct cost for both unchanged and 
updated rel nodes - when the rule is applied, child rel nodes were changed and 
cost of current rel node was calculated considering children costs instead of 
using the outdated cached value.
So ideally for the case when the tree wasn't changed, there is no need to 
invalidate the cache, since cached values should be correct.
One of the issues, fixed in the scope of CALCITE-2018 was the case when several 
instances caches are used in the same time - one cache instance was invalidated 
after the rule is applied, but another one was used during the next rule 
applying.

*As I see from the code, {{RuleQueue.dump()}} does not calculate rel nodes 
cost...*

I think something similar happens in this case so invalidating the cache after 
(or perhaps also before) calling {{dump()}} method will help to avoid this 
issue, but the problem with coexisting several cache instances will still be 
present.

Could you please check that CALCITE-2018 fixes the issue you have observed?

> Need to clear RelMetaDataQuery cache after dumping RelNodes and Graphviz
> 
>
> Key: CALCITE-3257
> URL: https://issues.apache.org/jira/browse/CALCITE-3257
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Reporter: Xiening Dai
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> To repro -
> 1. Set Log4J log level to TRACE. So planner will dump rel node info every 
> time at rule match 
> (https://github.com/apache/calcite/blob/3124a85b93ff2f1b79484c7bd4cc41835d4f1920/core/src/main/java/org/apache/calcite/plan/volcano/RuleQueue.java#L435)
> 2. Run JdbcTest.testNotExistsCorrelated. Get below exception -
> java.lang.AssertionError: rel 
> [rel#63:EnumerableAggregate.ENUMERABLE.[](input=RelSubset#62,group={0})] has 
> lower cost {131.0 rows, 216.0 cpu, 0.0 io} than best cost {131.5 rows, 216.0 
> cpu, 0.0 io} of subset [rel#60:Subset#4.ENUMERABLE.[]]
>   at org.apache.calcite.util.Litmus$1.fail(Litmus.java:31)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.isValid(VolcanoPlanner.java:889)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:852)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:869)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1928)
>   at 
> org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:129)
>   at 
> org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:236)
>   at 
> org.apache.calcite.rel.convert.ConverterRule.onMatch(ConverterRule.java:141)
>   at 
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:208)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:631)
>   at 
> org.apache.calcite.tools.Programs.lambda$standard$3(Programs.java:286)
>   at 
> org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:346)
>   at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:189)
>   at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:314)
>   at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:231)
>   at 
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:638)
>   at 
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:502)
>   at 
> org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:472)
>   at 
> org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:231)
>   at 
> org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:550)
>   at 
> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675)
>   at 
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
>   at 
> org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227)
>   at 
> org.apache.calcite.test.CalciteAssert.assertQuery(CalciteAssert.java:522)
>   at 
> org.apache.calcite.test.CalciteAssert$AssertQuery.lambda$returns$1(CalciteAssert.java:1466)
>   at 
>

[jira] [Resolved] (CALCITE-3214) Add UnionToUnionRule for materialization matching

2019-08-16 Thread Haisheng Yuan (JIRA)



 [ 
https://issues.apache.org/jira/browse/CALCITE-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haisheng Yuan resolved CALCITE-3214.

   Resolution: Fixed
Fix Version/s: 1.21.0

Fixed in 
https://github.com/apache/calcite/commit/a68bfc96bacc95b4b8ab5d185bb5cc1991113769,
 thanks for the PR, [~jinxing6...@126.com]!

> Add UnionToUnionRule for materialization matching
> -
>
> Key: CALCITE-3214
> URL: https://issues.apache.org/jira/browse/CALCITE-3214
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Reporter: jin xing
>Assignee: jin xing
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.21.0
>
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> Below materialization matching fails now
> {code:java}
>   @Test public void testDEV() {
> String sql0 = "select * from \"emps\" where \"empid\" < 300";
> String sql1 = "select * from \"emps\" where \"empid\" > 200";
> checkMaterialize(sql0 + " union all " + sql1, sql1 + " union all " + 
> sql0);
>   }
> {code}
> This issue proposes to add a rule for union matching



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Resolved] (CALCITE-3249) Substitution#getRexShuttle does not consider RexLiteral

2019-08-16 Thread Haisheng Yuan (JIRA)



 [ 
https://issues.apache.org/jira/browse/CALCITE-3249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haisheng Yuan resolved CALCITE-3249.

   Resolution: Fixed
Fix Version/s: 1.21.0

Fixed in 
https://github.com/apache/calcite/commit/0a6f53cb828cd39d78d1423361921bf40636943a,
 thanks for the PR, [~jinxing6...@126.com]!

> Substitution#getRexShuttle does not consider RexLiteral
> ---
>
> Key: CALCITE-3249
> URL: https://issues.apache.org/jira/browse/CALCITE-3249
> Project: Calcite
>  Issue Type: Improvement
>  Components: core
>Reporter: jin xing
>Assignee: jin xing
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 1.21.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Current {{Substitution#getRexShuttle}} does not take {{RexLiteral}} into 
> consideration.
> Thus below query & mv fails matching:
> {code:java}
> MV:
> select deptno, sum(salary), sum(commission), sum(k)
> from
>   (select deptno, salary, commission, 100 as k
>   from emps)
> group by deptno
> Query:
> select deptno, sum(salary), sum(k)
> from
>   (select deptno, salary, 100 as k
>   from emps)
> group by deptno
> {code}
> The root cause is that {{ProjectToProjectUnifyRule}} compensates a 
> {{Project}} which contains {{RexLiteral}}, but 
> {{AggregateOnProjectToAggregateUnifyRule}} works only when the {{Project}} in 
> query is a mapping
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/plan/SubstitutionVisitor.java#L1357



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Commented] (CALCITE-2018) Queries failed with AssertionError: rel has lower cost than best cost of subset

2019-08-16 Thread Xiening Dai (JIRA)



[ 
https://issues.apache.org/jira/browse/CALCITE-2018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909415#comment-16909415
 ] 

Xiening Dai commented on CALCITE-2018:
--

Hi [~julianhyde], [~vvysotskyi] also has some questions. We can continue the 
discussion under CALCITE-3257. Once we agree on the problem, I would 
update/clarify the CL. Thanks.

> Queries failed with AssertionError: rel has lower cost than best cost of 
> subset
> ---
>
> Key: CALCITE-2018
> URL: https://issues.apache.org/jira/browse/CALCITE-2018
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.13.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Danny Chan
>Priority: Critical
>  Labels: pull-request-available
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> *Problem description*
> When rootLogger level is DEBUG, unit tests 
> * MaterializationTest.testMaterializationSubstitution2
> * MaterializationTest.testJoinMaterializationUKFK8
> * MaterializationTest.testJoinMaterializationUKFK6
> * JdbcTest.testWhereNot
> unit tests are failed with error AssertionError: rel has lower cost than best 
> cost of subset.
> Full stack trace for test 
> {{MaterializationTest.testMaterializationSubstitution2}}:
> {noformat}
> java.lang.AssertionError: rel 
> [rel#245:EnumerableUnion.ENUMERABLE.[](input#0=rel#246:Subset#5.ENUMERABLE.[],input#1=rel#239:Subset#6.ENUMERABLE.[0],all=true)]
>  has lower cost {14.0 rows, 19.0 cpu, 0.0 io} than best cost {15.0 rows, 20.0 
> cpu, 0.0 io} of subset [rel#243:Subset#7.ENUMERABLE.[]]
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.validate(VolcanoPlanner.java:906)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:866)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:883)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:101)
>   at 
> org.apache.calcite.rel.AbstractRelNode.onRegister(AbstractRelNode.java:336)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1495)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:863)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:883)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1766)
>   at 
> org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:135)
>   at 
> org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:234)
>   at 
> org.apache.calcite.rel.rules.FilterProjectTransposeRule.onMatch(FilterProjectTransposeRule.java:143)
>   at 
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:212)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:650)
>   at org.apache.calcite.tools.Programs$5.run(Programs.java:326)
>   at 
> org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:387)
>   at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:187)
>   at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:318)
>   at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:229)
>   at 
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:786)
>   at 
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:640)
>   at 
> org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:610)
>   at org.apache.calcite.schema.Schemas.prepare(Schemas.java:346)
>   at 
> org.apache.calcite.materialize.MaterializationService$DefaultTableFactory.createTable(MaterializationService.java:374)
>   at 
> org.apache.calcite.materialize.MaterializationService.defineMaterialization(MaterializationService.java:137)
>   at 
> org.apache.calcite.materialize.MaterializationService.defineMaterialization(MaterializationService.java:99)
>   at 
> org.apache.calcite.schema.impl.MaterializedViewTable$MaterializedViewTableMacro.(MaterializedViewTable.java:110)
>   at 
> org.apache.calcite.schema.impl.MaterializedViewTable$MaterializedViewTableMacro.(MaterializedViewTable.java:100)
>   at 
> org.apache.calcite.schema.impl.MaterializedViewTable.create(MaterializedViewTable.java:81)
>   at org.apache.calcite.model.ModelHandler.visit(ModelHandler.java:364)
>   at 
> org.apache.calcite.model.JsonMaterialization.accept(JsonMaterialization.java:42)
>   at org.apache.calcite.model.JsonSchema.visitChildren(JsonSchema.java:98)
>   at 
>

[jira] [Commented] (CALCITE-3258) Upgrade jackson-databind from 2.9.9 to 2.9.9.2

2019-08-16 Thread Vladimir Sitnikov (JIRA)



[ 
https://issues.apache.org/jira/browse/CALCITE-3258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909408#comment-16909408
 ] 

Vladimir Sitnikov commented on CALCITE-3258:


I would suggest going with 2.9.9.3.
Is there a reason for not using it?

> Upgrade jackson-databind from 2.9.9 to 2.9.9.2
> --
>
> Key: CALCITE-3258
> URL: https://issues.apache.org/jira/browse/CALCITE-3258
> Project: Calcite
>  Issue Type: Bug
>Reporter: Julian Hyde
>Priority: Major
>
> Upgrade jackson-databind from 2.9.9 to 2.9.9.2.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Commented] (CALCITE-3257) Need to clear RelMetaDataQuery cache after dumping RelNodes and Graphviz

2019-08-16 Thread Xiening Dai (JIRA)



[ 
https://issues.apache.org/jira/browse/CALCITE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909402#comment-16909402
 ] 

Xiening Dai commented on CALCITE-3257:
--

I believe CALCITE-2018 is a separate issue while this one needs to be fixed 
too. From my understanding, the meta cache needs to be purged for every rule 
firing, dump() breaks this assumption. Adding this call would make sure cache 
is purged and would actually fix quite a number of failures under TRACE mode. 
But there are still cases where best cost is not properly calculated 
(CALCITE-2018). This fix cannot address those cases (one example 
JdbcTest.testInnerJoinValues) thus it doesn't "hide" anything.

> Need to clear RelMetaDataQuery cache after dumping RelNodes and Graphviz
> 
>
> Key: CALCITE-3257
> URL: https://issues.apache.org/jira/browse/CALCITE-3257
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Reporter: Xiening Dai
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> To repro -
> 1. Set Log4J log level to TRACE. So planner will dump rel node info every 
> time at rule match 
> (https://github.com/apache/calcite/blob/3124a85b93ff2f1b79484c7bd4cc41835d4f1920/core/src/main/java/org/apache/calcite/plan/volcano/RuleQueue.java#L435)
> 2. Run JdbcTest.testNotExistsCorrelated. Get below exception -
> java.lang.AssertionError: rel 
> [rel#63:EnumerableAggregate.ENUMERABLE.[](input=RelSubset#62,group={0})] has 
> lower cost {131.0 rows, 216.0 cpu, 0.0 io} than best cost {131.5 rows, 216.0 
> cpu, 0.0 io} of subset [rel#60:Subset#4.ENUMERABLE.[]]
>   at org.apache.calcite.util.Litmus$1.fail(Litmus.java:31)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.isValid(VolcanoPlanner.java:889)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:852)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:869)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1928)
>   at 
> org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:129)
>   at 
> org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:236)
>   at 
> org.apache.calcite.rel.convert.ConverterRule.onMatch(ConverterRule.java:141)
>   at 
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:208)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:631)
>   at 
> org.apache.calcite.tools.Programs.lambda$standard$3(Programs.java:286)
>   at 
> org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:346)
>   at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:189)
>   at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:314)
>   at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:231)
>   at 
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:638)
>   at 
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:502)
>   at 
> org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:472)
>   at 
> org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:231)
>   at 
> org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:550)
>   at 
> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675)
>   at 
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
>   at 
> org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227)
>   at 
> org.apache.calcite.test.CalciteAssert.assertQuery(CalciteAssert.java:522)
>   at 
> org.apache.calcite.test.CalciteAssert$AssertQuery.lambda$returns$1(CalciteAssert.java:1466)
>   at 
> org.apache.calcite.test.CalciteAssert$AssertQuery.withConnection(CalciteAssert.java:1398)
>   at 
> org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1464)
>   at 
> org.apache.calcite.test.CalciteAssert$AssertQuery.explainMatches(CalciteAssert.java:1561)
>   at 
> org.apache.calcite.test.CalciteAssert$AssertQuery.explainContains(CalciteAssert.java:1556)
>   at 
> org.apache.calcite.test.JdbcTest.testNotExistsCorrelated(JdbcTest.java:4562)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at

[jira] [Created] (CALCITE-3258) Upgrade jackson-databind from 2.9.9 to 2.9.9.2

2019-08-16 Thread Julian Hyde (JIRA)

Julian Hyde created CALCITE-3258:


 Summary: Upgrade jackson-databind from 2.9.9 to 2.9.9.2
 Key: CALCITE-3258
 URL: https://issues.apache.org/jira/browse/CALCITE-3258
 Project: Calcite
  Issue Type: Bug
Reporter: Julian Hyde


Upgrade jackson-databind from 2.9.9 to 2.9.9.2.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Commented] (CALCITE-3257) Need to clear RelMetaDataQuery cache after dumping RelNodes and Graphviz

2019-08-16 Thread Volodymyr Vysotskyi (JIRA)



[ 
https://issues.apache.org/jira/browse/CALCITE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909375#comment-16909375
 ] 

Volodymyr Vysotskyi commented on CALCITE-3257:
--

I don't think that adding a call to RelOptCluster.invalidateMetadataQuery() 
after dumping rel nodes is a good idea. As pointed in one of the comments in 
CALCITE-2018, this issue also happens for the case when a trace is not enabled. 
The proposed fix will just hide the issue for some cases, but more important 
and hard-to-investigate issues like incorrect planning and others will still be 
present.

> Need to clear RelMetaDataQuery cache after dumping RelNodes and Graphviz
> 
>
> Key: CALCITE-3257
> URL: https://issues.apache.org/jira/browse/CALCITE-3257
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Reporter: Xiening Dai
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> To repro -
> 1. Set Log4J log level to TRACE. So planner will dump rel node info every 
> time at rule match 
> (https://github.com/apache/calcite/blob/3124a85b93ff2f1b79484c7bd4cc41835d4f1920/core/src/main/java/org/apache/calcite/plan/volcano/RuleQueue.java#L435)
> 2. Run JdbcTest.testNotExistsCorrelated. Get below exception -
> java.lang.AssertionError: rel 
> [rel#63:EnumerableAggregate.ENUMERABLE.[](input=RelSubset#62,group={0})] has 
> lower cost {131.0 rows, 216.0 cpu, 0.0 io} than best cost {131.5 rows, 216.0 
> cpu, 0.0 io} of subset [rel#60:Subset#4.ENUMERABLE.[]]
>   at org.apache.calcite.util.Litmus$1.fail(Litmus.java:31)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.isValid(VolcanoPlanner.java:889)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:852)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:869)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1928)
>   at 
> org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:129)
>   at 
> org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:236)
>   at 
> org.apache.calcite.rel.convert.ConverterRule.onMatch(ConverterRule.java:141)
>   at 
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:208)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:631)
>   at 
> org.apache.calcite.tools.Programs.lambda$standard$3(Programs.java:286)
>   at 
> org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:346)
>   at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:189)
>   at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:314)
>   at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:231)
>   at 
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:638)
>   at 
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:502)
>   at 
> org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:472)
>   at 
> org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:231)
>   at 
> org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:550)
>   at 
> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675)
>   at 
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
>   at 
> org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227)
>   at 
> org.apache.calcite.test.CalciteAssert.assertQuery(CalciteAssert.java:522)
>   at 
> org.apache.calcite.test.CalciteAssert$AssertQuery.lambda$returns$1(CalciteAssert.java:1466)
>   at 
> org.apache.calcite.test.CalciteAssert$AssertQuery.withConnection(CalciteAssert.java:1398)
>   at 
> org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1464)
>   at 
> org.apache.calcite.test.CalciteAssert$AssertQuery.explainMatches(CalciteAssert.java:1561)
>   at 
> org.apache.calcite.test.CalciteAssert$AssertQuery.explainContains(CalciteAssert.java:1556)
>   at 
> org.apache.calcite.test.JdbcTest.testNotExistsCorrelated(JdbcTest.java:4562)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
>

[jira] [Updated] (CALCITE-3257) Need to clear RelMetaDataQuery cache after dumping RelNodes and Graphviz

2019-08-16 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/CALCITE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated CALCITE-3257:

Labels: pull-request-available  (was: )

> Need to clear RelMetaDataQuery cache after dumping RelNodes and Graphviz
> 
>
> Key: CALCITE-3257
> URL: https://issues.apache.org/jira/browse/CALCITE-3257
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Reporter: Xiening Dai
>Priority: Major
>  Labels: pull-request-available
>
> To repro -
> 1. Set Log4J log level to TRACE. So planner will dump rel node info every 
> time at rule match 
> (https://github.com/apache/calcite/blob/3124a85b93ff2f1b79484c7bd4cc41835d4f1920/core/src/main/java/org/apache/calcite/plan/volcano/RuleQueue.java#L435)
> 2. Run JdbcTest.testNotExistsCorrelated. Get below exception -
> java.lang.AssertionError: rel 
> [rel#63:EnumerableAggregate.ENUMERABLE.[](input=RelSubset#62,group={0})] has 
> lower cost {131.0 rows, 216.0 cpu, 0.0 io} than best cost {131.5 rows, 216.0 
> cpu, 0.0 io} of subset [rel#60:Subset#4.ENUMERABLE.[]]
>   at org.apache.calcite.util.Litmus$1.fail(Litmus.java:31)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.isValid(VolcanoPlanner.java:889)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:852)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:869)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1928)
>   at 
> org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:129)
>   at 
> org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:236)
>   at 
> org.apache.calcite.rel.convert.ConverterRule.onMatch(ConverterRule.java:141)
>   at 
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:208)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:631)
>   at 
> org.apache.calcite.tools.Programs.lambda$standard$3(Programs.java:286)
>   at 
> org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:346)
>   at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:189)
>   at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:314)
>   at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:231)
>   at 
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:638)
>   at 
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:502)
>   at 
> org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:472)
>   at 
> org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:231)
>   at 
> org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:550)
>   at 
> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675)
>   at 
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
>   at 
> org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227)
>   at 
> org.apache.calcite.test.CalciteAssert.assertQuery(CalciteAssert.java:522)
>   at 
> org.apache.calcite.test.CalciteAssert$AssertQuery.lambda$returns$1(CalciteAssert.java:1466)
>   at 
> org.apache.calcite.test.CalciteAssert$AssertQuery.withConnection(CalciteAssert.java:1398)
>   at 
> org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1464)
>   at 
> org.apache.calcite.test.CalciteAssert$AssertQuery.explainMatches(CalciteAssert.java:1561)
>   at 
> org.apache.calcite.test.CalciteAssert$AssertQuery.explainContains(CalciteAssert.java:1556)
>   at 
> org.apache.calcite.test.JdbcTest.testNotExistsCorrelated(JdbcTest.java:4562)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
>

[jira] [Updated] (CALCITE-3257) Need to clear RelMetaDataQuery cache after dumping RelNodes and Graphviz

2019-08-16 Thread Xiening Dai (JIRA)



 [ 
https://issues.apache.org/jira/browse/CALCITE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiening Dai updated CALCITE-3257:
-
Description: 
To repro -

1. Set Log4J log level to TRACE. So planner will dump rel node info every time 
at rule match 
(https://github.com/apache/calcite/blob/3124a85b93ff2f1b79484c7bd4cc41835d4f1920/core/src/main/java/org/apache/calcite/plan/volcano/RuleQueue.java#L435)

2. Run JdbcTest.testNotExistsCorrelated. Get below exception -

java.lang.AssertionError: rel 
[rel#63:EnumerableAggregate.ENUMERABLE.[](input=RelSubset#62,group={0})] has 
lower cost {131.0 rows, 216.0 cpu, 0.0 io} than best cost {131.5 rows, 216.0 
cpu, 0.0 io} of subset [rel#60:Subset#4.ENUMERABLE.[]]

at org.apache.calcite.util.Litmus$1.fail(Litmus.java:31)
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.isValid(VolcanoPlanner.java:889)
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:852)
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:869)
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1928)
at 
org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:129)
at 
org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:236)
at 
org.apache.calcite.rel.convert.ConverterRule.onMatch(ConverterRule.java:141)
at 
org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:208)
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:631)
at 
org.apache.calcite.tools.Programs.lambda$standard$3(Programs.java:286)
at 
org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:346)
at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:189)
at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:314)
at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:231)
at 
org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:638)
at 
org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:502)
at 
org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:472)
at 
org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:231)
at 
org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:550)
at 
org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675)
at 
org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
at 
org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227)
at 
org.apache.calcite.test.CalciteAssert.assertQuery(CalciteAssert.java:522)
at 
org.apache.calcite.test.CalciteAssert$AssertQuery.lambda$returns$1(CalciteAssert.java:1466)
at 
org.apache.calcite.test.CalciteAssert$AssertQuery.withConnection(CalciteAssert.java:1398)
at 
org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1464)
at 
org.apache.calcite.test.CalciteAssert$AssertQuery.explainMatches(CalciteAssert.java:1561)
at 
org.apache.calcite.test.CalciteAssert$AssertQuery.explainContains(CalciteAssert.java:1556)
at 
org.apache.calcite.test.JdbcTest.testNotExistsCorrelated(JdbcTest.java:4562)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at

[jira] [Commented] (CALCITE-3250) Support nested collection type for SqlDataTypeSpec

2019-08-16 Thread Julian Hyde (JIRA)



[ 
https://issues.apache.org/jira/browse/CALCITE-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909307#comment-16909307
 ] 

Julian Hyde commented on CALCITE-3250:
--

Also, this is an extension to SQL grammar, right? Lead with that. You make it 
sound like it was just a change to the SqlDataTypeSpec API.

> Support nested collection type for  SqlDataTypeSpec
> ---
>
> Key: CALCITE-3250
> URL: https://issues.apache.org/jira/browse/CALCITE-3250
> Project: Calcite
>  Issue Type: Sub-task
>  Components: core
>Affects Versions: 1.20.0
>Reporter: Danny Chan
>Assignee: Danny Chan
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.21.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> According to SQL-2011, collection type can be nested:
> {code:sql}
>  ::=
>ARRAY [bracket or trigraph> ]
>  ::=
>   
> {code}
> *Example*:
> {code:sql}
> int array array;
> int multiset array;
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Updated] (CALCITE-3246) RelJsonReader throws NullPointerException while deserializing from JSON a call to a user-defined function (UDF)

2019-08-16 Thread Julian Hyde (JIRA)



 [ 
https://issues.apache.org/jira/browse/CALCITE-3246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julian Hyde updated CALCITE-3246:
-
Summary: RelJsonReader throws NullPointerException while deserializing from 
JSON a call to a user-defined function (UDF)  (was: NullPointerException  while 
deserializing of udf.)

> RelJsonReader throws NullPointerException while deserializing from JSON a 
> call to a user-defined function (UDF)
> ---
>
> Key: CALCITE-3246
> URL: https://issues.apache.org/jira/browse/CALCITE-3246
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Reporter: Wang Yanlin
>Assignee: Chunwei Lei
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.21.0
>
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> when deserializing of logical rel with udf operator, NPE occurs.
> The exception stacktrace as follow.
> {code:java}
> java.lang.RuntimeException: java.lang.NullPointerException
>   at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:181)
>   at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:125)
>   at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:143)
>   at org.apache.calcite.plan.RelWriterTest.testUdf(RelWriterTest.java:598)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
>   at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
>   at 
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
>   at 
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
>   at 
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> Caused by: java.lang.NullPointerException
>   at java.util.Objects.requireNonNull(Objects.java:203)
>   at org.apache.calcite.rex.RexCall.(RexCall.java:83)
>   at org.apache.calcite.rex.RexBuilder.makeCall(RexBuilder.java:237)
>   at org.apache.calcite.rel.externalize.RelJson.toRex(RelJson.java:485)
>   at 
> org.apache.calcite.rel.externalize.RelJsonReader$2.getExpressionList(RelJsonReader.java:204)
>   at org.apache.calcite.rel.core.Project.(Project.java:100)
>   at 
> org.apache.calcite.rel.logical.LogicalProject.(LogicalProject.java:88)
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>   at 
> org.apache.calcite.rel.externalize.RelJsonReader.readRel(RelJsonReader.java:261)
>   at 
> org.apache.calcite.rel.externalize.RelJsonReader.readRels(RelJsonReader.java:91)
>   at 
> org.apache.calcite.rel.externalize.RelJsonReader.read(RelJsonReader.java:85)
>   at 
> org.apache.calcite.plan.RelWriterTest.lambda$testUdf$7(RelWriterTest.java:603)
>   at 
> org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:130)
>   at 
> org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:915)
>   at

[jira] [Commented] (CALCITE-3246) NullPointerException while deserializing of udf.

2019-08-16 Thread Julian Hyde (JIRA)



[ 
https://issues.apache.org/jira/browse/CALCITE-3246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909287#comment-16909287
 ] 

Julian Hyde commented on CALCITE-3246:
--

I force-pushed to master to remove your commit. Let's commit when we're ready.

> NullPointerException  while deserializing of udf.
> -
>
> Key: CALCITE-3246
> URL: https://issues.apache.org/jira/browse/CALCITE-3246
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Reporter: Wang Yanlin
>Assignee: Chunwei Lei
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.21.0
>
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> when deserializing of logical rel with udf operator, NPE occurs.
> The exception stacktrace as follow.
> {code:java}
> java.lang.RuntimeException: java.lang.NullPointerException
>   at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:181)
>   at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:125)
>   at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:143)
>   at org.apache.calcite.plan.RelWriterTest.testUdf(RelWriterTest.java:598)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
>   at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
>   at 
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
>   at 
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
>   at 
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> Caused by: java.lang.NullPointerException
>   at java.util.Objects.requireNonNull(Objects.java:203)
>   at org.apache.calcite.rex.RexCall.(RexCall.java:83)
>   at org.apache.calcite.rex.RexBuilder.makeCall(RexBuilder.java:237)
>   at org.apache.calcite.rel.externalize.RelJson.toRex(RelJson.java:485)
>   at 
> org.apache.calcite.rel.externalize.RelJsonReader$2.getExpressionList(RelJsonReader.java:204)
>   at org.apache.calcite.rel.core.Project.(Project.java:100)
>   at 
> org.apache.calcite.rel.logical.LogicalProject.(LogicalProject.java:88)
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>   at 
> org.apache.calcite.rel.externalize.RelJsonReader.readRel(RelJsonReader.java:261)
>   at 
> org.apache.calcite.rel.externalize.RelJsonReader.readRels(RelJsonReader.java:91)
>   at 
> org.apache.calcite.rel.externalize.RelJsonReader.read(RelJsonReader.java:85)
>   at 
> org.apache.calcite.plan.RelWriterTest.lambda$testUdf$7(RelWriterTest.java:603)
>   at 
> org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:130)
>   at 
> org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:915)
>   at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:179)
>   ... 25 more
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Commented] (CALCITE-2018) Queries failed with AssertionError: rel has lower cost than best cost of subset

2019-08-16 Thread Julian Hyde (JIRA)



[ 
https://issues.apache.org/jira/browse/CALCITE-2018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909209#comment-16909209
 ] 

Julian Hyde commented on CALCITE-2018:
--

I saw the issue just logged for this workaround. But it doesn’t make clear that 
it is just a workaround, nor state its relationship to this, the issue that 
describes the root cause. Can you make sure that the other issue is in proper 
context. 

> Queries failed with AssertionError: rel has lower cost than best cost of 
> subset
> ---
>
> Key: CALCITE-2018
> URL: https://issues.apache.org/jira/browse/CALCITE-2018
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.13.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Danny Chan
>Priority: Critical
>  Labels: pull-request-available
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> *Problem description*
> When rootLogger level is DEBUG, unit tests 
> * MaterializationTest.testMaterializationSubstitution2
> * MaterializationTest.testJoinMaterializationUKFK8
> * MaterializationTest.testJoinMaterializationUKFK6
> * JdbcTest.testWhereNot
> unit tests are failed with error AssertionError: rel has lower cost than best 
> cost of subset.
> Full stack trace for test 
> {{MaterializationTest.testMaterializationSubstitution2}}:
> {noformat}
> java.lang.AssertionError: rel 
> [rel#245:EnumerableUnion.ENUMERABLE.[](input#0=rel#246:Subset#5.ENUMERABLE.[],input#1=rel#239:Subset#6.ENUMERABLE.[0],all=true)]
>  has lower cost {14.0 rows, 19.0 cpu, 0.0 io} than best cost {15.0 rows, 20.0 
> cpu, 0.0 io} of subset [rel#243:Subset#7.ENUMERABLE.[]]
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.validate(VolcanoPlanner.java:906)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:866)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:883)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:101)
>   at 
> org.apache.calcite.rel.AbstractRelNode.onRegister(AbstractRelNode.java:336)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1495)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:863)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:883)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1766)
>   at 
> org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:135)
>   at 
> org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:234)
>   at 
> org.apache.calcite.rel.rules.FilterProjectTransposeRule.onMatch(FilterProjectTransposeRule.java:143)
>   at 
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:212)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:650)
>   at org.apache.calcite.tools.Programs$5.run(Programs.java:326)
>   at 
> org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:387)
>   at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:187)
>   at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:318)
>   at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:229)
>   at 
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:786)
>   at 
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:640)
>   at 
> org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:610)
>   at org.apache.calcite.schema.Schemas.prepare(Schemas.java:346)
>   at 
> org.apache.calcite.materialize.MaterializationService$DefaultTableFactory.createTable(MaterializationService.java:374)
>   at 
> org.apache.calcite.materialize.MaterializationService.defineMaterialization(MaterializationService.java:137)
>   at 
> org.apache.calcite.materialize.MaterializationService.defineMaterialization(MaterializationService.java:99)
>   at 
> org.apache.calcite.schema.impl.MaterializedViewTable$MaterializedViewTableMacro.(MaterializedViewTable.java:110)
>   at 
> org.apache.calcite.schema.impl.MaterializedViewTable$MaterializedViewTableMacro.(MaterializedViewTable.java:100)
>   at 
> org.apache.calcite.schema.impl.MaterializedViewTable.create(MaterializedViewTable.java:81)
>   at org.apache.calcite.model.ModelHandler.visit(ModelHandler.java:364)
>   at 
> org.apache.calcite.model.JsonMaterialization.accept(JsonMaterialization.java:42)
>   at org.apache.calcite.model.JsonSchema.visitChildren(JsonSchema.java:98)
>   at 
>

[jira] [Commented] (CALCITE-3254) AssertionError while deserializing of interval type.

2019-08-16 Thread Julian Hyde (JIRA)



[ 
https://issues.apache.org/jira/browse/CALCITE-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909202#comment-16909202
 ] 

Julian Hyde commented on CALCITE-3254:
--

Can you improve the description? Think about whether you are fixing a bug or 
adding a feature. 

> AssertionError while deserializing of interval type.
> 
>
> Key: CALCITE-3254
> URL: https://issues.apache.org/jira/browse/CALCITE-3254
> Project: Calcite
>  Issue Type: Bug
>Reporter: Wang Yanlin
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> When deserializing of logical rel haveing interval type, AssertionError 
> occurs.
> The exception stacktrace as follow:
> {code:java}
> java.lang.RuntimeException: java.lang.RuntimeException: 
> java.lang.AssertionError: use createSqlIntervalType() instead
>   at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:181)
>   at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:125)
>   at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:143)
>   at 
> org.apache.calcite.plan.RelWriterTest.testInterval(RelWriterTest.java:639)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
>   at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
>   at 
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
>   at 
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
>   at 
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> Caused by: java.lang.RuntimeException: java.lang.AssertionError: use 
> createSqlIntervalType() instead
>   at 
> org.apache.calcite.rel.externalize.RelJsonReader.readRel(RelJsonReader.java:271)
>   at 
> org.apache.calcite.rel.externalize.RelJsonReader.readRels(RelJsonReader.java:91)
>   at 
> org.apache.calcite.rel.externalize.RelJsonReader.read(RelJsonReader.java:85)
>   at 
> org.apache.calcite.plan.RelWriterTest.lambda$testInterval$8(RelWriterTest.java:644)
>   at 
> org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:130)
>   at 
> org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:915)
>   at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:179)
>   ... 25 more
> Caused by: java.lang.AssertionError: use createSqlIntervalType() instead
>   at 
> org.apache.calcite.sql.type.SqlTypeFactoryImpl.assertBasic(SqlTypeFactoryImpl.java:222)
>   at 
> org.apache.calcite.sql.type.SqlTypeFactoryImpl.createSqlType(SqlTypeFactoryImpl.java:75)
>   at org.apache.calcite.rel.externalize.RelJson.toType(RelJson.java:222)
>   at org.apache.calcite.rel.externalize.RelJson.toRex(RelJson.java:516)
>   at 
> org.apache.calcite.rel.externalize.RelJson.toRexList(RelJson.java:607)
>   at org.apache.calcite.rel.externalize.RelJson.toRex(RelJson.java:447)
>   at 
> org.apache.calcite.rel.externalize.RelJsonReader$2.getExpressionList(RelJsonReader.java:204)
>   at org.apache.calcite.rel.core.Project.(Project.java:100)
>   at 
> org.apache.calcite.rel.logical.LogicalProject.(LogicalProject.java:88)
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
>

[jira] [Updated] (CALCITE-3257) Need to clear RelMetaDataQuery cache after dumping RelNodes and Graphviz

2019-08-16 Thread Xiening Dai (JIRA)



 [ 
https://issues.apache.org/jira/browse/CALCITE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiening Dai updated CALCITE-3257:
-
Description: 
To repro -

1. Set Log4J log level to TRACE. So planner will dump rel node info every time 
at rule match 
(https://github.com/apache/calcite/blob/3124a85b93ff2f1b79484c7bd4cc41835d4f1920/core/src/main/java/org/apache/calcite/plan/volcano/RuleQueue.java#L435)

2. Run JdbcTest.testInnerJoinValues. Get below exception -

java.lang.AssertionError: rel 
[rel#2128:EnumerableProject.ENUMERABLE.[](input=RelSubset#2122,EMPNO=$0,DESC=$3)]
 has lower cost {309.75 rows, 215.5 cpu, 0.0 io} than best cost {322.5 rows, 
241.0 cpu, 0.0 io} of subset [rel#2119:Subset#5.ENUMERABLE.[]]

at org.apache.calcite.util.Litmus$1.fail(Litmus.java:31)
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.isValid(VolcanoPlanner.java:888)
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:851)
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:868)
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:90)
at 
org.apache.calcite.rel.AbstractRelNode.onRegister(AbstractRelNode.java:329)
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1656)
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:846)
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:868)
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1927)
at 
org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:129)
at 
org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:236)
at 
org.apache.calcite.rel.rules.AggregateProjectMergeRule.onMatch(AggregateProjectMergeRule.java:74)
at 
org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:208)
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:631)
at 
org.apache.calcite.tools.Programs.lambda$standard$3(Programs.java:286)
at 
org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:346)
at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:189)
at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:314)
at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:231)
at 
org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:638)
at 
org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:502)
at 
org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:472)
at 
org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:231)
at 
org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:550)
at 
org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675)
at 
org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
at 
org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227)
at 
org.apache.calcite.test.CalciteAssert.assertQuery(CalciteAssert.java:522)
at 
org.apache.calcite.test.CalciteAssert$AssertQuery.lambda$returns$1(CalciteAssert.java:1466)
at 
org.apache.calcite.test.CalciteAssert$AssertQuery.withConnection(CalciteAssert.java:1398)
at 
org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1464)
at 
org.apache.calcite.test.CalciteAssert$AssertQuery.explainMatches(CalciteAssert.java:1561)
at 
org.apache.calcite.test.CalciteAssert$AssertQuery.explainContains(CalciteAssert.java:1556)
at 
org.apache.calcite.test.JdbcTest.testInnerJoinValues(JdbcTest.java:2580)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at

[jira] [Comment Edited] (CALCITE-3257) Need to clear RelMetaDataQuery cache after dumping RelNodes and Graphviz

2019-08-16 Thread Xiening Dai (JIRA)



[ 
https://issues.apache.org/jira/browse/CALCITE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909156#comment-16909156
 ] 

Xiening Dai edited comment on CALCITE-3257 at 8/16/19 4:05 PM:
---

Based on current design, RelMetadataQuery.map needs to be cleared between each 
rule firing. This is achieved through RelOptCluster.invalidateMetadataQuery() 
by VolcanoRuleCall.transformTo(). But when trace is enabled, the dump process 
would actually rebuild the meta data cache from previous rel tree. Then the 
subsequent rule firing doesn't get a chance to update rel node cost as it's 
been in the cache.
A simple fix will just add a call to RelOptCluster.invalidateMetadataQuery() 
after dumping rel nodes.


was (Author: xndai):
Based on current design, RelMetadataQuery.map needs to be cleared between each 
rule firing. This is achieve through RelOptCluster.invalidateMetadataQuery() by 
VolcanoRuleCall.transformTo(). But when trace is enabled, the dump process 
would actually rebuild the meta data cache from previous rel tree. Then the 
subsequent rule firing doesn't get a chance to update rel node cost as it's 
been in the cache.
A simple fix will just add a call to RelOptCluster.invalidateMetadataQuery() 
after dumping rel nodes.

> Need to clear RelMetaDataQuery cache after dumping RelNodes and Graphviz
> 
>
> Key: CALCITE-3257
> URL: https://issues.apache.org/jira/browse/CALCITE-3257
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Reporter: Xiening Dai
>Priority: Major
>
> To repro -
> 1. Set Log4J log level to TRACE. So planner will dump rel node info every 
> time at rule match 
> (https://github.com/apache/calcite/blob/3124a85b93ff2f1b79484c7bd4cc41835d4f1920/core/src/main/java/org/apache/calcite/plan/volcano/RuleQueue.java#L435)
> 2. Run JdbcTest.testInnerJoinValues. Get below exception -
> java.lang.AssertionError: rel 
> [rel#2128:EnumerableProject.ENUMERABLE.[](input=RelSubset#2122,EMPNO=$0,DESC=$3)]
>  has lower cost {309.75 rows, 215.5 cpu, 0.0 io} than best cost {322.5 rows, 
> 241.0 cpu, 0.0 io} of subset [rel#2119:Subset#5.ENUMERABLE.[]]
>   at org.apache.calcite.util.Litmus$1.fail(Litmus.java:31)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.isValid(VolcanoPlanner.java:888)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:851)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:868)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:90)
>   at 
> org.apache.calcite.rel.AbstractRelNode.onRegister(AbstractRelNode.java:329)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1656)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:846)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:868)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1927)
>   at 
> org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:129)
>   at 
> org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:236)
>   at 
> org.apache.calcite.rel.rules.AggregateProjectMergeRule.onMatch(AggregateProjectMergeRule.java:74)
>   at 
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:208)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:631)
>   at 
> org.apache.calcite.tools.Programs.lambda$standard$3(Programs.java:286)
>   at 
> org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:346)
>   at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:189)
>   at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:314)
>   at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:231)
>   at 
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:638)
>   at 
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:502)
>   at 
> org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:472)
>   at 
> org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:231)
>   at 
> org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:550)
>   at 
> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675)
>   at 
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
>   at 
> org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227)
>

[jira] [Updated] (CALCITE-3257) Need to clear RelMetaDataQuery cache after dumping RelNodes and Graphviz

2019-08-16 Thread Xiening Dai (JIRA)



 [ 
https://issues.apache.org/jira/browse/CALCITE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiening Dai updated CALCITE-3257:
-
Description: 
To repro -

1. Set Log4J log level to TRACE. So planner will dump rel node info every time 
at rule match 
(https://github.com/apache/calcite/blob/3124a85b93ff2f1b79484c7bd4cc41835d4f1920/core/src/main/java/org/apache/calcite/plan/volcano/RuleQueue.java#L435)

2. Run JdbcTest.testInnerJoinValues. Get below exception -

java.lang.AssertionError: rel 
[rel#2128:EnumerableProject.ENUMERABLE.[](input=RelSubset#2122,EMPNO=$0,DESC=$3)]
 has lower cost {309.75 rows, 215.5 cpu, 0.0 io} than best cost {322.5 rows, 
241.0 cpu, 0.0 io} of subset [rel#2119:Subset#5.ENUMERABLE.[]]

at org.apache.calcite.util.Litmus$1.fail(Litmus.java:31)
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.isValid(VolcanoPlanner.java:888)
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:851)
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:868)
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:90)
at 
org.apache.calcite.rel.AbstractRelNode.onRegister(AbstractRelNode.java:329)
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1656)
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:846)
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:868)
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1927)
at 
org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:129)
at 
org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:236)
at 
org.apache.calcite.rel.rules.AggregateProjectMergeRule.onMatch(AggregateProjectMergeRule.java:74)
at 
org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:208)
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:631)
at 
org.apache.calcite.tools.Programs.lambda$standard$3(Programs.java:286)
at 
org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:346)
at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:189)
at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:314)
at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:231)
at 
org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:638)
at 
org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:502)
at 
org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:472)
at 
org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:231)
at 
org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:550)
at 
org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675)
at 
org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
at 
org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227)
at 
org.apache.calcite.test.CalciteAssert.assertQuery(CalciteAssert.java:522)
at 
org.apache.calcite.test.CalciteAssert$AssertQuery.lambda$returns$1(CalciteAssert.java:1466)
at 
org.apache.calcite.test.CalciteAssert$AssertQuery.withConnection(CalciteAssert.java:1398)
at 
org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1464)
at 
org.apache.calcite.test.CalciteAssert$AssertQuery.explainMatches(CalciteAssert.java:1561)
at 
org.apache.calcite.test.CalciteAssert$AssertQuery.explainContains(CalciteAssert.java:1556)
at 
org.apache.calcite.test.JdbcTest.testInnerJoinValues(JdbcTest.java:2580)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at

[jira] [Created] (CALCITE-3257) Need to clear RelMetaDataQuery cache after dumping RelNodes and Graphviz

2019-08-16 Thread Xiening Dai (JIRA)

Xiening Dai created CALCITE-3257:


 Summary: Need to clear RelMetaDataQuery cache after dumping 
RelNodes and Graphviz
 Key: CALCITE-3257
 URL: https://issues.apache.org/jira/browse/CALCITE-3257
 Project: Calcite
  Issue Type: Bug
  Components: core
Reporter: Xiening Dai


To repro -

1. Set Log4J log level to TRACE. So planner will dump rel node info every time 
at rule match 
(https://github.com/apache/calcite/blob/3124a85b93ff2f1b79484c7bd4cc41835d4f1920/core/src/main/java/org/apache/calcite/plan/volcano/RuleQueue.java#L435)

2. Run JdbcTest.testInnerJoinValues. Get below exception -

java.lang.AssertionError: rel 
[rel#2128:EnumerableProject.ENUMERABLE.[](input=RelSubset#2122,EMPNO=$0,DESC=$3)]
 has lower cost {309.75 rows, 215.5 cpu, 0.0 io} than best cost {322.5 rows, 
241.0 cpu, 0.0 io} of subset [rel#2119:Subset#5.ENUMERABLE.[]]

at org.apache.calcite.util.Litmus$1.fail(Litmus.java:31)
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.isValid(VolcanoPlanner.java:888)
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:851)
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:868)
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:90)
at 
org.apache.calcite.rel.AbstractRelNode.onRegister(AbstractRelNode.java:329)
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1656)
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:846)
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:868)
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1927)
at 
org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:129)
at 
org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:236)
at 
org.apache.calcite.rel.rules.AggregateProjectMergeRule.onMatch(AggregateProjectMergeRule.java:74)
at 
org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:208)
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:631)
at 
org.apache.calcite.tools.Programs.lambda$standard$3(Programs.java:286)
at 
org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:346)
at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:189)
at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:314)
at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:231)
at 
org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:638)
at 
org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:502)
at 
org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:472)
at 
org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:231)
at 
org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:550)
at 
org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675)
at 
org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
at 
org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227)
at 
org.apache.calcite.test.CalciteAssert.assertQuery(CalciteAssert.java:522)
at 
org.apache.calcite.test.CalciteAssert$AssertQuery.lambda$returns$1(CalciteAssert.java:1466)
at 
org.apache.calcite.test.CalciteAssert$AssertQuery.withConnection(CalciteAssert.java:1398)
at 
org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1464)
at 
org.apache.calcite.test.CalciteAssert$AssertQuery.explainMatches(CalciteAssert.java:1561)
at 
org.apache.calcite.test.CalciteAssert$AssertQuery.explainContains(CalciteAssert.java:1556)
at 
org.apache.calcite.test.JdbcTest.testInnerJoinValues(JdbcTest.java:2580)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at

[jira] [Commented] (CALCITE-3257) Need to clear RelMetaDataQuery cache after dumping RelNodes and Graphviz

2019-08-16 Thread Xiening Dai (JIRA)



[ 
https://issues.apache.org/jira/browse/CALCITE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909156#comment-16909156
 ] 

Xiening Dai commented on CALCITE-3257:
--

Based on current design, RelMetadataQuery.map needs to be cleared between each 
rule firing. This is achieve through RelOptCluster.invalidateMetadataQuery() by 
VolcanoRuleCall.transformTo(). But when trace is enabled, the dump process 
would actually rebuild the meta data cache from previous rel tree. Then the 
subsequent rule firing doesn't get a chance to update rel node cost as it's 
been in the cache.
A simple fix will just add a call to RelOptCluster.invalidateMetadataQuery() 
after dumping rel nodes.

> Need to clear RelMetaDataQuery cache after dumping RelNodes and Graphviz
> 
>
> Key: CALCITE-3257
> URL: https://issues.apache.org/jira/browse/CALCITE-3257
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Reporter: Xiening Dai
>Priority: Major
>
> To repro -
> 1. Set Log4J log level to TRACE. So planner will dump rel node info every 
> time at rule match 
> (https://github.com/apache/calcite/blob/3124a85b93ff2f1b79484c7bd4cc41835d4f1920/core/src/main/java/org/apache/calcite/plan/volcano/RuleQueue.java#L435)
> 2. Run JdbcTest.testInnerJoinValues. Get below exception -
> java.lang.AssertionError: rel 
> [rel#2128:EnumerableProject.ENUMERABLE.[](input=RelSubset#2122,EMPNO=$0,DESC=$3)]
>  has lower cost {309.75 rows, 215.5 cpu, 0.0 io} than best cost {322.5 rows, 
> 241.0 cpu, 0.0 io} of subset [rel#2119:Subset#5.ENUMERABLE.[]]
>   at org.apache.calcite.util.Litmus$1.fail(Litmus.java:31)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.isValid(VolcanoPlanner.java:888)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:851)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:868)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:90)
>   at 
> org.apache.calcite.rel.AbstractRelNode.onRegister(AbstractRelNode.java:329)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1656)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:846)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:868)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1927)
>   at 
> org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:129)
>   at 
> org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:236)
>   at 
> org.apache.calcite.rel.rules.AggregateProjectMergeRule.onMatch(AggregateProjectMergeRule.java:74)
>   at 
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:208)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:631)
>   at 
> org.apache.calcite.tools.Programs.lambda$standard$3(Programs.java:286)
>   at 
> org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:346)
>   at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:189)
>   at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:314)
>   at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:231)
>   at 
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:638)
>   at 
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:502)
>   at 
> org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:472)
>   at 
> org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:231)
>   at 
> org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:550)
>   at 
> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675)
>   at 
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
>   at 
> org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227)
>   at 
> org.apache.calcite.test.CalciteAssert.assertQuery(CalciteAssert.java:522)
>   at 
> org.apache.calcite.test.CalciteAssert$AssertQuery.lambda$returns$1(CalciteAssert.java:1466)
>   at 
> org.apache.calcite.test.CalciteAssert$AssertQuery.withConnection(CalciteAssert.java:1398)
>   at 
> org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1464)
>   at 
> org.apache.calcite.test.CalciteAssert$AssertQuery.explainMatches(CalciteAssert.java:1561)
>   at 
>

[jira] [Commented] (CALCITE-3246) NullPointerException while deserializing of udf.

2019-08-16 Thread Wang Yanlin (JIRA)



[ 
https://issues.apache.org/jira/browse/CALCITE-3246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909016#comment-16909016
 ] 

Wang Yanlin commented on CALCITE-3246:
--

The class name is put into the json string in [to json 
function|https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/rel/externalize/RelJson.java#L366].

Any more details on how to check the security implications?

> NullPointerException  while deserializing of udf.
> -
>
> Key: CALCITE-3246
> URL: https://issues.apache.org/jira/browse/CALCITE-3246
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Reporter: Wang Yanlin
>Assignee: Chunwei Lei
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.21.0
>
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> when deserializing of logical rel with udf operator, NPE occurs.
> The exception stacktrace as follow.
> {code:java}
> java.lang.RuntimeException: java.lang.NullPointerException
>   at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:181)
>   at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:125)
>   at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:143)
>   at org.apache.calcite.plan.RelWriterTest.testUdf(RelWriterTest.java:598)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
>   at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
>   at 
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
>   at 
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
>   at 
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> Caused by: java.lang.NullPointerException
>   at java.util.Objects.requireNonNull(Objects.java:203)
>   at org.apache.calcite.rex.RexCall.(RexCall.java:83)
>   at org.apache.calcite.rex.RexBuilder.makeCall(RexBuilder.java:237)
>   at org.apache.calcite.rel.externalize.RelJson.toRex(RelJson.java:485)
>   at 
> org.apache.calcite.rel.externalize.RelJsonReader$2.getExpressionList(RelJsonReader.java:204)
>   at org.apache.calcite.rel.core.Project.(Project.java:100)
>   at 
> org.apache.calcite.rel.logical.LogicalProject.(LogicalProject.java:88)
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>   at 
> org.apache.calcite.rel.externalize.RelJsonReader.readRel(RelJsonReader.java:261)
>   at 
> org.apache.calcite.rel.externalize.RelJsonReader.readRels(RelJsonReader.java:91)
>   at 
> org.apache.calcite.rel.externalize.RelJsonReader.read(RelJsonReader.java:85)
>   at 
> org.apache.calcite.plan.RelWriterTest.lambda$testUdf$7(RelWriterTest.java:603)
>   at 
> org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:130)
>   at 
> org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:915)
>   at

[jira] [Updated] (CALCITE-3256) Add ProjectOnProjectToProjectUnifyRule for materialization matching.

2019-08-16 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/CALCITE-3256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated CALCITE-3256:

Labels: pull-request-available  (was: )

> Add ProjectOnProjectToProjectUnifyRule for materialization matching.
> 
>
> Key: CALCITE-3256
> URL: https://issues.apache.org/jira/browse/CALCITE-3256
> Project: Calcite
>  Issue Type: Improvement
>  Components: core
>Reporter: jin xing
>Assignee: jin xing
>Priority: Minor
>  Labels: pull-request-available
>
> In current implementation of rules in SubstitutionVisitor.java & 
> MaterializedViewSubstitutionVisitor.java, it's quite common to add a 
> compensating Project on top of child node of target(MV-rel) during matching. 
> But afterwards the next round matching should be able to handle such a 
> compensated Project and match upward along the plan tree. Otherwise we fail 
> the matching. After all, the goal of matching is to transform the query and 
> let a complete 'target' show up in the transformed query plan.
> I found cases where the compensated Project cannot be properly handled.
> {code:java}
> MV:
> select deptno, sum(salary) + 2, sum(commission)
> from emps
> group by deptno
> Query:
> select deptno, sum(salary) + 2
> from emps
> group by deptno
> {code}
> After matching of the Aggregates, a compensating Project is added, but 
> afterwards matching fails to handle it.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (CALCITE-3256) Add ProjectOnProjectToProjectUnifyRule for materialization matching.

2019-08-16 Thread jin xing (JIRA)

jin xing created CALCITE-3256:
-

 Summary: Add ProjectOnProjectToProjectUnifyRule for 
materialization matching.
 Key: CALCITE-3256
 URL: https://issues.apache.org/jira/browse/CALCITE-3256
 Project: Calcite
  Issue Type: Improvement
  Components: core
Reporter: jin xing
Assignee: jin xing


In current implementation of rules in SubstitutionVisitor.java & 
MaterializedViewSubstitutionVisitor.java, it's quite common to add a 
compensating Project on top of child node of target(MV-rel) during matching. 
But afterwards the next round matching should be able to handle such a 
compensated Project and match upward along the plan tree. Otherwise we fail the 
matching. After all, the goal of matching is to transform the query and let a 
complete 'target' show up in the transformed query plan.

I found cases where the compensated Project cannot be properly handled.

{code:java}
MV:
select deptno, sum(salary) + 2, sum(commission)
from emps
group by deptno

Query:
select deptno, sum(salary) + 2
from emps
group by deptno
{code}

After matching of the Aggregates, a compensating Project is added, but 
afterwards matching fails to handle it.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Updated] (CALCITE-3255) CannotPlanException: There are not enough rules to produce a node with desired properties

2019-08-16 Thread Water Cut Off (JIRA)



 [ 
https://issues.apache.org/jira/browse/CALCITE-3255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Water Cut Off updated CALCITE-3255:
---
Description: 
I want to use calcite volcano planner to estimate sql cost,but it doesn't work 
when i try to test a case,The exception is blow:

{color:#FF}org.apache.calcite.plan.RelOptPlanner$CannotPlanException: There 
are not enough rules to produce a node with desired properties: 
convention=NONE, sort=[]. All the inputs have relevant nodes, however the cost 
is still infinite.{color}
{color:#FF} Root: rel#40:Subset#4.NONE.[]{color}
{color:#FF} Original rel:{color}

 

 

  was:
I want to use calcite volcano planner to estimate sql cost,but it doesn't work 
when i try to test a case,The exception is blow:



org.apache.calcite.plan.RelOptPlanner$CannotPlanException: There are not enough 
rules to produce a node with desired properties: convention=NONE, sort=[]. All 
the inputs have relevant nodes, however the cost is still infinite.
Root: rel#40:Subset#4.NONE.[]
Original rel:

 

 


> CannotPlanException: There are not enough rules to produce a node with 
> desired properties
> -
>
> Key: CALCITE-3255
> URL: https://issues.apache.org/jira/browse/CALCITE-3255
> Project: Calcite
>  Issue Type: Bug
> Environment: !image-2019-08-16-18-09-57-964.png!
>Reporter: Water Cut Off
>Priority: Major
> Attachments: image-2019-08-16-18-09-57-964.png
>
>
> I want to use calcite volcano planner to estimate sql cost,but it doesn't 
> work when i try to test a case,The exception is blow:
> {color:#FF}org.apache.calcite.plan.RelOptPlanner$CannotPlanException: 
> There are not enough rules to produce a node with desired properties: 
> convention=NONE, sort=[]. All the inputs have relevant nodes, however the 
> cost is still infinite.{color}
> {color:#FF} Root: rel#40:Subset#4.NONE.[]{color}
> {color:#FF} Original rel:{color}
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (CALCITE-3255) CannotPlanException: There are not enough rules to produce a node with desired properties

2019-08-16 Thread Water Cut Off (JIRA)

Water Cut Off created CALCITE-3255:
--

 Summary: CannotPlanException: There are not enough rules to 
produce a node with desired properties
 Key: CALCITE-3255
 URL: https://issues.apache.org/jira/browse/CALCITE-3255
 Project: Calcite
  Issue Type: Bug
 Environment: !image-2019-08-16-18-09-57-964.png!
Reporter: Water Cut Off
 Attachments: image-2019-08-16-18-09-57-964.png

I want to use calcite volcano planner to estimate sql cost,but it doesn't work 
when i try to test a case,The exception is blow:



org.apache.calcite.plan.RelOptPlanner$CannotPlanException: There are not enough 
rules to produce a node with desired properties: convention=NONE, sort=[]. All 
the inputs have relevant nodes, however the cost is still infinite.
Root: rel#40:Subset#4.NONE.[]
Original rel:

 

 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Updated] (CALCITE-3254) AssertionError while deserializing of interval type.

2019-08-16 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/CALCITE-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated CALCITE-3254:

Labels: pull-request-available  (was: )

> AssertionError while deserializing of interval type.
> 
>
> Key: CALCITE-3254
> URL: https://issues.apache.org/jira/browse/CALCITE-3254
> Project: Calcite
>  Issue Type: Bug
>Reporter: Wang Yanlin
>Priority: Major
>  Labels: pull-request-available
>
> When deserializing of logical rel haveing interval type, AssertionError 
> occurs.
> The exception stacktrace as follow:
> {code:java}
> java.lang.RuntimeException: java.lang.RuntimeException: 
> java.lang.AssertionError: use createSqlIntervalType() instead
>   at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:181)
>   at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:125)
>   at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:143)
>   at 
> org.apache.calcite.plan.RelWriterTest.testInterval(RelWriterTest.java:639)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
>   at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
>   at 
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
>   at 
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
>   at 
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> Caused by: java.lang.RuntimeException: java.lang.AssertionError: use 
> createSqlIntervalType() instead
>   at 
> org.apache.calcite.rel.externalize.RelJsonReader.readRel(RelJsonReader.java:271)
>   at 
> org.apache.calcite.rel.externalize.RelJsonReader.readRels(RelJsonReader.java:91)
>   at 
> org.apache.calcite.rel.externalize.RelJsonReader.read(RelJsonReader.java:85)
>   at 
> org.apache.calcite.plan.RelWriterTest.lambda$testInterval$8(RelWriterTest.java:644)
>   at 
> org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:130)
>   at 
> org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:915)
>   at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:179)
>   ... 25 more
> Caused by: java.lang.AssertionError: use createSqlIntervalType() instead
>   at 
> org.apache.calcite.sql.type.SqlTypeFactoryImpl.assertBasic(SqlTypeFactoryImpl.java:222)
>   at 
> org.apache.calcite.sql.type.SqlTypeFactoryImpl.createSqlType(SqlTypeFactoryImpl.java:75)
>   at org.apache.calcite.rel.externalize.RelJson.toType(RelJson.java:222)
>   at org.apache.calcite.rel.externalize.RelJson.toRex(RelJson.java:516)
>   at 
> org.apache.calcite.rel.externalize.RelJson.toRexList(RelJson.java:607)
>   at org.apache.calcite.rel.externalize.RelJson.toRex(RelJson.java:447)
>   at 
> org.apache.calcite.rel.externalize.RelJsonReader$2.getExpressionList(RelJsonReader.java:204)
>   at org.apache.calcite.rel.core.Project.(Project.java:100)
>   at 
> org.apache.calcite.rel.logical.LogicalProject.(LogicalProject.java:88)
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>   at 
>

[jira] [Created] (CALCITE-3254) AssertionError while deserializing of interval type.

2019-08-16 Thread Wang Yanlin (JIRA)

Wang Yanlin created CALCITE-3254:


 Summary: AssertionError while deserializing of interval type.
 Key: CALCITE-3254
 URL: https://issues.apache.org/jira/browse/CALCITE-3254
 Project: Calcite
  Issue Type: Bug
Reporter: Wang Yanlin


When deserializing of logical rel haveing interval type, AssertionError occurs.

The exception stacktrace as follow:
{code:java}
java.lang.RuntimeException: java.lang.RuntimeException: 
java.lang.AssertionError: use createSqlIntervalType() instead

at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:181)
at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:125)
at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:143)
at 
org.apache.calcite.plan.RelWriterTest.testInterval(RelWriterTest.java:639)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at 
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at 
com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
at 
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at 
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
Caused by: java.lang.RuntimeException: java.lang.AssertionError: use 
createSqlIntervalType() instead
at 
org.apache.calcite.rel.externalize.RelJsonReader.readRel(RelJsonReader.java:271)
at 
org.apache.calcite.rel.externalize.RelJsonReader.readRels(RelJsonReader.java:91)
at 
org.apache.calcite.rel.externalize.RelJsonReader.read(RelJsonReader.java:85)
at 
org.apache.calcite.plan.RelWriterTest.lambda$testInterval$8(RelWriterTest.java:644)
at 
org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:130)
at 
org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:915)
at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:179)
... 25 more
Caused by: java.lang.AssertionError: use createSqlIntervalType() instead
at 
org.apache.calcite.sql.type.SqlTypeFactoryImpl.assertBasic(SqlTypeFactoryImpl.java:222)
at 
org.apache.calcite.sql.type.SqlTypeFactoryImpl.createSqlType(SqlTypeFactoryImpl.java:75)
at org.apache.calcite.rel.externalize.RelJson.toType(RelJson.java:222)
at org.apache.calcite.rel.externalize.RelJson.toRex(RelJson.java:516)
at 
org.apache.calcite.rel.externalize.RelJson.toRexList(RelJson.java:607)
at org.apache.calcite.rel.externalize.RelJson.toRex(RelJson.java:447)
at 
org.apache.calcite.rel.externalize.RelJsonReader$2.getExpressionList(RelJsonReader.java:204)
at org.apache.calcite.rel.core.Project.(Project.java:100)
at 
org.apache.calcite.rel.logical.LogicalProject.(LogicalProject.java:88)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at 
org.apache.calcite.rel.externalize.RelJsonReader.readRel(RelJsonReader.java:261)
... 31 more
{code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Comment Edited] (CALCITE-2302) Implicit type cast support

2019-08-16 Thread Danny Chan (JIRA)



[ 
https://issues.apache.org/jira/browse/CALCITE-2302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908903#comment-16908903
 ] 

Danny Chan edited comment on CALCITE-2302 at 8/16/19 9:45 AM:
--

Thanks so much for your review [~hyuan] !

I have add a new SqlConformance for PostgreSql and SqlServer which seem the 
only engine that returns integer for 2 integers division. I checked again for 
the behavior:
 * Hive and Spark all return double, Hive also have a "A DIV B" function to 
return integer
 * PostgreSql and SQL_SERVER returns integer.
 * Mysql returns double while it has a DIV() function to return integer


was (Author: danny0405):
Thanks so much for your review [~hyuan] !

I have add a new SqlConformance for PostgreSql and SqlServer which seem the 
only engine that returns integer for 2 integers division. I checked again for 
the behavior:
 * Hive and Spark all return double, Hive also have a "A DIV B" function to 
returns integer
 * PostgreSql and SQL_SERVER returns integer.
 * Mysql returns double while it has a DIV() function to returns integer

> Implicit type cast support
> --
>
> Key: CALCITE-2302
> URL: https://issues.apache.org/jira/browse/CALCITE-2302
> Project: Calcite
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 1.17.0
>Reporter: Danny Chan
>Assignee: Danny Chan
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.21.0
>
>  Time Spent: 6.5h
>  Remaining Estimate: 0h
>
> Now many DBs have support implicit type cast, eg: SqlServer, Oracle, Hive.
> Implicit type cast is an useful function for many cases, So we should support 
> this.
> I checkout Calcite code and found that:
>  # Now we use a validator to validate our operands types[ through kinds of 
> namespaces and scopes ]
>  # Most of the validations will finally goes to
> {code:java}
> SqlOperator.validateOperands
> {code}
>  # which will use validation logic defined in corresponding 
> SqlOperandTypeChecker
> What i'm confused about is where should i put the implicit type cast logic 
> in? I figured out 2 ways:
>  # Supply a tool class/rules to add casts into a parsed SqlNode tree which 
> will then go through the validation logic later on.
>  # Unleash the validation logic in kinds of SqlOperandTypeChecker, then 
> modify the RelNode/RexNodes tree converted from a validated SqlNode tree to 
> add in casts through custom RelOptRules.
> So guys, which of the 2 ways should i go, or if there are better way to do 
> this?
> I need your help.
>  
> Updated 18-05-30:
> Hi guys, i have made a PR in 
> [CALCITE-2302|https://github.com/apache/calcite/pull/706]
> This is design doc: [Calcite Implicit Type Cast 
> Design|https://docs.google.com/document/d/1g2RUnLXyp_LjUlO-wbblKuP5hqEu3a_2Mt2k4dh6RwU/edit?usp=sharing].
> This is the conversion types mapping: [Conversion Types 
> Mapping|https://docs.google.com/spreadsheets/d/1GhleX5h5W8-kJKh7NMJ4vtoE78pwfaZRJl88ULX_MgU/edit?usp=sharing].
> I really appreciate your suggestions, thx.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Commented] (CALCITE-2302) Implicit type cast support

2019-08-16 Thread Danny Chan (JIRA)



[ 
https://issues.apache.org/jira/browse/CALCITE-2302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908903#comment-16908903
 ] 

Danny Chan commented on CALCITE-2302:
-

Thanks so much for your review [~hyuan] !

I have add a new SqlConformance for PostgreSql and SqlServer which seem the 
only engine that returns integer for 2 integers division. I checked again for 
the behavior:
 * Hive and Spark all return double, Hive also have a "A DIV B" function to 
returns integer
 * PostgreSql and SQL_SERVER returns integer.
 * Mysql returns double while it has a DIV() function to returns integer

> Implicit type cast support
> --
>
> Key: CALCITE-2302
> URL: https://issues.apache.org/jira/browse/CALCITE-2302
> Project: Calcite
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 1.17.0
>Reporter: Danny Chan
>Assignee: Danny Chan
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.21.0
>
>  Time Spent: 6.5h
>  Remaining Estimate: 0h
>
> Now many DBs have support implicit type cast, eg: SqlServer, Oracle, Hive.
> Implicit type cast is an useful function for many cases, So we should support 
> this.
> I checkout Calcite code and found that:
>  # Now we use a validator to validate our operands types[ through kinds of 
> namespaces and scopes ]
>  # Most of the validations will finally goes to
> {code:java}
> SqlOperator.validateOperands
> {code}
>  # which will use validation logic defined in corresponding 
> SqlOperandTypeChecker
> What i'm confused about is where should i put the implicit type cast logic 
> in? I figured out 2 ways:
>  # Supply a tool class/rules to add casts into a parsed SqlNode tree which 
> will then go through the validation logic later on.
>  # Unleash the validation logic in kinds of SqlOperandTypeChecker, then 
> modify the RelNode/RexNodes tree converted from a validated SqlNode tree to 
> add in casts through custom RelOptRules.
> So guys, which of the 2 ways should i go, or if there are better way to do 
> this?
> I need your help.
>  
> Updated 18-05-30:
> Hi guys, i have made a PR in 
> [CALCITE-2302|https://github.com/apache/calcite/pull/706]
> This is design doc: [Calcite Implicit Type Cast 
> Design|https://docs.google.com/document/d/1g2RUnLXyp_LjUlO-wbblKuP5hqEu3a_2Mt2k4dh6RwU/edit?usp=sharing].
> This is the conversion types mapping: [Conversion Types 
> Mapping|https://docs.google.com/spreadsheets/d/1GhleX5h5W8-kJKh7NMJ4vtoE78pwfaZRJl88ULX_MgU/edit?usp=sharing].
> I really appreciate your suggestions, thx.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

45 matches

Mail list logo