[jira] [Created] (CALCITE-2997) Avoid pushing down join condition in SqlToRelConverter

[jin xing (JIRA)]

jin xing created CALCITE-2997:
-

 Summary: Avoid pushing down join condition in SqlToRelConverter
 Key: CALCITE-2997
 URL: https://issues.apache.org/jira/browse/CALCITE-2997
 Project: Calcite
  Issue Type: Bug
Reporter: jin xing


In current code, *SqlToRelConverter:createJoin* is calling 
*RelOptUtil.pushDownJoinConditions* for optimization. And we can find below 
conversion from *SqlNode* to *RelNode*:
{code:java}
SqlNode:
select * from A join B on A.x = B.x * 2

RelNode (Logical-Plan):
Join (condition:col0=col1)
|-Project(x as col0)
| |-Scan
|-Project(x * 2 as col1)
  |-Scan{code}
As we can see the logical plan(*RelNode*) posted above is not the pure 
reflection of the original SQL String(*SqlNode*). The optimization is mixed 
into the phase on which AST is converted to Logical-Plan. Actually optimizing 
rule of JoinPushExpressionsRule is doing exactly the same kind of thing. Shall 
we just keep the optimization inside Optimized-Logical-Plan ? I mean shall we 
avoid calling *RelOptUtil.pushDownJoinConditions* in 
*SqlToRelConverter:createJoin*

I raised this issue because that we are doing something based on the 
Logical-Plan. And it makes us really confused that the Logical-Plan doesn't 
corresponds to SqlNode. 

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (CALCITE-2997) Avoid pushing down join condition in SqlToRelConverter

[jin xing (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-2997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing updated CALCITE-2997:
--
Description: 
In current code, *SqlToRelConverter:createJoin* is calling 
*RelOptUtil.pushDownJoinConditions* for optimization. And we can find below 
conversion from *SqlNode* to *RelNode*:
{code:java}
SqlNode:
select * from A join B on A.x = B.x * 2

RelNode (Logical-Plan):
Join (condition:col0=col1)
|-Project(x as col0)
| |-Scan(A)
|-Project(x * 2 as col1)
  |-Scan(B){code}
As we can see the logical plan(*RelNode*) posted above is not the pure 
reflection of the original SQL String(*SqlNode*). The optimization is mixed 
into the phase on which AST is converted to Logical-Plan. Actually optimizing 
rule of JoinPushExpressionsRule is doing exactly the same kind of thing. Shall 
we just keep the optimization inside Optimized-Logical-Plan ? I mean shall we 
avoid calling *RelOptUtil.pushDownJoinConditions* in 
*SqlToRelConverter:createJoin*

I raised this issue because that we are doing something based on the 
Logical-Plan. And it makes us really confused that the Logical-Plan doesn't 
corresponds to SqlNode. 

 

 

  was:
In current code, *SqlToRelConverter:createJoin* is calling 
*RelOptUtil.pushDownJoinConditions* for optimization. And we can find below 
conversion from *SqlNode* to *RelNode*:
{code:java}
SqlNode:
select * from A join B on A.x = B.x * 2

RelNode (Logical-Plan):
Join (condition:col0=col1)
|-Project(x as col0)
| |-Scan
|-Project(x * 2 as col1)
  |-Scan{code}
As we can see the logical plan(*RelNode*) posted above is not the pure 
reflection of the original SQL String(*SqlNode*). The optimization is mixed 
into the phase on which AST is converted to Logical-Plan. Actually optimizing 
rule of JoinPushExpressionsRule is doing exactly the same kind of thing. Shall 
we just keep the optimization inside Optimized-Logical-Plan ? I mean shall we 
avoid calling *RelOptUtil.pushDownJoinConditions* in 
*SqlToRelConverter:createJoin*

I raised this issue because that we are doing something based on the 
Logical-Plan. And it makes us really confused that the Logical-Plan doesn't 
corresponds to SqlNode. 

 

 


> Avoid pushing down join condition in SqlToRelConverter
> --
>
> Key: CALCITE-2997
> URL: https://issues.apache.org/jira/browse/CALCITE-2997
> Project: Calcite
>  Issue Type: Bug
>Reporter: jin xing
>Priority: Major
>
> In current code, *SqlToRelConverter:createJoin* is calling 
> *RelOptUtil.pushDownJoinConditions* for optimization. And we can find below 
> conversion from *SqlNode* to *RelNode*:
> {code:java}
> SqlNode:
> select * from A join B on A.x = B.x * 2
> RelNode (Logical-Plan):
> Join (condition:col0=col1)
> |-Project(x as col0)
> | |-Scan(A)
> |-Project(x * 2 as col1)
>   |-Scan(B){code}
> As we can see the logical plan(*RelNode*) posted above is not the pure 
> reflection of the original SQL String(*SqlNode*). The optimization is mixed 
> into the phase on which AST is converted to Logical-Plan. Actually optimizing 
> rule of JoinPushExpressionsRule is doing exactly the same kind of thing. 
> Shall we just keep the optimization inside Optimized-Logical-Plan ? I mean 
> shall we avoid calling *RelOptUtil.pushDownJoinConditions* in 
> *SqlToRelConverter:createJoin*
> I raised this issue because that we are doing something based on the 
> Logical-Plan. And it makes us really confused that the Logical-Plan doesn't 
> corresponds to SqlNode. 
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CALCITE-2997) Avoid pushing down join condition in SqlToRelConverter

[jin xing (JIRA)]

[ 
https://issues.apache.org/jira/browse/CALCITE-2997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16817584#comment-16817584
 ] 

jin xing commented on CALCITE-2997:
---

THX a lot for comments ~Chunwei and ~Danny,

Our internal framework have some parsing rules on the RelNode pattern.

Yes~ For sure and no doubt that the logical plan is right. But for the concept 
of parsing a SQL, there's no need to do the optimization redundantly in both 
phase of AST->Logical-Plan and phase of Logical-Plan->Optimized-Logical-Plan.

I just try to ask that if there's any possibility to make the phase of 
AST->Logical-Plan more clear.

 

 

> Avoid pushing down join condition in SqlToRelConverter
> --
>
> Key: CALCITE-2997
> URL: https://issues.apache.org/jira/browse/CALCITE-2997
> Project: Calcite
>  Issue Type: Bug
>Reporter: jin xing
>Priority: Major
>
> In current code, *SqlToRelConverter:createJoin* is calling 
> *RelOptUtil.pushDownJoinConditions* for optimization. And we can find below 
> conversion from *SqlNode* to *RelNode*:
> {code:java}
> SqlNode:
> select * from A join B on A.x = B.x * 2
> RelNode (Logical-Plan):
> Join (condition:col0=col1)
> |-Project(x as col0)
> | |-Scan(A)
> |-Project(x * 2 as col1)
>   |-Scan(B){code}
> As we can see the logical plan(*RelNode*) posted above is not the pure 
> reflection of the original SQL String(*SqlNode*). The optimization is mixed 
> into the phase on which AST is converted to Logical-Plan. Actually optimizing 
> rule of JoinPushExpressionsRule is doing exactly the same kind of thing. 
> Shall we just keep the optimization inside Optimized-Logical-Plan ? I mean 
> shall we avoid calling *RelOptUtil.pushDownJoinConditions* in 
> *SqlToRelConverter:createJoin*
> I raised this issue because that we are doing something based on the 
> Logical-Plan. And it makes us really confused that the Logical-Plan doesn't 
> corresponds to SqlNode. 
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (CALCITE-3167) Remove redundant overriding methods of equals&hashcode in EnumerableTableScan.java

[jin xing (JIRA)]

jin xing created CALCITE-3167:
-

 Summary: Remove redundant overriding methods of equals&hashcode in 
EnumerableTableScan.java
 Key: CALCITE-3167
 URL: https://issues.apache.org/jira/browse/CALCITE-3167
 Project: Calcite
  Issue Type: Improvement
  Components: core
Affects Versions: 1.19.0
Reporter: jin xing


In current code of EnumerableTableScan.java, methods of equals&hashCode are 
overrided for matching of EnumerableTableScans.
But after optimizing with the same HEP planner, EnumerableTableScans from two 
plans but with the same digest will the share the same Java object.
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/plan/RelOptMaterializations.java#L192
I think it's ok to remove the redundant overriding methods in 
EnumerableTableScan.java



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (CALCITE-3167) Remove redundant overriding methods of equals&hashcode in EnumerableTableScan.java

[jin xing (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing updated CALCITE-3167:
--
Description: 
In current code of EnumerableTableScan.java, methods of equals&hashCode are 
overrided for matching of EnumerableTableScans.
While after optimizing with the same HEP planner, EnumerableTableScans from two 
plans but with the same digest will the share the same Java object.
 
[https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/plan/RelOptMaterializations.java#L192]
 I think it's ok to remove the redundant overriding methods in 
EnumerableTableScan.java

  was:
In current code of EnumerableTableScan.java, methods of equals&hashCode are 
overrided for matching of EnumerableTableScans.
But after optimizing with the same HEP planner, EnumerableTableScans from two 
plans but with the same digest will the share the same Java object.
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/plan/RelOptMaterializations.java#L192
I think it's ok to remove the redundant overriding methods in 
EnumerableTableScan.java


> Remove redundant overriding methods of equals&hashcode in 
> EnumerableTableScan.java
> --
>
> Key: CALCITE-3167
> URL: https://issues.apache.org/jira/browse/CALCITE-3167
> Project: Calcite
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 1.19.0
>Reporter: jin xing
>Priority: Minor
>
> In current code of EnumerableTableScan.java, methods of equals&hashCode are 
> overrided for matching of EnumerableTableScans.
> While after optimizing with the same HEP planner, EnumerableTableScans from 
> two plans but with the same digest will the share the same Java object.
>  
> [https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/plan/RelOptMaterializations.java#L192]
>  I think it's ok to remove the redundant overriding methods in 
> EnumerableTableScan.java



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CALCITE-3167) Remove overriding equals and hashCode methods in EnumerableTableScan

[jin xing (JIRA)]

[ 
https://issues.apache.org/jira/browse/CALCITE-3167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16877833#comment-16877833
 ] 

jin xing commented on CALCITE-3167:
---

Thanks a lot for reply [~julianhyde]
 # I also think it's a good idea to prohibit overriding {{equals}} and 
{{hashCode}} for {{RelNode}} and users should define the identity case by case;
 # As for overriding {{equals}} and {{hashCode}} in {{AbstractRelNode}} and 
making them final, how about implement by {{digest ?}} i.e. {{RelNodes equals 
with each other when they have the same}} {{digests}} and we generate 
{{hashCode}} of {{RelNode}} by {{digest}}'s {{hashCode}}. User should guarantee 
that all key information should be included in {{digest}}
 # I think the idea in CALCITE-1216 is straightforward. Before that change, 
'materialization-substitution' only happens at the very start of 
{{findBestExp}}. After the change, {{MaterializedViewFilterScanRule.java does}} 
the materialization-substitution during running optimization-rules. Thus query 
plan has a  chance to be 'normalized' after some optimization rules and then 
get optimized by materialization-substitution. This might introduce extra cost 
since the materialization-substitution tends to be a heavy process. 
 # Digging the code, I did't find any necessity for the {{overrides}} in 
{{EnumerableScan}}. After removing them, I passed all the tests. I think 
existing tests are sufficient. Because {{EnumerableTableScanRule}} will create 
a new EnumerableTableScan every time it meets a LogicalTableScan, i.e. 
{{query}} and {{materialization}} don't share the same Java object for 
table-scan.

> Remove overriding equals and hashCode methods in EnumerableTableScan
> 
>
> Key: CALCITE-3167
> URL: https://issues.apache.org/jira/browse/CALCITE-3167
> Project: Calcite
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 1.19.0
>Reporter: jin xing
>Priority: Minor
>  Labels: pull-request-available
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> In current code of {{EnumerableTableScan}}, methods of equals and hashCode 
> are overridden for matching of {{EnumerableTableScan}}s.
> While after optimizing with the same HEP planner, {{EnumerableTableScan}}s 
> from two plans but with the same digest will the share the same Java object. 
> See 
> [RelOptMaterializations|https://github.com/apache/calcite/blob/adf4cc4dc5cdb9f5e49c85d10f46a2fdcd831ccf/core/src/main/java/org/apache/calcite/plan/RelOptMaterializations.java#L192].
> I think it's ok to remove the redundant overriding methods in 
> {{EnumerableTableScan}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (CALCITE-3167) Remove overriding equals and hashCode methods in EnumerableTableScan

[jin xing (JIRA)]

[ 
https://issues.apache.org/jira/browse/CALCITE-3167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16877833#comment-16877833
 ] 

jin xing edited comment on CALCITE-3167 at 7/3/19 1:28 PM:
---

Thanks a lot for reply [~julianhyde]
 # I also think it's a good idea to prohibit overriding {{equals}} and 
{{hashCode}} for {{RelNode}} and users should define the identity case by case;
 # As for overriding {{equals}} and {{hashCode}} in {{AbstractRelNode}} and 
making them final, how about implement by {{digest ?}} i.e. {{RelNodes}} equals 
with each other when they have the same {{digests}} and we generate 
{{hashCode}} of {{RelNode}} by {{digest}}'s {{hashCode}}. User should guarantee 
that all key information should be included in {{digest}}
 # I think the idea in CALCITE-1216 is straightforward. Before that change, 
'materialization-substitution' only happens at the very start of 
{{findBestExp}}. After the change, {{MaterializedViewFilterScanRule.java does}} 
the materialization-substitution during running optimization-rules. Thus query 
plan has a  chance to be 'normalized' after some optimization rules and then 
get optimized by materialization-substitution. This might introduce extra cost 
since the materialization-substitution tends to be a heavy process. 
 # Digging the code, I did't find any necessity for the {{overrides}} in 
{{EnumerableScan}}. After removing them, I passed all the tests. I think 
existing tests are sufficient. Because {{EnumerableTableScanRule}} will create 
a new EnumerableTableScan every time it meets a LogicalTableScan, i.e. 
{{query}} and {{materialization}} don't share the same Java object for 
table-scan.


was (Author: jinxing6...@126.com):
Thanks a lot for reply [~julianhyde]
 # I also think it's a good idea to prohibit overriding {{equals}} and 
{{hashCode}} for {{RelNode}} and users should define the identity case by case;
 # As for overriding {{equals}} and {{hashCode}} in {{AbstractRelNode}} and 
making them final, how about implement by {{digest ?}} i.e. {{RelNodes equals 
with each other when they have the same}} {{digests}} and we generate 
{{hashCode}} of {{RelNode}} by {{digest}}'s {{hashCode}}. User should guarantee 
that all key information should be included in {{digest}}
 # I think the idea in CALCITE-1216 is straightforward. Before that change, 
'materialization-substitution' only happens at the very start of 
{{findBestExp}}. After the change, {{MaterializedViewFilterScanRule.java does}} 
the materialization-substitution during running optimization-rules. Thus query 
plan has a  chance to be 'normalized' after some optimization rules and then 
get optimized by materialization-substitution. This might introduce extra cost 
since the materialization-substitution tends to be a heavy process. 
 # Digging the code, I did't find any necessity for the {{overrides}} in 
{{EnumerableScan}}. After removing them, I passed all the tests. I think 
existing tests are sufficient. Because {{EnumerableTableScanRule}} will create 
a new EnumerableTableScan every time it meets a LogicalTableScan, i.e. 
{{query}} and {{materialization}} don't share the same Java object for 
table-scan.

> Remove overriding equals and hashCode methods in EnumerableTableScan
> 
>
> Key: CALCITE-3167
> URL: https://issues.apache.org/jira/browse/CALCITE-3167
> Project: Calcite
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 1.19.0
>Reporter: jin xing
>Priority: Minor
>  Labels: pull-request-available
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> In current code of {{EnumerableTableScan}}, methods of equals and hashCode 
> are overridden for matching of {{EnumerableTableScan}}s.
> While after optimizing with the same HEP planner, {{EnumerableTableScan}}s 
> from two plans but with the same digest will the share the same Java object. 
> See 
> [RelOptMaterializations|https://github.com/apache/calcite/blob/adf4cc4dc5cdb9f5e49c85d10f46a2fdcd831ccf/core/src/main/java/org/apache/calcite/plan/RelOptMaterializations.java#L192].
> I think it's ok to remove the redundant overriding methods in 
> {{EnumerableTableScan}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CALCITE-3167) Remove overriding equals and hashCode methods in EnumerableTableScan

[jin xing (JIRA)]

[ 
https://issues.apache.org/jira/browse/CALCITE-3167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16878284#comment-16878284
 ] 

jin xing commented on CALCITE-3167:
---

[~amargoor]  Gentle ping~ It's great if you can share some thoughts on this :)

> Remove overriding equals and hashCode methods in EnumerableTableScan
> 
>
> Key: CALCITE-3167
> URL: https://issues.apache.org/jira/browse/CALCITE-3167
> Project: Calcite
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 1.19.0
>Reporter: jin xing
>Priority: Minor
>  Labels: pull-request-available
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> In current code of {{EnumerableTableScan}}, methods of equals and hashCode 
> are overridden for matching of {{EnumerableTableScan}}s.
> While after optimizing with the same HEP planner, {{EnumerableTableScan}}s 
> from two plans but with the same digest will the share the same Java object. 
> See 
> [RelOptMaterializations|https://github.com/apache/calcite/blob/adf4cc4dc5cdb9f5e49c85d10f46a2fdcd831ccf/core/src/main/java/org/apache/calcite/plan/RelOptMaterializations.java#L192].
> I think it's ok to remove the redundant overriding methods in 
> {{EnumerableTableScan}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (CALCITE-3182) Trim unused fields for plan of materialized-view before matching.

[jin xing (JIRA)]

jin xing created CALCITE-3182:
-

 Summary: Trim unused fields for plan of materialized-view before 
matching.
 Key: CALCITE-3182
 URL: https://issues.apache.org/jira/browse/CALCITE-3182
 Project: Calcite
  Issue Type: Improvement
Reporter: jin xing


In current code, before matching query with materialized-view, unused fields of 
query is trimed but materialized-view is not. Thus below simple SQL fails to be 
matched though query and materialized-
view are exactly the same
```
 @Test public void testMaterializationAfterTrimingOfUnusedFields() {
 String sql =
 "select \"y\".\"deptno\", \"y\".\"name\", \"x\".\"sum_salary\"\n" +
 "from\n" +
 " (select \"deptno\", sum(\"salary\") \"sum_salary\" from \"emps\" group by 
\"deptno\") \"x\"\n" +
 " join\n" +
 " \"depts\" \"y\"\n" +
 " on \"x\".\"deptno\"=\"y\".\"deptno\"\n";
 checkMaterialize(sql, sql);
 }
```
Checking \{{CalciteMaterializer}} 
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/prepare/CalciteMaterializer.java#L83
 , I think the code intends to do the trimming, but didn't call the method.
Since query is trimed for unused fields in anyway 
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/prepare/CalcitePrepareImpl.java#L995
 , thus I think there's no n
ecessity to keep the materialized-view un-trimmed



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (CALCITE-3182) Trim unused fields for plan of materialized-view before matching.

[jin xing (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing updated CALCITE-3182:
--
Description: 
In current code, before matching query with materialized-view, unused fields of 
query is trimed but materialized-view is not. Thus below simple SQL fails to be 
matched though query and materialized-
view are exactly the same
```
  @Test public void testMaterializationAfterTrimingOfUnusedFields() {
String sql =
  "select \"y\".\"deptno\", \"y\".\"name\", \"x\".\"sum_salary\"\n" +
"from\n" +
"  (select \"deptno\", sum(\"salary\") \"sum_salary\" from \"emps\" 
group by \"deptno\") \"x\"\n" +
"  join\n" +
"  \"depts\" \"y\"\n" +
"  on \"x\".\"deptno\"=\"y\".\"deptno\"\n";
checkMaterialize(sql, sql);
  }
```
Checking {{CalciteMaterializer}} 
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/prepare/CalciteMaterializer.java#L83
 , I think the code intends to do the trimming, but didn't call the method.
Since query is trimed for unused fields in anyway 
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/prepare/CalcitePrepareImpl.java#L995
 , thus I think there's no n
ecessity to keep the materialized-view un-trimmed

  was:
In current code, before matching query with materialized-view, unused fields of 
query is trimed but materialized-view is not. Thus below simple SQL fails to be 
matched though query and materialized-
view are exactly the same
```
 @Test public void testMaterializationAfterTrimingOfUnusedFields() {
 String sql =
 "select \"y\".\"deptno\", \"y\".\"name\", \"x\".\"sum_salary\"\n" +
 "from\n" +
 " (select \"deptno\", sum(\"salary\") \"sum_salary\" from \"emps\" group by 
\"deptno\") \"x\"\n" +
 " join\n" +
 " \"depts\" \"y\"\n" +
 " on \"x\".\"deptno\"=\"y\".\"deptno\"\n";
 checkMaterialize(sql, sql);
 }
```
Checking \{{CalciteMaterializer}} 
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/prepare/CalciteMaterializer.java#L83
 , I think the code intends to do the trimming, but didn't call the method.
Since query is trimed for unused fields in anyway 
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/prepare/CalcitePrepareImpl.java#L995
 , thus I think there's no n
ecessity to keep the materialized-view un-trimmed


> Trim unused fields for plan of materialized-view before matching.
> -
>
> Key: CALCITE-3182
> URL: https://issues.apache.org/jira/browse/CALCITE-3182
> Project: Calcite
>  Issue Type: Improvement
>Reporter: jin xing
>Priority: Major
>
> In current code, before matching query with materialized-view, unused fields 
> of query is trimed but materialized-view is not. Thus below simple SQL fails 
> to be matched though query and materialized-
> view are exactly the same
> ```
>   @Test public void testMaterializationAfterTrimingOfUnusedFields() {
> String sql =
>   "select \"y\".\"deptno\", \"y\".\"name\", \"x\".\"sum_salary\"\n" +
> "from\n" +
> "  (select \"deptno\", sum(\"salary\") \"sum_salary\" from \"emps\" 
> group by \"deptno\") \"x\"\n" +
> "  join\n" +
> "  \"depts\" \"y\"\n" +
> "  on \"x\".\"deptno\"=\"y\".\"deptno\"\n";
> checkMaterialize(sql, sql);
>   }
> ```
> Checking {{CalciteMaterializer}} 
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/prepare/CalciteMaterializer.java#L83
>  , I think the code intends to do the trimming, but didn't call the method.
> Since query is trimed for unused fields in anyway 
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/prepare/CalcitePrepareImpl.java#L995
>  , thus I think there's no n
> ecessity to keep the materialized-view un-trimmed



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (CALCITE-3182) Trim unused fields for plan of materialized-view before matching.

[jin xing (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing updated CALCITE-3182:
--
Description: 
In current code, before matching query with materialized-view, unused fields of 
query is trimmed but materialized-view is not. Thus below simple SQL fails to 
be matched though query and materialized-
 view are exactly the same:
{code:java}
@Test public void testMaterializationAfterTrimingOfUnusedFields() {
String sql =
"select \"y\".\"deptno\", \"y\".\"name\", \"x\".\"sum_salary\"\n" +
"from\n" +
" (select \"deptno\", sum(\"salary\") \"sum_salary\" from \"emps\" group by 
\"deptno\") \"x\"\n" +
" join\n" +
" \"depts\" \"y\"\n" +
" on \"x\".\"deptno\"=\"y\".\"deptno\"\n";
checkMaterialize(sql, sql);
}{code}
 Checking {{CalciteMaterializer}} 
[https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/prepare/CalciteMaterializer.java#L83]
 , I think the code intends to do the trimming, but didn't call the method.
 Since unused fields of query is trimmed anyway 
[https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/prepare/CalcitePrepareImpl.java#L995]
 , thus I think there's no necessity to keep the materialized-view un-trimmed

  was:
In current code, before matching query with materialized-view, unused fields of 
query is trimmed but materialized-view is not. Thus below simple SQL fails to 
be matched though query and materialized-
 view are exactly the same:
{code:java}
@Test public void testMaterializationAfterTrimingOfUnusedFields() {
String sql =
"select \"y\".\"deptno\", \"y\".\"name\", \"x\".\"sum_salary\"\n" +
"from\n" +
" (select \"deptno\", sum(\"salary\") \"sum_salary\" from \"emps\" group by 
\"deptno\") \"x\"\n" +
" join\n" +
" \"depts\" \"y\"\n" +
" on \"x\".\"deptno\"=\"y\".\"deptno\"\n";
checkMaterialize(sql, sql);
}{code}
 

Checking {{CalciteMaterializer}} 
[https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/prepare/CalciteMaterializer.java#L83]
 , I think the code intends to do the trimming, but didn't call the method.
 Since unused fields of query is trimmed anyway 
[https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/prepare/CalcitePrepareImpl.java#L995]
 , thus I think there's no necessity to keep the materialized-view un-trimmed


> Trim unused fields for plan of materialized-view before matching.
> -
>
> Key: CALCITE-3182
> URL: https://issues.apache.org/jira/browse/CALCITE-3182
> Project: Calcite
>  Issue Type: Improvement
>Reporter: jin xing
>Priority: Major
>
> In current code, before matching query with materialized-view, unused fields 
> of query is trimmed but materialized-view is not. Thus below simple SQL fails 
> to be matched though query and materialized-
>  view are exactly the same:
> {code:java}
> @Test public void testMaterializationAfterTrimingOfUnusedFields() {
> String sql =
> "select \"y\".\"deptno\", \"y\".\"name\", \"x\".\"sum_salary\"\n" +
> "from\n" +
> " (select \"deptno\", sum(\"salary\") \"sum_salary\" from \"emps\" group by 
> \"deptno\") \"x\"\n" +
> " join\n" +
> " \"depts\" \"y\"\n" +
> " on \"x\".\"deptno\"=\"y\".\"deptno\"\n";
> checkMaterialize(sql, sql);
> }{code}
>  Checking {{CalciteMaterializer}} 
> [https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/prepare/CalciteMaterializer.java#L83]
>  , I think the code intends to do the trimming, but didn't call the method.
>  Since unused fields of query is trimmed anyway 
> [https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/prepare/CalcitePrepareImpl.java#L995]
>  , thus I think there's no necessity to keep the materialized-view un-trimmed



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (CALCITE-3182) Trim unused fields for plan of materialized-view before matching.

[jin xing (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing updated CALCITE-3182:
--
Description: 
In current code, before matching query with materialized-view, unused fields of 
query is trimmed but materialized-view is not. Thus below simple SQL fails to 
be matched though query and materialized-
 view are exactly the same:
{code:java}
@Test public void testMaterializationAfterTrimingOfUnusedFields() {
String sql =
"select \"y\".\"deptno\", \"y\".\"name\", \"x\".\"sum_salary\"\n" +
"from\n" +
" (select \"deptno\", sum(\"salary\") \"sum_salary\" from \"emps\" group by 
\"deptno\") \"x\"\n" +
" join\n" +
" \"depts\" \"y\"\n" +
" on \"x\".\"deptno\"=\"y\".\"deptno\"\n";
checkMaterialize(sql, sql);
}{code}
 

Checking {{CalciteMaterializer}} 
[https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/prepare/CalciteMaterializer.java#L83]
 , I think the code intends to do the trimming, but didn't call the method.
 Since unused fields of query is trimmed anyway 
[https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/prepare/CalcitePrepareImpl.java#L995]
 , thus I think there's no necessity to keep the materialized-view un-trimmed

  was:
In current code, before matching query with materialized-view, unused fields of 
query is trimed but materialized-view is not. Thus below simple SQL fails to be 
matched though query and materialized-
view are exactly the same
```
  @Test public void testMaterializationAfterTrimingOfUnusedFields() {
String sql =
  "select \"y\".\"deptno\", \"y\".\"name\", \"x\".\"sum_salary\"\n" +
"from\n" +
"  (select \"deptno\", sum(\"salary\") \"sum_salary\" from \"emps\" 
group by \"deptno\") \"x\"\n" +
"  join\n" +
"  \"depts\" \"y\"\n" +
"  on \"x\".\"deptno\"=\"y\".\"deptno\"\n";
checkMaterialize(sql, sql);
  }
```
Checking {{CalciteMaterializer}} 
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/prepare/CalciteMaterializer.java#L83
 , I think the code intends to do the trimming, but didn't call the method.
Since query is trimed for unused fields in anyway 
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/prepare/CalcitePrepareImpl.java#L995
 , thus I think there's no n
ecessity to keep the materialized-view un-trimmed


> Trim unused fields for plan of materialized-view before matching.
> -
>
> Key: CALCITE-3182
> URL: https://issues.apache.org/jira/browse/CALCITE-3182
> Project: Calcite
>  Issue Type: Improvement
>Reporter: jin xing
>Priority: Major
>
> In current code, before matching query with materialized-view, unused fields 
> of query is trimmed but materialized-view is not. Thus below simple SQL fails 
> to be matched though query and materialized-
>  view are exactly the same:
> {code:java}
> @Test public void testMaterializationAfterTrimingOfUnusedFields() {
> String sql =
> "select \"y\".\"deptno\", \"y\".\"name\", \"x\".\"sum_salary\"\n" +
> "from\n" +
> " (select \"deptno\", sum(\"salary\") \"sum_salary\" from \"emps\" group by 
> \"deptno\") \"x\"\n" +
> " join\n" +
> " \"depts\" \"y\"\n" +
> " on \"x\".\"deptno\"=\"y\".\"deptno\"\n";
> checkMaterialize(sql, sql);
> }{code}
>  
> Checking {{CalciteMaterializer}} 
> [https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/prepare/CalciteMaterializer.java#L83]
>  , I think the code intends to do the trimming, but didn't call the method.
>  Since unused fields of query is trimmed anyway 
> [https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/prepare/CalcitePrepareImpl.java#L995]
>  , thus I think there's no necessity to keep the materialized-view un-trimmed



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (CALCITE-3182) Trim unused fields for plan of materialized-view before matching.

[jin xing (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing updated CALCITE-3182:
--
Issue Type: Bug  (was: Improvement)

> Trim unused fields for plan of materialized-view before matching.
> -
>
> Key: CALCITE-3182
> URL: https://issues.apache.org/jira/browse/CALCITE-3182
> Project: Calcite
>  Issue Type: Bug
>Reporter: jin xing
>Priority: Major
>
> In current code, before matching query with materialized-view, unused fields 
> of query is trimmed but materialized-view is not. Thus below simple SQL fails 
> to be matched though query and materialized-
>  view are exactly the same:
> {code:java}
> @Test public void testMaterializationAfterTrimingOfUnusedFields() {
> String sql =
> "select \"y\".\"deptno\", \"y\".\"name\", \"x\".\"sum_salary\"\n" +
> "from\n" +
> " (select \"deptno\", sum(\"salary\") \"sum_salary\" from \"emps\" group by 
> \"deptno\") \"x\"\n" +
> " join\n" +
> " \"depts\" \"y\"\n" +
> " on \"x\".\"deptno\"=\"y\".\"deptno\"\n";
> checkMaterialize(sql, sql);
> }{code}
>  Checking {{CalciteMaterializer}} 
> [https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/prepare/CalciteMaterializer.java#L83]
>  , I think the code intends to do the trimming, but didn't call the method.
>  Since unused fields of query is trimmed anyway 
> [https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/prepare/CalcitePrepareImpl.java#L995]
>  , thus I think there's no necessity to keep the materialized-view un-trimmed



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CALCITE-3113) Equivalent MutableAggregates with different row types fail with AssertionError

[jin xing (JIRA)]

[ 
https://issues.apache.org/jira/browse/CALCITE-3113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880958#comment-16880958
 ] 

jin xing commented on CALCITE-3113:
---

In SQLs below, Aggregations operator in both materialization and query should 
definitely matches with each other and regarded as equivalent. 
{code:java}
"select count as c from \"emps\" group by \"empid\"",
"select count + 1 as c from \"emps\" group by \"empid\"");{code}

row-type checking in 
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/plan/SubstitutionVisitor.java#L498
 is too strict. In my understanding, we might just need to assure that the 
final result from {{SubstitutionVisitor}} should have same row-type as before.

> Equivalent MutableAggregates with different row types fail with AssertionError
> --
>
> Key: CALCITE-3113
> URL: https://issues.apache.org/jira/browse/CALCITE-3113
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.19.0
>Reporter: Feng Zhu
>Priority: Major
>
> Add test case in MaterializationTest: 
> {code:java}
> @Test public void testAggregateAlias() {
>   checkMaterialize(
>   "select count(*) as c from \"emps\" group by \"empid\"",
>   "select count(*) + 1 as c from \"emps\" group by \"empid\"");
> }
> {code}
>  It fails due to different rowtype.
> {code:java}
> java.lang.AssertionError
>     at 
> org.apache.calcite.plan.SubstitutionVisitor.go(SubstitutionVisitor.java:504)
>     at 
> org.apache.calcite.plan.SubstitutionVisitor.go(SubstitutionVisitor.java:465)
>     at 
> org.apache.calcite.plan.MaterializedViewSubstitutionVisitor.go(MaterializedViewSubstitutionVisitor.java:56)
>     at 
> org.apache.calcite.plan.RelOptMaterializations.substitute(RelOptMaterializations.java:200)
>     at 
> org.apache.calcite.plan.RelOptMaterializations.useMaterializedViews(RelOptMaterializations.java:72)
>     at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.registerMaterializations(VolcanoPlanner.java:347)
> {code}
> However, according to MutableAggregate's hashCode&equals implementation, this 
> materialization can be reused, i.e., queryDescedant=targetDescendant.
> {code:java}
> queryDescendant: RecordType(JavaType(int) empid, BIGINT $f1)
> =
> Aggregate(groupSet: {0}, groupSets: [{0}], calls: [COUNT()])
>   Project(projects: [$0])
>     Scan(table: [hr, emps])
> targetDescendant: RecordType(JavaType(int) empid, BIGINT C)
> =
> Aggregate(groupSet: {0}, groupSets: [{0}], calls: [COUNT()])
>   Project(projects: [$0])
>     Scan(table: [hr, emps])
> {code}
> So, how can we align them?
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (CALCITE-3113) Equivalent MutableAggregates with different row types fail with AssertionError

[jin xing (JIRA)]

[ 
https://issues.apache.org/jira/browse/CALCITE-3113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880958#comment-16880958
 ] 

jin xing edited comment on CALCITE-3113 at 7/9/19 6:11 AM:
---

In SQLs below, Aggregations operator in both materialization and query should 
definitely matches with each other and regarded as equivalent. 
{code:java}
materialization:   "select count as c from \"emps\" group by \"empid\"",
query:   "select count + 1 as c from \"emps\" group by \"empid\"");{code}

row-type checking in 
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/plan/SubstitutionVisitor.java#L498
 is too strict. In my understanding, we might just need to assure that the 
final result from {{SubstitutionVisitor}} should have same row-type as before.


was (Author: jinxing6...@126.com):
In SQLs below, Aggregations operator in both materialization and query should 
definitely matches with each other and regarded as equivalent. 
{code:java}
"select count as c from \"emps\" group by \"empid\"",
"select count + 1 as c from \"emps\" group by \"empid\"");{code}

row-type checking in 
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/plan/SubstitutionVisitor.java#L498
 is too strict. In my understanding, we might just need to assure that the 
final result from {{SubstitutionVisitor}} should have same row-type as before.

> Equivalent MutableAggregates with different row types fail with AssertionError
> --
>
> Key: CALCITE-3113
> URL: https://issues.apache.org/jira/browse/CALCITE-3113
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.19.0
>Reporter: Feng Zhu
>Priority: Major
>
> Add test case in MaterializationTest: 
> {code:java}
> @Test public void testAggregateAlias() {
>   checkMaterialize(
>   "select count(*) as c from \"emps\" group by \"empid\"",
>   "select count(*) + 1 as c from \"emps\" group by \"empid\"");
> }
> {code}
>  It fails due to different rowtype.
> {code:java}
> java.lang.AssertionError
>     at 
> org.apache.calcite.plan.SubstitutionVisitor.go(SubstitutionVisitor.java:504)
>     at 
> org.apache.calcite.plan.SubstitutionVisitor.go(SubstitutionVisitor.java:465)
>     at 
> org.apache.calcite.plan.MaterializedViewSubstitutionVisitor.go(MaterializedViewSubstitutionVisitor.java:56)
>     at 
> org.apache.calcite.plan.RelOptMaterializations.substitute(RelOptMaterializations.java:200)
>     at 
> org.apache.calcite.plan.RelOptMaterializations.useMaterializedViews(RelOptMaterializations.java:72)
>     at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.registerMaterializations(VolcanoPlanner.java:347)
> {code}
> However, according to MutableAggregate's hashCode&equals implementation, this 
> materialization can be reused, i.e., queryDescedant=targetDescendant.
> {code:java}
> queryDescendant: RecordType(JavaType(int) empid, BIGINT $f1)
> =
> Aggregate(groupSet: {0}, groupSets: [{0}], calls: [COUNT()])
>   Project(projects: [$0])
>     Scan(table: [hr, emps])
> targetDescendant: RecordType(JavaType(int) empid, BIGINT C)
> =
> Aggregate(groupSet: {0}, groupSets: [{0}], calls: [COUNT()])
>   Project(projects: [$0])
>     Scan(table: [hr, emps])
> {code}
> So, how can we align them?
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CALCITE-3113) Equivalent MutableAggregates with different row types fail with AssertionError

[jin xing (JIRA)]

[ 
https://issues.apache.org/jira/browse/CALCITE-3113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16881286#comment-16881286
 ] 

jin xing commented on CALCITE-3113:
---

I made a PR to illustrate my idea, I'm not sure it's as your expectation 
[~rubenql][~donnyzone]
Please give a brief look at it when you have time. If [~donnyzone] you are 
already working on this, I can close my PR.

> Equivalent MutableAggregates with different row types fail with AssertionError
> --
>
> Key: CALCITE-3113
> URL: https://issues.apache.org/jira/browse/CALCITE-3113
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.19.0
>Reporter: Feng Zhu
>Priority: Major
>
> Add test case in MaterializationTest: 
> {code:java}
> @Test public void testAggregateAlias() {
>   checkMaterialize(
>   "select count(*) as c from \"emps\" group by \"empid\"",
>   "select count(*) + 1 as c from \"emps\" group by \"empid\"");
> }
> {code}
>  It fails due to different rowtype.
> {code:java}
> java.lang.AssertionError
>     at 
> org.apache.calcite.plan.SubstitutionVisitor.go(SubstitutionVisitor.java:504)
>     at 
> org.apache.calcite.plan.SubstitutionVisitor.go(SubstitutionVisitor.java:465)
>     at 
> org.apache.calcite.plan.MaterializedViewSubstitutionVisitor.go(MaterializedViewSubstitutionVisitor.java:56)
>     at 
> org.apache.calcite.plan.RelOptMaterializations.substitute(RelOptMaterializations.java:200)
>     at 
> org.apache.calcite.plan.RelOptMaterializations.useMaterializedViews(RelOptMaterializations.java:72)
>     at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.registerMaterializations(VolcanoPlanner.java:347)
> {code}
> However, according to MutableAggregate's hashCode&equals implementation, this 
> materialization can be reused, i.e., queryDescedant=targetDescendant.
> {code:java}
> queryDescendant: RecordType(JavaType(int) empid, BIGINT $f1)
> =
> Aggregate(groupSet: {0}, groupSets: [{0}], calls: [COUNT()])
>   Project(projects: [$0])
>     Scan(table: [hr, emps])
> targetDescendant: RecordType(JavaType(int) empid, BIGINT C)
> =
> Aggregate(groupSet: {0}, groupSets: [{0}], calls: [COUNT()])
>   Project(projects: [$0])
>     Scan(table: [hr, emps])
> {code}
> So, how can we align them?
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (CALCITE-3113) Equivalent MutableAggregates with different row types fail with AssertionError

[jin xing (JIRA)]

[ 
https://issues.apache.org/jira/browse/CALCITE-3113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16881286#comment-16881286
 ] 

jin xing edited comment on CALCITE-3113 at 7/9/19 3:02 PM:
---

I made a PR (https://github.com/apache/calcite/pull/1310) to illustrate my 
idea, I'm not sure it's as your expectation [~rubenql][~donnyzone]
Please give a brief look at it when you have time. If [~donnyzone] you are 
already working on this, I can close my PR.


was (Author: jinxing6...@126.com):
I made a PR to illustrate my idea, I'm not sure it's as your expectation 
[~rubenql][~donnyzone]
Please give a brief look at it when you have time. If [~donnyzone] you are 
already working on this, I can close my PR.

> Equivalent MutableAggregates with different row types fail with AssertionError
> --
>
> Key: CALCITE-3113
> URL: https://issues.apache.org/jira/browse/CALCITE-3113
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.19.0
>Reporter: Feng Zhu
>Priority: Major
>
> Add test case in MaterializationTest: 
> {code:java}
> @Test public void testAggregateAlias() {
>   checkMaterialize(
>   "select count(*) as c from \"emps\" group by \"empid\"",
>   "select count(*) + 1 as c from \"emps\" group by \"empid\"");
> }
> {code}
>  It fails due to different rowtype.
> {code:java}
> java.lang.AssertionError
>     at 
> org.apache.calcite.plan.SubstitutionVisitor.go(SubstitutionVisitor.java:504)
>     at 
> org.apache.calcite.plan.SubstitutionVisitor.go(SubstitutionVisitor.java:465)
>     at 
> org.apache.calcite.plan.MaterializedViewSubstitutionVisitor.go(MaterializedViewSubstitutionVisitor.java:56)
>     at 
> org.apache.calcite.plan.RelOptMaterializations.substitute(RelOptMaterializations.java:200)
>     at 
> org.apache.calcite.plan.RelOptMaterializations.useMaterializedViews(RelOptMaterializations.java:72)
>     at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.registerMaterializations(VolcanoPlanner.java:347)
> {code}
> However, according to MutableAggregate's hashCode&equals implementation, this 
> materialization can be reused, i.e., queryDescedant=targetDescendant.
> {code:java}
> queryDescendant: RecordType(JavaType(int) empid, BIGINT $f1)
> =
> Aggregate(groupSet: {0}, groupSets: [{0}], calls: [COUNT()])
>   Project(projects: [$0])
>     Scan(table: [hr, emps])
> targetDescendant: RecordType(JavaType(int) empid, BIGINT C)
> =
> Aggregate(groupSet: {0}, groupSets: [{0}], calls: [COUNT()])
>   Project(projects: [$0])
>     Scan(table: [hr, emps])
> {code}
> So, how can we align them?
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CALCITE-3113) Equivalent MutableAggregates with different row types fail with AssertionError

[jin xing (JIRA)]

[ 
https://issues.apache.org/jira/browse/CALCITE-3113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16881388#comment-16881388
 ] 

jin xing commented on CALCITE-3113:
---

THX a lot [~rubenql], I will refine by your comments.

> Equivalent MutableAggregates with different row types fail with AssertionError
> --
>
> Key: CALCITE-3113
> URL: https://issues.apache.org/jira/browse/CALCITE-3113
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.19.0
>Reporter: Feng Zhu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Add test case in MaterializationTest: 
> {code:java}
> @Test public void testAggregateAlias() {
>   checkMaterialize(
>   "select count(*) as c from \"emps\" group by \"empid\"",
>   "select count(*) + 1 as c from \"emps\" group by \"empid\"");
> }
> {code}
>  It fails due to different rowtype.
> {code:java}
> java.lang.AssertionError
>     at 
> org.apache.calcite.plan.SubstitutionVisitor.go(SubstitutionVisitor.java:504)
>     at 
> org.apache.calcite.plan.SubstitutionVisitor.go(SubstitutionVisitor.java:465)
>     at 
> org.apache.calcite.plan.MaterializedViewSubstitutionVisitor.go(MaterializedViewSubstitutionVisitor.java:56)
>     at 
> org.apache.calcite.plan.RelOptMaterializations.substitute(RelOptMaterializations.java:200)
>     at 
> org.apache.calcite.plan.RelOptMaterializations.useMaterializedViews(RelOptMaterializations.java:72)
>     at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.registerMaterializations(VolcanoPlanner.java:347)
> {code}
> However, according to MutableAggregate's hashCode&equals implementation, this 
> materialization can be reused, i.e., queryDescedant=targetDescendant.
> {code:java}
> queryDescendant: RecordType(JavaType(int) empid, BIGINT $f1)
> =
> Aggregate(groupSet: {0}, groupSets: [{0}], calls: [COUNT()])
>   Project(projects: [$0])
>     Scan(table: [hr, emps])
> targetDescendant: RecordType(JavaType(int) empid, BIGINT C)
> =
> Aggregate(groupSet: {0}, groupSets: [{0}], calls: [COUNT()])
>   Project(projects: [$0])
>     Scan(table: [hr, emps])
> {code}
> So, how can we align them?
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (CALCITE-3197) Convert data of Timestamp/Time/Date as original form when enumerating from ArrayTable

[jin xing (JIRA)]

jin xing created CALCITE-3197:
-

 Summary: Convert data of Timestamp/Time/Date as original form when 
enumerating from ArrayTable
 Key: CALCITE-3197
 URL: https://issues.apache.org/jira/browse/CALCITE-3197
 Project: Calcite
  Issue Type: Bug
  Components: core
Reporter: jin xing


In current implementation ColumnLoader, data of 
{{Rep.JAVA_SQL_TIMESTAMP/Rep.JAVA_SQL_TIME/Rep.JAVA_SQL_DATE}} are converted as 
numeric during loading. 
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/adapter/clone/ColumnLoader.java#L234
But current code seems forgot to revert the data back to original form when 
enumerating.
As a result, below test is failing now
{code:java}
// MaterializationTest.java

@Test public void testTimestampType() {
  String sql = "select \"eventid\", \"ts\"\n"
+ "from \"events\"\n"
+ "where \"eventid\" > 5";
  checkMaterialize(sql, sql);
}{code}
For type of {{Rep.JAVA_SQL_TIMESTAMP/Rep.JAVA_SQL_TIME/Rep.JAVA_SQL_DATE}}, 
cursor acesses {{TimestampAccessor/TimeAccessor/DateAccessor}}, which expect 
column value as {{Timestamp/Time/Date}}.
It make sense to 'unwrap' the data as original form when enumerating from 
{{ArrayTable}}.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Updated] (CALCITE-3197) Convert data of Timestamp/Time/Date as original form when enumerating from ArrayTable

[jin xing (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing updated CALCITE-3197:
--
Description: 
In current implementation ColumnLoader, data of 
{{Rep.JAVA_SQL_TIMESTAMP/Rep.JAVA_SQL_TIME/Rep.JAVA_SQL_DATE}} are converted as 
numeric during loading. 
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/adapter/clone/ColumnLoader.java#L234
But current code seems forgot to revert the data back to original form when 
enumerating.
As a result, below test is failing now
{code:java}
// MaterializationTest.java

@Test public void testTimestampType() {
  String sql = "select \"eventid\", \"ts\"\n"
+ "from \"events\"\n"
+ "where \"eventid\" > 5";
  checkMaterialize(sql, sql);
}{code}
For type of {{Rep.JAVA_SQL_TIMESTAMP/Rep.JAVA_SQL_TIME/Rep.JAVA_SQL_DATE}}, 
cursor acesses by {{TimestampAccessor/TimeAccessor/DateAccessor}}, which expect 
column value as {{Timestamp/Time/Date}}.
It make sense to 'unwrap' the data as original form when enumerating from 
{{ArrayTable}}.

  was:
In current implementation ColumnLoader, data of 
{{Rep.JAVA_SQL_TIMESTAMP/Rep.JAVA_SQL_TIME/Rep.JAVA_SQL_DATE}} are converted as 
numeric during loading. 
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/adapter/clone/ColumnLoader.java#L234
But current code seems forgot to revert the data back to original form when 
enumerating.
As a result, below test is failing now
{code:java}
// MaterializationTest.java

@Test public void testTimestampType() {
  String sql = "select \"eventid\", \"ts\"\n"
+ "from \"events\"\n"
+ "where \"eventid\" > 5";
  checkMaterialize(sql, sql);
}{code}
For type of {{Rep.JAVA_SQL_TIMESTAMP/Rep.JAVA_SQL_TIME/Rep.JAVA_SQL_DATE}}, 
cursor acesses {{TimestampAccessor/TimeAccessor/DateAccessor}}, which expect 
column value as {{Timestamp/Time/Date}}.
It make sense to 'unwrap' the data as original form when enumerating from 
{{ArrayTable}}.


> Convert data of Timestamp/Time/Date as original form when enumerating from 
> ArrayTable
> -
>
> Key: CALCITE-3197
> URL: https://issues.apache.org/jira/browse/CALCITE-3197
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Reporter: jin xing
>Priority: Major
>
> In current implementation ColumnLoader, data of 
> {{Rep.JAVA_SQL_TIMESTAMP/Rep.JAVA_SQL_TIME/Rep.JAVA_SQL_DATE}} are converted 
> as numeric during loading. 
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/adapter/clone/ColumnLoader.java#L234
> But current code seems forgot to revert the data back to original form when 
> enumerating.
> As a result, below test is failing now
> {code:java}
> // MaterializationTest.java
> @Test public void testTimestampType() {
>   String sql = "select \"eventid\", \"ts\"\n"
> + "from \"events\"\n"
> + "where \"eventid\" > 5";
>   checkMaterialize(sql, sql);
> }{code}
> For type of {{Rep.JAVA_SQL_TIMESTAMP/Rep.JAVA_SQL_TIME/Rep.JAVA_SQL_DATE}}, 
> cursor acesses by {{TimestampAccessor/TimeAccessor/DateAccessor}}, which 
> expect column value as {{Timestamp/Time/Date}}.
> It make sense to 'unwrap' the data as original form when enumerating from 
> {{ArrayTable}}.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Commented] (CALCITE-3197) Convert data of Timestamp/Time/Date as original form when enumerating from ArrayTable

[jin xing (JIRA)]

[ 
https://issues.apache.org/jira/browse/CALCITE-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16885816#comment-16885816
 ] 

jin xing commented on CALCITE-3197:
---

[~julianhyde]
THX a lot for comment. 
“The idea that a representation knows what it really should be seems very 
wrong”  -- This really helps me to get clear about the design.

I will refine my change by your comment.

> Convert data of Timestamp/Time/Date as original form when enumerating from 
> ArrayTable
> -
>
> Key: CALCITE-3197
> URL: https://issues.apache.org/jira/browse/CALCITE-3197
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Reporter: jin xing
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> In current implementation ColumnLoader, data of 
> {{Rep.JAVA_SQL_TIMESTAMP/Rep.JAVA_SQL_TIME/Rep.JAVA_SQL_DATE}} are converted 
> as numeric during loading. 
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/adapter/clone/ColumnLoader.java#L234
> But current code seems forgot to revert the data back to original form when 
> enumerating.
> As a result, below test is failing now
> {code:java}
> // MaterializationTest.java
> @Test public void testTimestampType() {
>   String sql = "select \"eventid\", \"ts\"\n"
> + "from \"events\"\n"
> + "where \"eventid\" > 5";
>   checkMaterialize(sql, sql);
> }{code}
> For type of {{Rep.JAVA_SQL_TIMESTAMP/Rep.JAVA_SQL_TIME/Rep.JAVA_SQL_DATE}}, 
> cursor acesses by {{TimestampAccessor/TimeAccessor/DateAccessor}}, which 
> expect column value as {{Timestamp/Time/Date}}.
> It make sense to 'unwrap' the data as original form when enumerating from 
> {{ArrayTable}}.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Commented] (CALCITE-3197) Convert data of Timestamp/Time/Date as original form when enumerating from ArrayTable

[jin xing (JIRA)]

[ 
https://issues.apache.org/jira/browse/CALCITE-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1610#comment-1610
 ] 

jin xing commented on CALCITE-3197:
---

[~julianhyde]
As you shepherded -- it's bad idea to let representation know what it "really 
should be", In current change I set the {{unwrapper}} into 
{{ArrayTable.Column}}  and unwrap when enumerate records. But checking the 
patch, I think it's still bigger than your expectation.

Please check the patch again when you have time. If the fix is far from your 
idea, it's great if you can point better direction

> Convert data of Timestamp/Time/Date as original form when enumerating from 
> ArrayTable
> -
>
> Key: CALCITE-3197
> URL: https://issues.apache.org/jira/browse/CALCITE-3197
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Reporter: jin xing
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> In current implementation ColumnLoader, data of 
> {{Rep.JAVA_SQL_TIMESTAMP/Rep.JAVA_SQL_TIME/Rep.JAVA_SQL_DATE}} are converted 
> as numeric during loading. 
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/adapter/clone/ColumnLoader.java#L234
> But current code seems forgot to revert the data back to original form when 
> enumerating.
> As a result, below test is failing now
> {code:java}
> // MaterializationTest.java
> @Test public void testTimestampType() {
>   String sql = "select \"eventid\", \"ts\"\n"
> + "from \"events\"\n"
> + "where \"eventid\" > 5";
>   checkMaterialize(sql, sql);
> }{code}
> For type of {{Rep.JAVA_SQL_TIMESTAMP/Rep.JAVA_SQL_TIME/Rep.JAVA_SQL_DATE}}, 
> cursor acesses by {{TimestampAccessor/TimeAccessor/DateAccessor}}, which 
> expect column value as {{Timestamp/Time/Date}}.
> It make sense to 'unwrap' the data as original form when enumerating from 
> {{ArrayTable}}.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Commented] (CALCITE-3167) Remove overriding equals and hashCode methods in EnumerableTableScan

[jin xing (JIRA)]

[ 
https://issues.apache.org/jira/browse/CALCITE-3167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1611#comment-1611
 ] 

jin xing commented on CALCITE-3167:
---

[~amargoor]
Gentle ping ~ ideas on this ?

> Remove overriding equals and hashCode methods in EnumerableTableScan
> 
>
> Key: CALCITE-3167
> URL: https://issues.apache.org/jira/browse/CALCITE-3167
> Project: Calcite
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 1.19.0
>Reporter: jin xing
>Priority: Minor
>  Labels: pull-request-available
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> In current code of {{EnumerableTableScan}}, methods of equals and hashCode 
> are overridden for matching of {{EnumerableTableScan}}s.
> While after optimizing with the same HEP planner, {{EnumerableTableScan}}s 
> from two plans but with the same digest will the share the same Java object. 
> See 
> [RelOptMaterializations|https://github.com/apache/calcite/blob/adf4cc4dc5cdb9f5e49c85d10f46a2fdcd831ccf/core/src/main/java/org/apache/calcite/plan/RelOptMaterializations.java#L192].
> I think it's ok to remove the redundant overriding methods in 
> {{EnumerableTableScan}}.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (CALCITE-3203) When matching materializations, match Project with child of Aggregate

[jin xing (JIRA)]

jin xing created CALCITE-3203:
-

 Summary: When matching materializations, match Project with child 
of Aggregate
 Key: CALCITE-3203
 URL: https://issues.apache.org/jira/browse/CALCITE-3203
 Project: Calcite
  Issue Type: Improvement
  Components: core
Reporter: jin xing






--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Updated] (CALCITE-3203) When matching materializations, match Project with child of Aggregate

[jin xing (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing updated CALCITE-3203:
--
Description: 
In current code, {{SubstitutionVisitor}} & 
{{MaterializedViewSubstitutionVisitor}} fail to support below matching:

{code:java}
   query:   Project(projects: [$0, *(2, $1)])
  Aggregate(groupSet: {0}, groupSets: [{0}], calls: [SUM($1)])
 Scan(table: [hr, emps])
   target:  Project(projects: [$0, *(2, $1), *(2, $2)])
  Aggregate(groupSet: {0}, groupSets: [{0}], calls: [SUM($1), 
COUNT()])
 Scan(table: [hr, emps])
{code}

And below test fails

{code:java}
MaterializationTest.java
  @Test public void testAggregate() {
checkMaterialize(
  "select \"deptno\", count(1), 2 * sum(\"empid\") from "
+ "(select * from \"emps\" union all select * from \"emps\")"
+ "group by \"deptno\"",
  "select \"deptno\", 2 * sum(\"empid\") from "
+ "(select * from \"emps\" union all select * from \"emps\")"
+ "group by \"deptno\"");
  }
{code}

The reason is that {{Project&Aggregate}} are not taken into consideration at 
the same time in current matching rules.
It might make sense to create a rule of 
{{ProjectOnAggregateToProjectOnAggregateUnifyRule}} to handle such case.


> When matching materializations, match Project with child of Aggregate
> -
>
> Key: CALCITE-3203
> URL: https://issues.apache.org/jira/browse/CALCITE-3203
> Project: Calcite
>  Issue Type: Improvement
>  Components: core
>Reporter: jin xing
>Priority: Major
>
> In current code, {{SubstitutionVisitor}} & 
> {{MaterializedViewSubstitutionVisitor}} fail to support below matching:
> {code:java}
>query:   Project(projects: [$0, *(2, $1)])
>   Aggregate(groupSet: {0}, groupSets: [{0}], calls: [SUM($1)])
>  Scan(table: [hr, emps])
>target:  Project(projects: [$0, *(2, $1), *(2, $2)])
>   Aggregate(groupSet: {0}, groupSets: [{0}], calls: [SUM($1), 
> COUNT()])
>  Scan(table: [hr, emps])
> {code}
> And below test fails
> {code:java}
> MaterializationTest.java
>   @Test public void testAggregate() {
> checkMaterialize(
>   "select \"deptno\", count(1), 2 * sum(\"empid\") from "
> + "(select * from \"emps\" union all select * from \"emps\")"
> + "group by \"deptno\"",
>   "select \"deptno\", 2 * sum(\"empid\") from "
> + "(select * from \"emps\" union all select * from \"emps\")"
> + "group by \"deptno\"");
>   }
> {code}
> The reason is that {{Project&Aggregate}} are not taken into consideration at 
> the same time in current matching rules.
> It might make sense to create a rule of 
> {{ProjectOnAggregateToProjectOnAggregateUnifyRule}} to handle such case.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Updated] (CALCITE-3203) When matching materializations, match Project with child of Aggregate

[jin xing (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing updated CALCITE-3203:
--
Description: 
In current code, {{SubstitutionVisitor}} & 
{{MaterializedViewSubstitutionVisitor}} fail to support below matching:

{code:java}
   query:   Project(projects: [$0, *(2, $1)])
  Aggregate(groupSet: {0}, groupSets: [{0}], calls: [SUM($1)])
 Scan(table: [hr, emps])
   target:  Project(projects: [$0, *(2, $1), *(2, $2)])
  Aggregate(groupSet: {0}, groupSets: [{0}], calls: [SUM($1), 
COUNT()])
 Scan(table: [hr, emps])
{code}

And below test fails

{code:java}
// MaterializationTest.java
  @Test public void testAggregate() {
checkMaterialize(
  "select \"deptno\", count(1), 2 * sum(\"empid\") from "
+ "(select * from \"emps\" union all select * from \"emps\")"
+ "group by \"deptno\"",
  "select \"deptno\", 2 * sum(\"empid\") from "
+ "(select * from \"emps\" union all select * from \"emps\")"
+ "group by \"deptno\"");
  }
{code}

The reason is that {{Project&Aggregate}} are not taken into consideration at 
the same time in current matching rules.
It might make sense to create a rule of 
{{ProjectOnAggregateToProjectOnAggregateUnifyRule}} to handle such case.


  was:
In current code, {{SubstitutionVisitor}} & 
{{MaterializedViewSubstitutionVisitor}} fail to support below matching:

{code:java}
   query:   Project(projects: [$0, *(2, $1)])
  Aggregate(groupSet: {0}, groupSets: [{0}], calls: [SUM($1)])
 Scan(table: [hr, emps])
   target:  Project(projects: [$0, *(2, $1), *(2, $2)])
  Aggregate(groupSet: {0}, groupSets: [{0}], calls: [SUM($1), 
COUNT()])
 Scan(table: [hr, emps])
{code}

And below test fails

{code:java}
MaterializationTest.java
  @Test public void testAggregate() {
checkMaterialize(
  "select \"deptno\", count(1), 2 * sum(\"empid\") from "
+ "(select * from \"emps\" union all select * from \"emps\")"
+ "group by \"deptno\"",
  "select \"deptno\", 2 * sum(\"empid\") from "
+ "(select * from \"emps\" union all select * from \"emps\")"
+ "group by \"deptno\"");
  }
{code}

The reason is that {{Project&Aggregate}} are not taken into consideration at 
the same time in current matching rules.
It might make sense to create a rule of 
{{ProjectOnAggregateToProjectOnAggregateUnifyRule}} to handle such case.



> When matching materializations, match Project with child of Aggregate
> -
>
> Key: CALCITE-3203
> URL: https://issues.apache.org/jira/browse/CALCITE-3203
> Project: Calcite
>  Issue Type: Improvement
>  Components: core
>Reporter: jin xing
>Priority: Major
>
> In current code, {{SubstitutionVisitor}} & 
> {{MaterializedViewSubstitutionVisitor}} fail to support below matching:
> {code:java}
>query:   Project(projects: [$0, *(2, $1)])
>   Aggregate(groupSet: {0}, groupSets: [{0}], calls: [SUM($1)])
>  Scan(table: [hr, emps])
>target:  Project(projects: [$0, *(2, $1), *(2, $2)])
>   Aggregate(groupSet: {0}, groupSets: [{0}], calls: [SUM($1), 
> COUNT()])
>  Scan(table: [hr, emps])
> {code}
> And below test fails
> {code:java}
> // MaterializationTest.java
>   @Test public void testAggregate() {
> checkMaterialize(
>   "select \"deptno\", count(1), 2 * sum(\"empid\") from "
> + "(select * from \"emps\" union all select * from \"emps\")"
> + "group by \"deptno\"",
>   "select \"deptno\", 2 * sum(\"empid\") from "
> + "(select * from \"emps\" union all select * from \"emps\")"
> + "group by \"deptno\"");
>   }
> {code}
> The reason is that {{Project&Aggregate}} are not taken into consideration at 
> the same time in current matching rules.
> It might make sense to create a rule of 
> {{ProjectOnAggregateToProjectOnAggregateUnifyRule}} to handle such case.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Commented] (CALCITE-3197) Convert data of Timestamp/Time/Date as original form when enumerating from ArrayTable

[jin xing (JIRA)]

[ 
https://issues.apache.org/jira/browse/CALCITE-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16890125#comment-16890125
 ] 

jin xing commented on CALCITE-3197:
---

THX for your shepherd ~ [~julianhyde]

Now I can understand the point :)

> Convert data of Timestamp/Time/Date as original form when enumerating from 
> ArrayTable
> -
>
> Key: CALCITE-3197
> URL: https://issues.apache.org/jira/browse/CALCITE-3197
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Reporter: jin xing
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> In current implementation ColumnLoader, data of 
> {{Rep.JAVA_SQL_TIMESTAMP/Rep.JAVA_SQL_TIME/Rep.JAVA_SQL_DATE}} are converted 
> as numeric during loading. 
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/adapter/clone/ColumnLoader.java#L234
> But current code seems forgot to revert the data back to original form when 
> enumerating.
> As a result, below test is failing now
> {code:java}
> // MaterializationTest.java
> @Test public void testTimestampType() {
>   String sql = "select \"eventid\", \"ts\"\n"
> + "from \"events\"\n"
> + "where \"eventid\" > 5";
>   checkMaterialize(sql, sql);
> }{code}
> For type of {{Rep.JAVA_SQL_TIMESTAMP/Rep.JAVA_SQL_TIME/Rep.JAVA_SQL_DATE}}, 
> cursor acesses by {{TimestampAccessor/TimeAccessor/DateAccessor}}, which 
> expect column value as {{Timestamp/Time/Date}}.
> It make sense to 'unwrap' the data as original form when enumerating from 
> {{ArrayTable}}.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Comment Edited] (CALCITE-3197) Convert data of Timestamp/Time/Date as original form when enumerating from ArrayTable

[jin xing (JIRA)]

[ 
https://issues.apache.org/jira/browse/CALCITE-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16890125#comment-16890125
 ] 

jin xing edited comment on CALCITE-3197 at 7/22/19 12:42 PM:
-

THX for your shepherd ~ [~julianhyde]

I got the point now.


was (Author: jinxing6...@126.com):
THX for your shepherd ~ [~julianhyde]

Now I can understand the point :)

> Convert data of Timestamp/Time/Date as original form when enumerating from 
> ArrayTable
> -
>
> Key: CALCITE-3197
> URL: https://issues.apache.org/jira/browse/CALCITE-3197
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Reporter: jin xing
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> In current implementation ColumnLoader, data of 
> {{Rep.JAVA_SQL_TIMESTAMP/Rep.JAVA_SQL_TIME/Rep.JAVA_SQL_DATE}} are converted 
> as numeric during loading. 
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/adapter/clone/ColumnLoader.java#L234
> But current code seems forgot to revert the data back to original form when 
> enumerating.
> As a result, below test is failing now
> {code:java}
> // MaterializationTest.java
> @Test public void testTimestampType() {
>   String sql = "select \"eventid\", \"ts\"\n"
> + "from \"events\"\n"
> + "where \"eventid\" > 5";
>   checkMaterialize(sql, sql);
> }{code}
> For type of {{Rep.JAVA_SQL_TIMESTAMP/Rep.JAVA_SQL_TIME/Rep.JAVA_SQL_DATE}}, 
> cursor acesses by {{TimestampAccessor/TimeAccessor/DateAccessor}}, which 
> expect column value as {{Timestamp/Time/Date}}.
> It make sense to 'unwrap' the data as original form when enumerating from 
> {{ArrayTable}}.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Closed] (CALCITE-3197) Convert data of Timestamp/Time/Date as original form when enumerating from ArrayTable

[jin xing (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing closed CALCITE-3197.
-
Resolution: Invalid

> Convert data of Timestamp/Time/Date as original form when enumerating from 
> ArrayTable
> -
>
> Key: CALCITE-3197
> URL: https://issues.apache.org/jira/browse/CALCITE-3197
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Reporter: jin xing
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> In current implementation ColumnLoader, data of 
> {{Rep.JAVA_SQL_TIMESTAMP/Rep.JAVA_SQL_TIME/Rep.JAVA_SQL_DATE}} are converted 
> as numeric during loading. 
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/adapter/clone/ColumnLoader.java#L234
> But current code seems forgot to revert the data back to original form when 
> enumerating.
> As a result, below test is failing now
> {code:java}
> // MaterializationTest.java
> @Test public void testTimestampType() {
>   String sql = "select \"eventid\", \"ts\"\n"
> + "from \"events\"\n"
> + "where \"eventid\" > 5";
>   checkMaterialize(sql, sql);
> }{code}
> For type of {{Rep.JAVA_SQL_TIMESTAMP/Rep.JAVA_SQL_TIME/Rep.JAVA_SQL_DATE}}, 
> cursor acesses by {{TimestampAccessor/TimeAccessor/DateAccessor}}, which 
> expect column value as {{Timestamp/Time/Date}}.
> It make sense to 'unwrap' the data as original form when enumerating from 
> {{ArrayTable}}.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (CALCITE-3206) Better error message when param type incompatibility

[jin xing (JIRA)]

jin xing created CALCITE-3206:
-

 Summary: Better error message when param type incompatibility
 Key: CALCITE-3206
 URL: https://issues.apache.org/jira/browse/CALCITE-3206
 Project: Calcite
  Issue Type: Improvement
  Components: core
Reporter: jin xing






--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Updated] (CALCITE-3206) Better error message when param type incompatibility

[jin xing (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing updated CALCITE-3206:
--
Description: 
In current code, when we query like below

{code:java}
  @Test public void test() {
CalciteAssert.model(HR_FKUK_MODEL)
  .query("select * from \"events\" where \"eventid\"='123'")
  .returns(CalciteAssert.outputResult());
  }
{code}
Where {{eventid}} is of INTEGER type, Calcite will throw exception as below

{code:java}
Caused by: org.codehaus.commons.compiler.CompileException: Line 13, Column 100: 
Cannot compare types "int" and "java.lang.String"
at 
org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:12211)
at 
org.codehaus.janino.UnitCompiler.compileBoolean2(UnitCompiler.java:4239)
at org.codehaus.janino.UnitCompiler.access$6600(UnitCompiler.java:215)
at 
org.codehaus.janino.UnitCompiler$14.visitBinaryOperation(UnitCompiler.java:3957)
at 
org.codehaus.janino.UnitCompiler$14.visitBinaryOperation(UnitCompiler.java:3935)
at org.codehaus.janino.Java$BinaryOperation.accept(Java.java:4864)
at 
org.codehaus.janino.UnitCompiler.compileBoolean(UnitCompiler.java:3935)
at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:2475)
{code}

It's kind of hard to understand for user to refine the sql



> Better error message when param type incompatibility
> 
>
> Key: CALCITE-3206
> URL: https://issues.apache.org/jira/browse/CALCITE-3206
> Project: Calcite
>  Issue Type: Improvement
>  Components: core
>Reporter: jin xing
>Priority: Major
>
> In current code, when we query like below
> {code:java}
>   @Test public void test() {
> CalciteAssert.model(HR_FKUK_MODEL)
>   .query("select * from \"events\" where \"eventid\"='123'")
>   .returns(CalciteAssert.outputResult());
>   }
> {code}
> Where {{eventid}} is of INTEGER type, Calcite will throw exception as below
> {code:java}
> Caused by: org.codehaus.commons.compiler.CompileException: Line 13, Column 
> 100: Cannot compare types "int" and "java.lang.String"
>   at 
> org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:12211)
>   at 
> org.codehaus.janino.UnitCompiler.compileBoolean2(UnitCompiler.java:4239)
>   at org.codehaus.janino.UnitCompiler.access$6600(UnitCompiler.java:215)
>   at 
> org.codehaus.janino.UnitCompiler$14.visitBinaryOperation(UnitCompiler.java:3957)
>   at 
> org.codehaus.janino.UnitCompiler$14.visitBinaryOperation(UnitCompiler.java:3935)
>   at org.codehaus.janino.Java$BinaryOperation.accept(Java.java:4864)
>   at 
> org.codehaus.janino.UnitCompiler.compileBoolean(UnitCompiler.java:3935)
>   at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:2475)
> {code}
> It's kind of hard to understand for user to refine the sql



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Commented] (CALCITE-3207) Bug of 'JoinRelNode Convert SqlStatement'

[jin xing (JIRA)]

[ 
https://issues.apache.org/jira/browse/CALCITE-3207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16890195#comment-16890195
 ] 

jin xing commented on CALCITE-3207:
---

Hi [~wojustme]
I create a test like below

 
{code:java}
// RelToSqlConverterTest.java
@Test public void testSelectQueryWithGroupByHaving2() {
  String query = " select \"product\".\"product_id\"\n"
  + "from \"product\"\n"
  + "inner join \"sales_fact_1997\"\n"
  + "on \"product\".\"product_id\" = \"sales_fact_1997\".\"product_id\""
  + "and \"product\".\"product_name\" like 'apple'\n";
  sql(query);
}
{code}
 

The output is like below:

 
{code:java}
SELECT "t"."product_id"
FROM (SELECT "product_class_id", "product_id", "brand_name", "product_name", 
"SKU", "SRP", "gross_weight", "net_weight", "recyclable_package", "low_fat", 
"units_per_case", "cases_per_pallet", "shelf_width", "shelf_height", 
"shelf_depth", "product_name" LIKE 'apple' AS "$f15"
FROM "foodmart"."product") AS "t"
INNER JOIN "foodmart"."sales_fact_1997" ON "t"."product_id" = 
"sales_fact_1997"."product_id" AND "t"."$f15"

{code}
 

Seems different from your description. Could you please create a test case in 
RelToSqlConverterTest.java to illustrate your issue ?

 

> Bug of 'JoinRelNode Convert SqlStatement'
> -
>
> Key: CALCITE-3207
> URL: https://issues.apache.org/jira/browse/CALCITE-3207
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Reporter: Xurenhe
>Priority: Minor
>
> Hi all
> I have a problem when using calcite, which is 'relnode convert sql statement'
> example:
> ```
> Input relNode: 
> LogicalProject(name=[$1], score=[$4])
>  LogicalJoin(condition=[AND(=($0, $3), LIKE($0, 'lucy'))], joinType=[left])
>  LogicalTableScan(table=[[db0, user_table]])
>  LogicalTableScan(table=[[db0, score_table]])
> Expect output sql statement: 
> select name, score
> from user_table
> left join on score_table
> and name like 'lucy'
> ```
> this relNode cannot convert sql statement.
> I try to debug it, 
> `org.apache.calcite.rel.rel2sql.SqlImplementor#convertConditionToSqlNode` may 
> forgot to handle this case, which sqlKind is SqlKind.LIKE.
> If it is a bug, I am willing to fix it.
> Thanks.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Comment Edited] (CALCITE-3207) Bug of 'JoinRelNode Convert SqlStatement'

[jin xing (JIRA)]

[ 
https://issues.apache.org/jira/browse/CALCITE-3207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16890195#comment-16890195
 ] 

jin xing edited comment on CALCITE-3207 at 7/22/19 2:15 PM:


Hi [~wojustme]
 I create a test like below
{code:java}
// RelToSqlConverterTest.java
@Test public void testSelectQueryWithGroupByHaving2() {
  String query = " select \"product\".\"product_id\"\n"
  + "from \"product\"\n"
  + "inner join \"sales_fact_1997\"\n"
  + "on \"product\".\"product_id\" = \"sales_fact_1997\".\"product_id\""
  + "and \"product\".\"product_name\" like 'apple'\n";
  sql(query);
}
{code}
The output is like below: 
{code:java}
SELECT "t"."product_id"
FROM (SELECT "product_class_id", "product_id", "brand_name", "product_name", 
"SKU", "SRP", "gross_weight", "net_weight", "recyclable_package", "low_fat", 
"units_per_case", "cases_per_pallet", "shelf_width", "shelf_height", 
"shelf_depth", "product_name" LIKE 'apple' AS "$f15"
FROM "foodmart"."product") AS "t"
INNER JOIN "foodmart"."sales_fact_1997" ON "t"."product_id" = 
"sales_fact_1997"."product_id" AND "t"."$f15"

{code}
Seems different from your description.

Could you please create a test case in RelToSqlConverterTest.java to illustrate 
your issue ?

 


was (Author: jinxing6...@126.com):
Hi [~wojustme]
 I create a test like below
{code:java}
// RelToSqlConverterTest.java
@Test public void testSelectQueryWithGroupByHaving2() {
  String query = " select \"product\".\"product_id\"\n"
  + "from \"product\"\n"
  + "inner join \"sales_fact_1997\"\n"
  + "on \"product\".\"product_id\" = \"sales_fact_1997\".\"product_id\""
  + "and \"product\".\"product_name\" like 'apple'\n";
  sql(query);
}
{code}
 

The output is like below: 
{code:java}
SELECT "t"."product_id"
FROM (SELECT "product_class_id", "product_id", "brand_name", "product_name", 
"SKU", "SRP", "gross_weight", "net_weight", "recyclable_package", "low_fat", 
"units_per_case", "cases_per_pallet", "shelf_width", "shelf_height", 
"shelf_depth", "product_name" LIKE 'apple' AS "$f15"
FROM "foodmart"."product") AS "t"
INNER JOIN "foodmart"."sales_fact_1997" ON "t"."product_id" = 
"sales_fact_1997"."product_id" AND "t"."$f15"

{code}
 

Seems different from your description. Could you please create a test case in 
RelToSqlConverterTest.java to illustrate your issue ?

 

> Bug of 'JoinRelNode Convert SqlStatement'
> -
>
> Key: CALCITE-3207
> URL: https://issues.apache.org/jira/browse/CALCITE-3207
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Reporter: Xurenhe
>Priority: Minor
>
> Hi all
> I have a problem when using calcite, which is 'relnode convert sql statement'
> example:
> ```
> Input relNode: 
> LogicalProject(name=[$1], score=[$4])
>  LogicalJoin(condition=[AND(=($0, $3), LIKE($0, 'lucy'))], joinType=[left])
>  LogicalTableScan(table=[[db0, user_table]])
>  LogicalTableScan(table=[[db0, score_table]])
> Expect output sql statement: 
> select name, score
> from user_table
> left join on score_table
> and name like 'lucy'
> ```
> this relNode cannot convert sql statement.
> I try to debug it, 
> `org.apache.calcite.rel.rel2sql.SqlImplementor#convertConditionToSqlNode` may 
> forgot to handle this case, which sqlKind is SqlKind.LIKE.
> If it is a bug, I am willing to fix it.
> Thanks.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Comment Edited] (CALCITE-3207) Bug of 'JoinRelNode Convert SqlStatement'

[jin xing (JIRA)]

[ 
https://issues.apache.org/jira/browse/CALCITE-3207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16890195#comment-16890195
 ] 

jin xing edited comment on CALCITE-3207 at 7/22/19 2:15 PM:


Hi [~wojustme]
 I create a test like below
{code:java}
// RelToSqlConverterTest.java
@Test public void testSelectQueryWithGroupByHaving2() {
  String query = " select \"product\".\"product_id\"\n"
  + "from \"product\"\n"
  + "inner join \"sales_fact_1997\"\n"
  + "on \"product\".\"product_id\" = \"sales_fact_1997\".\"product_id\""
  + "and \"product\".\"product_name\" like 'apple'\n";
  sql(query);
}
{code}
 

The output is like below: 
{code:java}
SELECT "t"."product_id"
FROM (SELECT "product_class_id", "product_id", "brand_name", "product_name", 
"SKU", "SRP", "gross_weight", "net_weight", "recyclable_package", "low_fat", 
"units_per_case", "cases_per_pallet", "shelf_width", "shelf_height", 
"shelf_depth", "product_name" LIKE 'apple' AS "$f15"
FROM "foodmart"."product") AS "t"
INNER JOIN "foodmart"."sales_fact_1997" ON "t"."product_id" = 
"sales_fact_1997"."product_id" AND "t"."$f15"

{code}
 

Seems different from your description. Could you please create a test case in 
RelToSqlConverterTest.java to illustrate your issue ?

 


was (Author: jinxing6...@126.com):
Hi [~wojustme]
I create a test like below

 
{code:java}
// RelToSqlConverterTest.java
@Test public void testSelectQueryWithGroupByHaving2() {
  String query = " select \"product\".\"product_id\"\n"
  + "from \"product\"\n"
  + "inner join \"sales_fact_1997\"\n"
  + "on \"product\".\"product_id\" = \"sales_fact_1997\".\"product_id\""
  + "and \"product\".\"product_name\" like 'apple'\n";
  sql(query);
}
{code}
 

The output is like below:

 
{code:java}
SELECT "t"."product_id"
FROM (SELECT "product_class_id", "product_id", "brand_name", "product_name", 
"SKU", "SRP", "gross_weight", "net_weight", "recyclable_package", "low_fat", 
"units_per_case", "cases_per_pallet", "shelf_width", "shelf_height", 
"shelf_depth", "product_name" LIKE 'apple' AS "$f15"
FROM "foodmart"."product") AS "t"
INNER JOIN "foodmart"."sales_fact_1997" ON "t"."product_id" = 
"sales_fact_1997"."product_id" AND "t"."$f15"

{code}
 

Seems different from your description. Could you please create a test case in 
RelToSqlConverterTest.java to illustrate your issue ?

 

> Bug of 'JoinRelNode Convert SqlStatement'
> -
>
> Key: CALCITE-3207
> URL: https://issues.apache.org/jira/browse/CALCITE-3207
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Reporter: Xurenhe
>Priority: Minor
>
> Hi all
> I have a problem when using calcite, which is 'relnode convert sql statement'
> example:
> ```
> Input relNode: 
> LogicalProject(name=[$1], score=[$4])
>  LogicalJoin(condition=[AND(=($0, $3), LIKE($0, 'lucy'))], joinType=[left])
>  LogicalTableScan(table=[[db0, user_table]])
>  LogicalTableScan(table=[[db0, score_table]])
> Expect output sql statement: 
> select name, score
> from user_table
> left join on score_table
> and name like 'lucy'
> ```
> this relNode cannot convert sql statement.
> I try to debug it, 
> `org.apache.calcite.rel.rel2sql.SqlImplementor#convertConditionToSqlNode` may 
> forgot to handle this case, which sqlKind is SqlKind.LIKE.
> If it is a bug, I am willing to fix it.
> Thanks.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Commented] (CALCITE-3206) Better error message when param type incompatibility

[jin xing (JIRA)]

[ 
https://issues.apache.org/jira/browse/CALCITE-3206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16890684#comment-16890684
 ] 

jin xing commented on CALCITE-3206:
---

THX [~danny0405]

I will close this issue

> Better error message when param type incompatibility
> 
>
> Key: CALCITE-3206
> URL: https://issues.apache.org/jira/browse/CALCITE-3206
> Project: Calcite
>  Issue Type: Improvement
>  Components: core
>Reporter: jin xing
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> In current code, when we query like below
> {code:java}
>   @Test public void test() {
> CalciteAssert.model(HR_FKUK_MODEL)
>   .query("select * from \"events\" where \"eventid\"='123'")
>   .returns(CalciteAssert.outputResult());
>   }
> {code}
> Where {{eventid}} is of INTEGER type, Calcite will throw exception as below
> {code:java}
> Caused by: org.codehaus.commons.compiler.CompileException: Line 13, Column 
> 100: Cannot compare types "int" and "java.lang.String"
>   at 
> org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:12211)
>   at 
> org.codehaus.janino.UnitCompiler.compileBoolean2(UnitCompiler.java:4239)
>   at org.codehaus.janino.UnitCompiler.access$6600(UnitCompiler.java:215)
>   at 
> org.codehaus.janino.UnitCompiler$14.visitBinaryOperation(UnitCompiler.java:3957)
>   at 
> org.codehaus.janino.UnitCompiler$14.visitBinaryOperation(UnitCompiler.java:3935)
>   at org.codehaus.janino.Java$BinaryOperation.accept(Java.java:4864)
>   at 
> org.codehaus.janino.UnitCompiler.compileBoolean(UnitCompiler.java:3935)
>   at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:2475)
> {code}
> It's kind of hard to understand for user to refine the sql



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Closed] (CALCITE-3206) Better error message when param type incompatibility

[jin xing (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing closed CALCITE-3206.
-
Resolution: Duplicate

> Better error message when param type incompatibility
> 
>
> Key: CALCITE-3206
> URL: https://issues.apache.org/jira/browse/CALCITE-3206
> Project: Calcite
>  Issue Type: Improvement
>  Components: core
>Reporter: jin xing
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> In current code, when we query like below
> {code:java}
>   @Test public void test() {
> CalciteAssert.model(HR_FKUK_MODEL)
>   .query("select * from \"events\" where \"eventid\"='123'")
>   .returns(CalciteAssert.outputResult());
>   }
> {code}
> Where {{eventid}} is of INTEGER type, Calcite will throw exception as below
> {code:java}
> Caused by: org.codehaus.commons.compiler.CompileException: Line 13, Column 
> 100: Cannot compare types "int" and "java.lang.String"
>   at 
> org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:12211)
>   at 
> org.codehaus.janino.UnitCompiler.compileBoolean2(UnitCompiler.java:4239)
>   at org.codehaus.janino.UnitCompiler.access$6600(UnitCompiler.java:215)
>   at 
> org.codehaus.janino.UnitCompiler$14.visitBinaryOperation(UnitCompiler.java:3957)
>   at 
> org.codehaus.janino.UnitCompiler$14.visitBinaryOperation(UnitCompiler.java:3935)
>   at org.codehaus.janino.Java$BinaryOperation.accept(Java.java:4864)
>   at 
> org.codehaus.janino.UnitCompiler.compileBoolean(UnitCompiler.java:3935)
>   at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:2475)
> {code}
> It's kind of hard to understand for user to refine the sql



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (CALCITE-3209) Store inputs of MutableMultiRel by ArrayList rather than ImmutableList

[jin xing (JIRA)]

jin xing created CALCITE-3209:
-

 Summary: Store inputs of MutableMultiRel by ArrayList rather than 
ImmutableList
 Key: CALCITE-3209
 URL: https://issues.apache.org/jira/browse/CALCITE-3209
 Project: Calcite
  Issue Type: Bug
  Components: core
Reporter: jin xing


In current implementation of {{MutableMultiRel}}, {{inputs}} is implemented by 
a {{ImmutableList}}, thus impossible to be changed by
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/rel/mutable/MutableMultiRel.java#L42
{code:java}
 @Override public void setInput(int ordinalInParent, MutableRel input)
{code}
Seems a mistake.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Updated] (CALCITE-3209) Store inputs of MutableMultiRel by ArrayList rather than ImmutableList

[jin xing (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing updated CALCITE-3209:
--
Description: 
When digging code of materialization matching. I found the current 
implementation of {{MutableMultiRel::}}{{inputs}} is by an {{ImmutableList}}, 
thus impossible to be changed by
 
[https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/rel/mutable/MutableMultiRel.java#L42]
{code:java}
 @Override public void setInput(int ordinalInParent, MutableRel input)
{code}
Seems a mistake.

  was:
In current implementation of {{MutableMultiRel}}, {{inputs}} is implemented by 
a {{ImmutableList}}, thus impossible to be changed by
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/rel/mutable/MutableMultiRel.java#L42
{code:java}
 @Override public void setInput(int ordinalInParent, MutableRel input)
{code}
Seems a mistake.


> Store inputs of MutableMultiRel by ArrayList rather than ImmutableList
> --
>
> Key: CALCITE-3209
> URL: https://issues.apache.org/jira/browse/CALCITE-3209
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Reporter: jin xing
>Priority: Major
>
> When digging code of materialization matching. I found the current 
> implementation of {{MutableMultiRel::}}{{inputs}} is by an {{ImmutableList}}, 
> thus impossible to be changed by
>  
> [https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/rel/mutable/MutableMultiRel.java#L42]
> {code:java}
>  @Override public void setInput(int ordinalInParent, MutableRel input)
> {code}
> Seems a mistake.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Commented] (CALCITE-3209) Store inputs of MutableMultiRel by ArrayList rather than ImmutableList

[jin xing (JIRA)]

[ 
https://issues.apache.org/jira/browse/CALCITE-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16892356#comment-16892356
 ] 

jin xing commented on CALCITE-3209:
---

THX [~julianhyde]
I will update the commit-header & issue title soon.

> Store inputs of MutableMultiRel by ArrayList rather than ImmutableList
> --
>
> Key: CALCITE-3209
> URL: https://issues.apache.org/jira/browse/CALCITE-3209
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Reporter: jin xing
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> When digging code of materialization matching. I found the current 
> implementation of {{MutableMultiRel::}}{{inputs}} is by an {{ImmutableList}}, 
> thus impossible to be changed by
>  
> [https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/rel/mutable/MutableMultiRel.java#L42]
> {code:java}
>  @Override public void setInput(int ordinalInParent, MutableRel input)
> {code}
> Seems a mistake.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Updated] (CALCITE-3209) When calling MutableMultiRel:: setInput, exception thrown

[jin xing (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing updated CALCITE-3209:
--
Summary: When calling MutableMultiRel:: setInput, exception thrown  (was: 
Store inputs of MutableMultiRel by ArrayList rather than ImmutableList)

> When calling MutableMultiRel:: setInput, exception thrown
> -
>
> Key: CALCITE-3209
> URL: https://issues.apache.org/jira/browse/CALCITE-3209
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Reporter: jin xing
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> When digging code of materialization matching. I found the current 
> implementation of {{MutableMultiRel::}}{{inputs}} is by an {{ImmutableList}}, 
> thus impossible to be changed by
>  
> [https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/rel/mutable/MutableMultiRel.java#L42]
> {code:java}
>  @Override public void setInput(int ordinalInParent, MutableRel input)
> {code}
> Seems a mistake.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Updated] (CALCITE-3209) When calling MutableMultiRel::setInput, exception thrown

[jin xing (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing updated CALCITE-3209:
--
Summary: When calling MutableMultiRel::setInput, exception thrown  (was: 
When calling MutableMultiRel:: setInput, exception thrown)

> When calling MutableMultiRel::setInput, exception thrown
> 
>
> Key: CALCITE-3209
> URL: https://issues.apache.org/jira/browse/CALCITE-3209
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Reporter: jin xing
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> When digging code of materialization matching. I found the current 
> implementation of {{MutableMultiRel::}}{{inputs}} is by an {{ImmutableList}}, 
> thus impossible to be changed by
>  
> [https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/rel/mutable/MutableMultiRel.java#L42]
> {code:java}
>  @Override public void setInput(int ordinalInParent, MutableRel input)
> {code}
> Seems a mistake.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (CALCITE-3211) MutableRel returned from MutableRels::toMutables may lose reference to parent

[jin xing (JIRA)]

jin xing created CALCITE-3211:
-

 Summary: MutableRel returned from MutableRels::toMutables may lose 
reference to parent
 Key: CALCITE-3211
 URL: https://issues.apache.org/jira/browse/CALCITE-3211
 Project: Calcite
  Issue Type: Bug
Reporter: jin xing






--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Updated] (CALCITE-3211) MutableRel returned from MutableRels::toMutables may lose reference to parent

[jin xing (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing updated CALCITE-3211:
--
Description: 
Current implementation of {{MutableRels::toMutables}} is as below:

{code:java}
  private static List toMutables(List nodes) {
return Lists.transform(nodes, MutableRels::toMutable);
  }
{code}

Thus every time we {{get}} from the result list, a new {{MutableRel}} will be 
created:

{code:java}
  private static class TransformingRandomAccessList extends 
AbstractList
  implements RandomAccess, Serializable {
final List fromList;
final Function function;

TransformingRandomAccessList(List fromList, Function function) {
  this.fromList = checkNotNull(fromList);
  this.function = checkNotNull(function);
}

@Override
public T get(int index) {
  return function.apply(fromList.get(index));
}
..
{code}
As a result, the {{parent}} information will be lost.


> MutableRel returned from MutableRels::toMutables may lose reference to parent
> -
>
> Key: CALCITE-3211
> URL: https://issues.apache.org/jira/browse/CALCITE-3211
> Project: Calcite
>  Issue Type: Bug
>Reporter: jin xing
>Priority: Major
>
> Current implementation of {{MutableRels::toMutables}} is as below:
> {code:java}
>   private static List toMutables(List nodes) {
> return Lists.transform(nodes, MutableRels::toMutable);
>   }
> {code}
> Thus every time we {{get}} from the result list, a new {{MutableRel}} will be 
> created:
> {code:java}
>   private static class TransformingRandomAccessList extends 
> AbstractList
>   implements RandomAccess, Serializable {
> final List fromList;
> final Function function;
> TransformingRandomAccessList(List fromList, Function extends T> function) {
>   this.fromList = checkNotNull(fromList);
>   this.function = checkNotNull(function);
> }
> @Override
> public T get(int index) {
>   return function.apply(fromList.get(index));
> }
> ..
> {code}
> As a result, the {{parent}} information will be lost.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (CALCITE-3214) Add UnionToUnionRule for materialization matching

[jin xing (JIRA)]

jin xing created CALCITE-3214:
-

 Summary: Add UnionToUnionRule for materialization matching
 Key: CALCITE-3214
 URL: https://issues.apache.org/jira/browse/CALCITE-3214
 Project: Calcite
  Issue Type: Bug
  Components: core
Reporter: jin xing


Below materialization matching fails now

{code:java}
  @Test public void testDEV() {
String sql0 = "select * from \"emps\" where \"empid\" < 300";
String sql1 = "select * from \"emps\" where \"empid\" > 200";
checkMaterialize(sql0 + " union all " + sql1, sql1 + " union all " + sql0);
  }
{code}

This issue proposes to add a rule for union matching




--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Updated] (CALCITE-3211) MutableRel returned from MutableRels::toMutables may lose reference to parent

[jin xing (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing updated CALCITE-3211:
--
Description: 
Current implementation of {{MutableRels::toMutables}} is as below:

{code:java}
  private static List toMutables(List nodes) {
return Lists.transform(nodes, MutableRels::toMutable);
  }
{code}

Thus every time we {{get}} from the result list, a new {{MutableRel}} will be 
created:

{code:java}
  private static class TransformingRandomAccessList extends 
AbstractList
  implements RandomAccess, Serializable {
final List fromList;
final Function function;

TransformingRandomAccessList(List fromList, Function function) {
  this.fromList = checkNotNull(fromList);
  this.function = checkNotNull(function);
}

@Override
public T get(int index) {
  return function.apply(fromList.get(index));
}
..
{code}
As a result, 
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/plan/SubstitutionVisitor.java#L514
   {{SubstitutionVisitor}} will fail to check whether a node has been 
met/matched before


  was:
Current implementation of {{MutableRels::toMutables}} is as below:

{code:java}
  private static List toMutables(List nodes) {
return Lists.transform(nodes, MutableRels::toMutable);
  }
{code}

Thus every time we {{get}} from the result list, a new {{MutableRel}} will be 
created:

{code:java}
  private static class TransformingRandomAccessList extends 
AbstractList
  implements RandomAccess, Serializable {
final List fromList;
final Function function;

TransformingRandomAccessList(List fromList, Function function) {
  this.fromList = checkNotNull(fromList);
  this.function = checkNotNull(function);
}

@Override
public T get(int index) {
  return function.apply(fromList.get(index));
}
..
{code}
As a result, the {{parent}} information will be lost.



> MutableRel returned from MutableRels::toMutables may lose reference to parent
> -
>
> Key: CALCITE-3211
> URL: https://issues.apache.org/jira/browse/CALCITE-3211
> Project: Calcite
>  Issue Type: Bug
>Reporter: jin xing
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Current implementation of {{MutableRels::toMutables}} is as below:
> {code:java}
>   private static List toMutables(List nodes) {
> return Lists.transform(nodes, MutableRels::toMutable);
>   }
> {code}
> Thus every time we {{get}} from the result list, a new {{MutableRel}} will be 
> created:
> {code:java}
>   private static class TransformingRandomAccessList extends 
> AbstractList
>   implements RandomAccess, Serializable {
> final List fromList;
> final Function function;
> TransformingRandomAccessList(List fromList, Function extends T> function) {
>   this.fromList = checkNotNull(fromList);
>   this.function = checkNotNull(function);
> }
> @Override
> public T get(int index) {
>   return function.apply(fromList.get(index));
> }
> ..
> {code}
> As a result, 
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/plan/SubstitutionVisitor.java#L514
>{{SubstitutionVisitor}} will fail to check whether a node has been 
> met/matched before



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Commented] (CALCITE-3211) MutableRel returned from MutableRels::toMutables may lose reference to parent

[jin xing (JIRA)]

[ 
https://issues.apache.org/jira/browse/CALCITE-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16893336#comment-16893336
 ] 

jin xing commented on CALCITE-3211:
---

[~julianhyde]
Thanks a lot for reply ~

I updated the description of the issue. I'm working on a rule for 
materialization matching of {{Union}} 
https://issues.apache.org/jira/browse/CALCITE-3214

When I run below test, {{SubstitutionVisitor}} suffers into a infinite loop

{code:java}
  @Test public void testDEV() {
String sql0 = "select * from \"emps\" where \"empid\" < 300";
String sql1 = "select * from \"emps\" where \"empid\" > 200";
checkMaterialize(sql0 + " union all " + sql1, sql1 + " union all " + sql0);
  }
{code}

I think the problem is from here:

{code:java}
Lists.transform(nodes, MutableRels::toMutable);
{code}

When we {{get}} from the result list, a new {{MutableRel}} is created.



> MutableRel returned from MutableRels::toMutables may lose reference to parent
> -
>
> Key: CALCITE-3211
> URL: https://issues.apache.org/jira/browse/CALCITE-3211
> Project: Calcite
>  Issue Type: Bug
>Reporter: jin xing
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Current implementation of {{MutableRels::toMutables}} is as below:
> {code:java}
>   private static List toMutables(List nodes) {
> return Lists.transform(nodes, MutableRels::toMutable);
>   }
> {code}
> Thus every time we {{get}} from the result list, a new {{MutableRel}} will be 
> created:
> {code:java}
>   private static class TransformingRandomAccessList extends 
> AbstractList
>   implements RandomAccess, Serializable {
> final List fromList;
> final Function function;
> TransformingRandomAccessList(List fromList, Function extends T> function) {
>   this.fromList = checkNotNull(fromList);
>   this.function = checkNotNull(function);
> }
> @Override
> public T get(int index) {
>   return function.apply(fromList.get(index));
> }
> ..
> {code}
> As a result, 
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/plan/SubstitutionVisitor.java#L514
>{{SubstitutionVisitor}} will fail to check whether a node has been 
> met/matched before



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Comment Edited] (CALCITE-3211) MutableRel returned from MutableRels::toMutables may lose reference to parent

[jin xing (JIRA)]

[ 
https://issues.apache.org/jira/browse/CALCITE-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16893336#comment-16893336
 ] 

jin xing edited comment on CALCITE-3211 at 7/26/19 5:28 AM:


[~julianhyde]
Thanks a lot for reply ~

I updated the description of the issue. I'm working on a rule for 
materialization matching of {{Union}} 
https://issues.apache.org/jira/browse/CALCITE-3214

When I run below test, {{SubstitutionVisitor}} suffers into a infinite loop

{code:java}
  @Test public void testDEV() {
String sql0 = "select * from \"emps\" where \"empid\" < 300";
String sql1 = "select * from \"emps\" where \"empid\" > 200";
checkMaterialize(sql0 + " union all " + sql1, sql1 + " union all " + sql0);
  }
{code}

My previous understanding is wrong. I think the problem is from here:

{code:java}
Lists.transform(nodes, MutableRels::toMutable);
{code}

When we {{get}} from the result list, a new {{MutableRel}} is created.




was (Author: jinxing6...@126.com):
[~julianhyde]
Thanks a lot for reply ~

I updated the description of the issue. I'm working on a rule for 
materialization matching of {{Union}} 
https://issues.apache.org/jira/browse/CALCITE-3214

When I run below test, {{SubstitutionVisitor}} suffers into a infinite loop

{code:java}
  @Test public void testDEV() {
String sql0 = "select * from \"emps\" where \"empid\" < 300";
String sql1 = "select * from \"emps\" where \"empid\" > 200";
checkMaterialize(sql0 + " union all " + sql1, sql1 + " union all " + sql0);
  }
{code}

I think the problem is from here:

{code:java}
Lists.transform(nodes, MutableRels::toMutable);
{code}

When we {{get}} from the result list, a new {{MutableRel}} is created.



> MutableRel returned from MutableRels::toMutables may lose reference to parent
> -
>
> Key: CALCITE-3211
> URL: https://issues.apache.org/jira/browse/CALCITE-3211
> Project: Calcite
>  Issue Type: Bug
>Reporter: jin xing
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Current implementation of {{MutableRels::toMutables}} is as below:
> {code:java}
>   private static List toMutables(List nodes) {
> return Lists.transform(nodes, MutableRels::toMutable);
>   }
> {code}
> Thus every time we {{get}} from the result list, a new {{MutableRel}} will be 
> created:
> {code:java}
>   private static class TransformingRandomAccessList extends 
> AbstractList
>   implements RandomAccess, Serializable {
> final List fromList;
> final Function function;
> TransformingRandomAccessList(List fromList, Function extends T> function) {
>   this.fromList = checkNotNull(fromList);
>   this.function = checkNotNull(function);
> }
> @Override
> public T get(int index) {
>   return function.apply(fromList.get(index));
> }
> ..
> {code}
> As a result, 
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/plan/SubstitutionVisitor.java#L514
>{{SubstitutionVisitor}} will fail to check whether a node has been 
> met/matched before



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Comment Edited] (CALCITE-3211) MutableRel returned from MutableRels::toMutables may lose reference to parent

[jin xing (JIRA)]

[ 
https://issues.apache.org/jira/browse/CALCITE-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16893336#comment-16893336
 ] 

jin xing edited comment on CALCITE-3211 at 7/26/19 5:28 AM:


[~julianhyde]
Thanks a lot for reply ~

I updated the description of the issue. I'm working on a rule for 
materialization matching of {{Union}} 
https://issues.apache.org/jira/browse/CALCITE-3214

When I run below test, {{SubstitutionVisitor}} suffers into an infinite loop

{code:java}
  @Test public void testDEV() {
String sql0 = "select * from \"emps\" where \"empid\" < 300";
String sql1 = "select * from \"emps\" where \"empid\" > 200";
checkMaterialize(sql0 + " union all " + sql1, sql1 + " union all " + sql0);
  }
{code}

My previous understanding is wrong. I think the problem is from here:

{code:java}
Lists.transform(nodes, MutableRels::toMutable);
{code}

When we {{get}} from the result list, a new {{MutableRel}} is created.




was (Author: jinxing6...@126.com):
[~julianhyde]
Thanks a lot for reply ~

I updated the description of the issue. I'm working on a rule for 
materialization matching of {{Union}} 
https://issues.apache.org/jira/browse/CALCITE-3214

When I run below test, {{SubstitutionVisitor}} suffers into a infinite loop

{code:java}
  @Test public void testDEV() {
String sql0 = "select * from \"emps\" where \"empid\" < 300";
String sql1 = "select * from \"emps\" where \"empid\" > 200";
checkMaterialize(sql0 + " union all " + sql1, sql1 + " union all " + sql0);
  }
{code}

My previous understanding is wrong. I think the problem is from here:

{code:java}
Lists.transform(nodes, MutableRels::toMutable);
{code}

When we {{get}} from the result list, a new {{MutableRel}} is created.



> MutableRel returned from MutableRels::toMutables may lose reference to parent
> -
>
> Key: CALCITE-3211
> URL: https://issues.apache.org/jira/browse/CALCITE-3211
> Project: Calcite
>  Issue Type: Bug
>Reporter: jin xing
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Current implementation of {{MutableRels::toMutables}} is as below:
> {code:java}
>   private static List toMutables(List nodes) {
> return Lists.transform(nodes, MutableRels::toMutable);
>   }
> {code}
> Thus every time we {{get}} from the result list, a new {{MutableRel}} will be 
> created:
> {code:java}
>   private static class TransformingRandomAccessList extends 
> AbstractList
>   implements RandomAccess, Serializable {
> final List fromList;
> final Function function;
> TransformingRandomAccessList(List fromList, Function extends T> function) {
>   this.fromList = checkNotNull(fromList);
>   this.function = checkNotNull(function);
> }
> @Override
> public T get(int index) {
>   return function.apply(fromList.get(index));
> }
> ..
> {code}
> As a result, 
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/plan/SubstitutionVisitor.java#L514
>{{SubstitutionVisitor}} will fail to check whether a node has been 
> met/matched before



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Comment Edited] (CALCITE-3211) MutableRel returned from MutableRels::toMutables may lose reference to parent

[jin xing (JIRA)]

[ 
https://issues.apache.org/jira/browse/CALCITE-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16893336#comment-16893336
 ] 

jin xing edited comment on CALCITE-3211 at 7/26/19 5:29 AM:


[~julianhyde]
Thanks a lot for reply ~

I updated the description of the issue. I'm working on a rule for 
materialization matching of {{Union}} 
https://issues.apache.org/jira/browse/CALCITE-3214

When I run below test, {{SubstitutionVisitor}} suffers into an infinite loop

{code:java}
  @Test public void testDEV() {
String sql0 = "select * from \"emps\" where \"empid\" < 300";
String sql1 = "select * from \"emps\" where \"empid\" > 200";
checkMaterialize(sql0 + " union all " + sql1, sql1 + " union all " + sql0);
  }
{code}

My previous understanding is wrong. I think the problem is from here:

{code:java}
Lists.transform(nodes, MutableRels::toMutable);
{code}

When we {{get}} from the result list, a new {{MutableRel}} is created.
As a result, 
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/plan/SubstitutionVisitor.java#L514
 SubstitutionVisitor will fail to check whether a node has been met/matched 
before




was (Author: jinxing6...@126.com):
[~julianhyde]
Thanks a lot for reply ~

I updated the description of the issue. I'm working on a rule for 
materialization matching of {{Union}} 
https://issues.apache.org/jira/browse/CALCITE-3214

When I run below test, {{SubstitutionVisitor}} suffers into an infinite loop

{code:java}
  @Test public void testDEV() {
String sql0 = "select * from \"emps\" where \"empid\" < 300";
String sql1 = "select * from \"emps\" where \"empid\" > 200";
checkMaterialize(sql0 + " union all " + sql1, sql1 + " union all " + sql0);
  }
{code}

My previous understanding is wrong. I think the problem is from here:

{code:java}
Lists.transform(nodes, MutableRels::toMutable);
{code}

When we {{get}} from the result list, a new {{MutableRel}} is created.



> MutableRel returned from MutableRels::toMutables may lose reference to parent
> -
>
> Key: CALCITE-3211
> URL: https://issues.apache.org/jira/browse/CALCITE-3211
> Project: Calcite
>  Issue Type: Bug
>Reporter: jin xing
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Current implementation of {{MutableRels::toMutables}} is as below:
> {code:java}
>   private static List toMutables(List nodes) {
> return Lists.transform(nodes, MutableRels::toMutable);
>   }
> {code}
> Thus every time we {{get}} from the result list, a new {{MutableRel}} will be 
> created:
> {code:java}
>   private static class TransformingRandomAccessList extends 
> AbstractList
>   implements RandomAccess, Serializable {
> final List fromList;
> final Function function;
> TransformingRandomAccessList(List fromList, Function extends T> function) {
>   this.fromList = checkNotNull(fromList);
>   this.function = checkNotNull(function);
> }
> @Override
> public T get(int index) {
>   return function.apply(fromList.get(index));
> }
> ..
> {code}
> As a result, 
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/plan/SubstitutionVisitor.java#L514
>{{SubstitutionVisitor}} will fail to check whether a node has been 
> met/matched before



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Updated] (CALCITE-3211) MutableRel returned from MutableRels::toMutables may fail to be identified by Subs

[jin xing (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing updated CALCITE-3211:
--
Summary: MutableRel returned from MutableRels::toMutables  may fail to be 
identified by Subs  (was: MutableRel returned from MutableRels::toMutables may 
lose reference to parent)

> MutableRel returned from MutableRels::toMutables  may fail to be identified 
> by Subs
> ---
>
> Key: CALCITE-3211
> URL: https://issues.apache.org/jira/browse/CALCITE-3211
> Project: Calcite
>  Issue Type: Bug
>Reporter: jin xing
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Current implementation of {{MutableRels::toMutables}} is as below:
> {code:java}
>   private static List toMutables(List nodes) {
> return Lists.transform(nodes, MutableRels::toMutable);
>   }
> {code}
> Thus every time we {{get}} from the result list, a new {{MutableRel}} will be 
> created:
> {code:java}
>   private static class TransformingRandomAccessList extends 
> AbstractList
>   implements RandomAccess, Serializable {
> final List fromList;
> final Function function;
> TransformingRandomAccessList(List fromList, Function extends T> function) {
>   this.fromList = checkNotNull(fromList);
>   this.function = checkNotNull(function);
> }
> @Override
> public T get(int index) {
>   return function.apply(fromList.get(index));
> }
> ..
> {code}
> As a result, 
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/plan/SubstitutionVisitor.java#L514
>{{SubstitutionVisitor}} will fail to check whether a node has been 
> met/matched before



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Updated] (CALCITE-3211) MutableRel returned from MutableRels::toMutables may fail to be identified by SubstitutionVisitor during matching

[jin xing (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing updated CALCITE-3211:
--
Summary: MutableRel returned from MutableRels::toMutables  may fail to be 
identified by SubstitutionVisitor during matching  (was: MutableRel returned 
from MutableRels::toMutables  may fail to be identified by Subs)

> MutableRel returned from MutableRels::toMutables  may fail to be identified 
> by SubstitutionVisitor during matching
> --
>
> Key: CALCITE-3211
> URL: https://issues.apache.org/jira/browse/CALCITE-3211
> Project: Calcite
>  Issue Type: Bug
>Reporter: jin xing
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Current implementation of {{MutableRels::toMutables}} is as below:
> {code:java}
>   private static List toMutables(List nodes) {
> return Lists.transform(nodes, MutableRels::toMutable);
>   }
> {code}
> Thus every time we {{get}} from the result list, a new {{MutableRel}} will be 
> created:
> {code:java}
>   private static class TransformingRandomAccessList extends 
> AbstractList
>   implements RandomAccess, Serializable {
> final List fromList;
> final Function function;
> TransformingRandomAccessList(List fromList, Function extends T> function) {
>   this.fromList = checkNotNull(fromList);
>   this.function = checkNotNull(function);
> }
> @Override
> public T get(int index) {
>   return function.apply(fromList.get(index));
> }
> ..
> {code}
> As a result, 
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/plan/SubstitutionVisitor.java#L514
>{{SubstitutionVisitor}} will fail to check whether a node has been 
> met/matched before



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Comment Edited] (CALCITE-3211) MutableRel returned from MutableRels::toMutables may fail to be identified by SubstitutionVisitor during matching

[jin xing (JIRA)]

[ 
https://issues.apache.org/jira/browse/CALCITE-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16893336#comment-16893336
 ] 

jin xing edited comment on CALCITE-3211 at 7/29/19 2:33 AM:


[~julianhyde]
Thanks a lot for reply ~

I updated the description of the issue. I'm working on a rule for 
materialization matching of {{Union}} 
https://issues.apache.org/jira/browse/CALCITE-3214

When I run below test, {{SubstitutionVisitor}} suffers into an 
{color:red}infinite/dead loop{color} and {{SubstitutionVisitor}} never return.

{code:java}
  @Test public void testDEV() {
String sql0 = "select * from \"emps\" where \"empid\" < 300";
String sql1 = "select * from \"emps\" where \"empid\" > 200";
checkMaterialize(sql0 + " union all " + sql1, sql1 + " union all " + sql0);
  }
{code}

My previous understanding is wrong. I think the problem is from here:

{code:java}
Lists.transform(nodes, MutableRels::toMutable);
{code}

When we {{get}} from the result list, a new {{MutableRel}} is created.
As a result, 
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/plan/SubstitutionVisitor.java#L514
 SubstitutionVisitor will fail to check whether a node has been met/matched 
before




was (Author: jinxing6...@126.com):
[~julianhyde]
Thanks a lot for reply ~

I updated the description of the issue. I'm working on a rule for 
materialization matching of {{Union}} 
https://issues.apache.org/jira/browse/CALCITE-3214

When I run below test, {{SubstitutionVisitor}} suffers into an infinite loop

{code:java}
  @Test public void testDEV() {
String sql0 = "select * from \"emps\" where \"empid\" < 300";
String sql1 = "select * from \"emps\" where \"empid\" > 200";
checkMaterialize(sql0 + " union all " + sql1, sql1 + " union all " + sql0);
  }
{code}

My previous understanding is wrong. I think the problem is from here:

{code:java}
Lists.transform(nodes, MutableRels::toMutable);
{code}

When we {{get}} from the result list, a new {{MutableRel}} is created.
As a result, 
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/plan/SubstitutionVisitor.java#L514
 SubstitutionVisitor will fail to check whether a node has been met/matched 
before



> MutableRel returned from MutableRels::toMutables  may fail to be identified 
> by SubstitutionVisitor during matching
> --
>
> Key: CALCITE-3211
> URL: https://issues.apache.org/jira/browse/CALCITE-3211
> Project: Calcite
>  Issue Type: Bug
>Reporter: jin xing
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Current implementation of {{MutableRels::toMutables}} is as below:
> {code:java}
>   private static List toMutables(List nodes) {
> return Lists.transform(nodes, MutableRels::toMutable);
>   }
> {code}
> Thus every time we {{get}} from the result list, a new {{MutableRel}} will be 
> created:
> {code:java}
>   private static class TransformingRandomAccessList extends 
> AbstractList
>   implements RandomAccess, Serializable {
> final List fromList;
> final Function function;
> TransformingRandomAccessList(List fromList, Function extends T> function) {
>   this.fromList = checkNotNull(fromList);
>   this.function = checkNotNull(function);
> }
> @Override
> public T get(int index) {
>   return function.apply(fromList.get(index));
> }
> ..
> {code}
> As a result, 
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/plan/SubstitutionVisitor.java#L514
>{{SubstitutionVisitor}} will fail to check whether a node has been 
> met/matched before



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Updated] (CALCITE-3211) MutableRel returned from MutableRels::toMutables may fail to be identified by SubstitutionVisitor during matching

[jin xing (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing updated CALCITE-3211:
--
Issue Type: Improvement  (was: Bug)

> MutableRel returned from MutableRels::toMutables  may fail to be identified 
> by SubstitutionVisitor during matching
> --
>
> Key: CALCITE-3211
> URL: https://issues.apache.org/jira/browse/CALCITE-3211
> Project: Calcite
>  Issue Type: Improvement
>Reporter: jin xing
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Current implementation of {{MutableRels::toMutables}} is as below:
> {code:java}
>   private static List toMutables(List nodes) {
> return Lists.transform(nodes, MutableRels::toMutable);
>   }
> {code}
> Thus every time we {{get}} from the result list, a new {{MutableRel}} will be 
> created:
> {code:java}
>   private static class TransformingRandomAccessList extends 
> AbstractList
>   implements RandomAccess, Serializable {
> final List fromList;
> final Function function;
> TransformingRandomAccessList(List fromList, Function extends T> function) {
>   this.fromList = checkNotNull(fromList);
>   this.function = checkNotNull(function);
> }
> @Override
> public T get(int index) {
>   return function.apply(fromList.get(index));
> }
> ..
> {code}
> As a result, 
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/plan/SubstitutionVisitor.java#L514
>{{SubstitutionVisitor}} will fail to check whether a node has been 
> met/matched before



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Updated] (CALCITE-3211) MutableRel returned from MutableRels::toMutables may fail to be identified by SubstitutionVisitor during matching

[jin xing (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing updated CALCITE-3211:
--
Issue Type: Bug  (was: Improvement)

> MutableRel returned from MutableRels::toMutables  may fail to be identified 
> by SubstitutionVisitor during matching
> --
>
> Key: CALCITE-3211
> URL: https://issues.apache.org/jira/browse/CALCITE-3211
> Project: Calcite
>  Issue Type: Bug
>Reporter: jin xing
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Current implementation of {{MutableRels::toMutables}} is as below:
> {code:java}
>   private static List toMutables(List nodes) {
> return Lists.transform(nodes, MutableRels::toMutable);
>   }
> {code}
> Thus every time we {{get}} from the result list, a new {{MutableRel}} will be 
> created:
> {code:java}
>   private static class TransformingRandomAccessList extends 
> AbstractList
>   implements RandomAccess, Serializable {
> final List fromList;
> final Function function;
> TransformingRandomAccessList(List fromList, Function extends T> function) {
>   this.fromList = checkNotNull(fromList);
>   this.function = checkNotNull(function);
> }
> @Override
> public T get(int index) {
>   return function.apply(fromList.get(index));
> }
> ..
> {code}
> As a result, 
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/plan/SubstitutionVisitor.java#L514
>{{SubstitutionVisitor}} will fail to check whether a node has been 
> met/matched before



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Updated] (CALCITE-3211) List of MutableRel may fail to be identified by SubstitutionVisitor during matching

[jin xing (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing updated CALCITE-3211:
--
Summary: List of MutableRel may fail to be identified by 
SubstitutionVisitor during matching  (was: MutableRel returned from 
MutableRels::toMutables  may fail to be identified by SubstitutionVisitor 
during matching)

> List of MutableRel may fail to be identified by SubstitutionVisitor during 
> matching
> ---
>
> Key: CALCITE-3211
> URL: https://issues.apache.org/jira/browse/CALCITE-3211
> Project: Calcite
>  Issue Type: Bug
>Reporter: jin xing
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Current implementation of {{MutableRels::toMutables}} is as below:
> {code:java}
>   private static List toMutables(List nodes) {
> return Lists.transform(nodes, MutableRels::toMutable);
>   }
> {code}
> Thus every time we {{get}} from the result list, a new {{MutableRel}} will be 
> created:
> {code:java}
>   private static class TransformingRandomAccessList extends 
> AbstractList
>   implements RandomAccess, Serializable {
> final List fromList;
> final Function function;
> TransformingRandomAccessList(List fromList, Function extends T> function) {
>   this.fromList = checkNotNull(fromList);
>   this.function = checkNotNull(function);
> }
> @Override
> public T get(int index) {
>   return function.apply(fromList.get(index));
> }
> ..
> {code}
> As a result, 
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/plan/SubstitutionVisitor.java#L514
>{{SubstitutionVisitor}} will fail to check whether a node has been 
> met/matched before



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Commented] (CALCITE-3211) List of MutableRel may fail to be identified by SubstitutionVisitor during matching

[jin xing (JIRA)]

[ 
https://issues.apache.org/jira/browse/CALCITE-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16895748#comment-16895748
 ] 

jin xing commented on CALCITE-3211:
---

Thanks a lot [~julianhyde]
"People should not call {{Lists.transform}} if they are going to use the list 
more than once and expect the same results" -- This is true.

I updated the PR according to your comment, it's great if you can take a look 
when you have time :)

> List of MutableRel may fail to be identified by SubstitutionVisitor during 
> matching
> ---
>
> Key: CALCITE-3211
> URL: https://issues.apache.org/jira/browse/CALCITE-3211
> Project: Calcite
>  Issue Type: Bug
>Reporter: jin xing
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Current implementation of {{MutableRels::toMutables}} is as below:
> {code:java}
>   private static List toMutables(List nodes) {
> return Lists.transform(nodes, MutableRels::toMutable);
>   }
> {code}
> Thus every time we {{get}} from the result list, a new {{MutableRel}} will be 
> created:
> {code:java}
>   private static class TransformingRandomAccessList extends 
> AbstractList
>   implements RandomAccess, Serializable {
> final List fromList;
> final Function function;
> TransformingRandomAccessList(List fromList, Function extends T> function) {
>   this.fromList = checkNotNull(fromList);
>   this.function = checkNotNull(function);
> }
> @Override
> public T get(int index) {
>   return function.apply(fromList.get(index));
> }
> ..
> {code}
> As a result, 
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/plan/SubstitutionVisitor.java#L514
>{{SubstitutionVisitor}} will fail to check whether a node has been 
> met/matched before



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (CALCITE-3223) Non-RexInputRef may fails the matching of FilterToProjectUnifyRule during 'invert' by mistake.

[jin xing (JIRA)]

jin xing created CALCITE-3223:
-

 Summary: Non-RexInputRef may fails the matching of 
FilterToProjectUnifyRule during 'invert' by mistake.
 Key: CALCITE-3223
 URL: https://issues.apache.org/jira/browse/CALCITE-3223
 Project: Calcite
  Issue Type: Bug
  Components: core
Reporter: jin xing


In current code of 
{{FilterToProjectUnifyRule::invert}}(https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/plan/SubstitutionVisitor.java#L1124),
  the implementation 
1. Fails the matching when there is Non-RexInputRef  in the projects
2. Didn't check if all {{exprList}} has already ben set correctly.
As a result below tests  fails.

{code:java}
  @Test public void testFilterToProject0() {
String union =
"select * from \"emps\" where \"empid\" > 300\n"
+ "union all select * from \"emps\" where \"empid\" < 200";
String mv = "select *, \"empid\" * 2 from (" + union + ")";
String query = "select * from (" + union + ") where (\"empid\" * 2) > 3";
checkMaterialize(mv, query);
  }

  @Test public void testFilterToProject1() {
String agg =
"select \"deptno\", count(*) as \"c\", sum(\"salary\") as \"s\"\n"
+ "from \"emps\" group by \"deptno\"";
String mv = "select \"c\", \"s\", \"s\" from (" + agg + ")";
String query = "select * from (" + agg + ") where (\"s\" * 0.8) > 1";
checkNoMaterialize(mv, query, HR_FKUK_MODEL);
  }
{code}




--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Updated] (CALCITE-3223) Non-RexInputRef may fails the matching of FilterToProjectUnifyRule during 'invert' by mistake.

[jin xing (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing updated CALCITE-3223:
--
Description: 
In current code of 
{{FilterToProjectUnifyRule::invert}}(https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/plan/SubstitutionVisitor.java#L1124),
  the implementation 
1. Fails the matching when there is Non-RexInputRef  in the projects
2. Didn't check if all items of {{exprList}} has already been set correctly.
As a result below tests  fails.

{code:java}
  @Test public void testFilterToProject0() {
String union =
"select * from \"emps\" where \"empid\" > 300\n"
+ "union all select * from \"emps\" where \"empid\" < 200";
String mv = "select *, \"empid\" * 2 from (" + union + ")";
String query = "select * from (" + union + ") where (\"empid\" * 2) > 3";
checkMaterialize(mv, query);
  }

  @Test public void testFilterToProject1() {
String agg =
"select \"deptno\", count(*) as \"c\", sum(\"salary\") as \"s\"\n"
+ "from \"emps\" group by \"deptno\"";
String mv = "select \"c\", \"s\", \"s\" from (" + agg + ")";
String query = "select * from (" + agg + ") where (\"s\" * 0.8) > 1";
checkNoMaterialize(mv, query, HR_FKUK_MODEL);
  }
{code}


  was:
In current code of 
{{FilterToProjectUnifyRule::invert}}(https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/plan/SubstitutionVisitor.java#L1124),
  the implementation 
1. Fails the matching when there is Non-RexInputRef  in the projects
2. Didn't check if all items of {{exprList}} has already ben set correctly.
As a result below tests  fails.

{code:java}
  @Test public void testFilterToProject0() {
String union =
"select * from \"emps\" where \"empid\" > 300\n"
+ "union all select * from \"emps\" where \"empid\" < 200";
String mv = "select *, \"empid\" * 2 from (" + union + ")";
String query = "select * from (" + union + ") where (\"empid\" * 2) > 3";
checkMaterialize(mv, query);
  }

  @Test public void testFilterToProject1() {
String agg =
"select \"deptno\", count(*) as \"c\", sum(\"salary\") as \"s\"\n"
+ "from \"emps\" group by \"deptno\"";
String mv = "select \"c\", \"s\", \"s\" from (" + agg + ")";
String query = "select * from (" + agg + ") where (\"s\" * 0.8) > 1";
checkNoMaterialize(mv, query, HR_FKUK_MODEL);
  }
{code}



> Non-RexInputRef may fails the matching of FilterToProjectUnifyRule during 
> 'invert' by mistake.
> --
>
> Key: CALCITE-3223
> URL: https://issues.apache.org/jira/browse/CALCITE-3223
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Reporter: jin xing
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> In current code of 
> {{FilterToProjectUnifyRule::invert}}(https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/plan/SubstitutionVisitor.java#L1124),
>   the implementation 
> 1. Fails the matching when there is Non-RexInputRef  in the projects
> 2. Didn't check if all items of {{exprList}} has already been set correctly.
> As a result below tests  fails.
> {code:java}
>   @Test public void testFilterToProject0() {
> String union =
> "select * from \"emps\" where \"empid\" > 300\n"
> + "union all select * from \"emps\" where \"empid\" < 200";
> String mv = "select *, \"empid\" * 2 from (" + union + ")";
> String query = "select * from (" + union + ") where (\"empid\" * 2) > 3";
> checkMaterialize(mv, query);
>   }
>   @Test public void testFilterToProject1() {
> String agg =
> "select \"deptno\", count(*) as \"c\", sum(\"salary\") as \"s\"\n"
> + "from \"emps\" group by \"deptno\"";
> String mv = "select \"c\", \"s\", \"s\" from (" + agg + ")";
> String query = "select * from (" + agg + ") where (\"s\" * 0.8) > 1";
> checkNoMaterialize(mv, query, HR_FKUK_MODEL);
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Updated] (CALCITE-3223) Non-RexInputRef may fails the matching of FilterToProjectUnifyRule during 'invert' by mistake.

[jin xing (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing updated CALCITE-3223:
--
Description: 
In current code of 
{{FilterToProjectUnifyRule::invert}}(https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/plan/SubstitutionVisitor.java#L1124),
  the implementation 
1. Fails the matching when there is Non-RexInputRef  in the projects
2. Didn't check if all items of {{exprList}} has already ben set correctly.
As a result below tests  fails.

{code:java}
  @Test public void testFilterToProject0() {
String union =
"select * from \"emps\" where \"empid\" > 300\n"
+ "union all select * from \"emps\" where \"empid\" < 200";
String mv = "select *, \"empid\" * 2 from (" + union + ")";
String query = "select * from (" + union + ") where (\"empid\" * 2) > 3";
checkMaterialize(mv, query);
  }

  @Test public void testFilterToProject1() {
String agg =
"select \"deptno\", count(*) as \"c\", sum(\"salary\") as \"s\"\n"
+ "from \"emps\" group by \"deptno\"";
String mv = "select \"c\", \"s\", \"s\" from (" + agg + ")";
String query = "select * from (" + agg + ") where (\"s\" * 0.8) > 1";
checkNoMaterialize(mv, query, HR_FKUK_MODEL);
  }
{code}


  was:
In current code of 
{{FilterToProjectUnifyRule::invert}}(https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/plan/SubstitutionVisitor.java#L1124),
  the implementation 
1. Fails the matching when there is Non-RexInputRef  in the projects
2. Didn't check if all {{exprList}} has already ben set correctly.
As a result below tests  fails.

{code:java}
  @Test public void testFilterToProject0() {
String union =
"select * from \"emps\" where \"empid\" > 300\n"
+ "union all select * from \"emps\" where \"empid\" < 200";
String mv = "select *, \"empid\" * 2 from (" + union + ")";
String query = "select * from (" + union + ") where (\"empid\" * 2) > 3";
checkMaterialize(mv, query);
  }

  @Test public void testFilterToProject1() {
String agg =
"select \"deptno\", count(*) as \"c\", sum(\"salary\") as \"s\"\n"
+ "from \"emps\" group by \"deptno\"";
String mv = "select \"c\", \"s\", \"s\" from (" + agg + ")";
String query = "select * from (" + agg + ") where (\"s\" * 0.8) > 1";
checkNoMaterialize(mv, query, HR_FKUK_MODEL);
  }
{code}



> Non-RexInputRef may fails the matching of FilterToProjectUnifyRule during 
> 'invert' by mistake.
> --
>
> Key: CALCITE-3223
> URL: https://issues.apache.org/jira/browse/CALCITE-3223
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Reporter: jin xing
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> In current code of 
> {{FilterToProjectUnifyRule::invert}}(https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/plan/SubstitutionVisitor.java#L1124),
>   the implementation 
> 1. Fails the matching when there is Non-RexInputRef  in the projects
> 2. Didn't check if all items of {{exprList}} has already ben set correctly.
> As a result below tests  fails.
> {code:java}
>   @Test public void testFilterToProject0() {
> String union =
> "select * from \"emps\" where \"empid\" > 300\n"
> + "union all select * from \"emps\" where \"empid\" < 200";
> String mv = "select *, \"empid\" * 2 from (" + union + ")";
> String query = "select * from (" + union + ") where (\"empid\" * 2) > 3";
> checkMaterialize(mv, query);
>   }
>   @Test public void testFilterToProject1() {
> String agg =
> "select \"deptno\", count(*) as \"c\", sum(\"salary\") as \"s\"\n"
> + "from \"emps\" group by \"deptno\"";
> String mv = "select \"c\", \"s\", \"s\" from (" + agg + ")";
> String query = "select * from (" + agg + ") where (\"s\" * 0.8) > 1";
> checkNoMaterialize(mv, query, HR_FKUK_MODEL);
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (CALCITE-3226) RelBuilder doesn't keep the alias when scan from an expanded view

[jin xing (JIRA)]

jin xing created CALCITE-3226:
-

 Summary: RelBuilder doesn't keep the alias when scan from an 
expanded view
 Key: CALCITE-3226
 URL: https://issues.apache.org/jira/browse/CALCITE-3226
 Project: Calcite
  Issue Type: Bug
  Components: core
Reporter: jin xing


In current implementation of {{RelBuilder::scan}} 
([https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L1048]),
 the alias can be derived and recorded into {{Frame}} only when the {{RelNode}} 
is a {{TableScan}} 
([https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L2754]).
 But when {{RelBuilder::scan}} from an expanded view, the node is not a 
{{TableScan}} and the alias is not kept.  Below test failed - we cannot 
reference a field by alias "MYVIEW".
{code:java}
  @Test public void testExpandViewShouldKeepAlias() throws SQLException {
try (Connection connection = DriverManager.getConnection("jdbc:calcite:")) {
  final Frameworks.ConfigBuilder configBuilder =
  expandingConfig(connection);
  final RelOptTable.ViewExpander viewExpander =
  (RelOptTable.ViewExpander) 
Frameworks.getPlanner(configBuilder.build());
  final RelFactories.TableScanFactory tableScanFactory =
  RelFactories.expandingScanFactory(viewExpander,
  RelFactories.DEFAULT_TABLE_SCAN_FACTORY);
  configBuilder.context(Contexts.of(tableScanFactory));
  final RelBuilder builder = RelBuilder.create(configBuilder.build());
  RelNode node =
  builder.scan("MYVIEW")
  .project(
  builder.field(1, "MYVIEW", "EMPNO"),
  builder.field(1, "MYVIEW", "ENAME"))
  .build();
  String expected =
  "LogicalProject(EMPNO=[$0], ENAME=[$1])\n"
  + "  LogicalFilter(condition=[=(1, 1)])\n"
  + "LogicalTableScan(table=[[scott, EMP]])\n";
  assertThat(node, hasTree(expected));
}
  }
{code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Updated] (CALCITE-3226) RelBuilder doesn't keep the alias when scan from an expanded view

[jin xing (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing updated CALCITE-3226:
--
Description: 
In current implementation of {{RelBuilder::scan}} 
([https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L1048]),
 the alias can be derived and recorded into {{Frame}} only when the {{RelNode}} 
is a {{TableScan}} 
([https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L2754]).
 But when {{RelBuilder::scan}} from an expanded view, the node is not a 
{{TableScan}} and the alias is not kept.  Below test failed - we cannot 
reference a field by alias "MYVIEW".
{code:java}
  @Test public void testExpandViewShouldKeepAlias() throws SQLException {
try (Connection connection = DriverManager.getConnection("jdbc:calcite:")) {
  final Frameworks.ConfigBuilder configBuilder =
  expandingConfig(connection);
  final RelOptTable.ViewExpander viewExpander =
  (RelOptTable.ViewExpander) 
Frameworks.getPlanner(configBuilder.build());
  final RelFactories.TableScanFactory tableScanFactory =
  RelFactories.expandingScanFactory(viewExpander,
  RelFactories.DEFAULT_TABLE_SCAN_FACTORY);
  configBuilder.context(Contexts.of(tableScanFactory));
  final RelBuilder builder = RelBuilder.create(configBuilder.build());
  RelNode node =
  builder.scan("MYVIEW")
  .project(
  builder.field(1, "MYVIEW", "EMPNO"),   // Exception 
thrown from here.
  builder.field(1, "MYVIEW", "ENAME"))
  .build();
  String expected =
  "LogicalProject(EMPNO=[$0], ENAME=[$1])\n"
  + "  LogicalFilter(condition=[=(1, 1)])\n"
  + "LogicalTableScan(table=[[scott, EMP]])\n";
  assertThat(node, hasTree(expected));
}
  }
{code}

  was:
In current implementation of {{RelBuilder::scan}} 
([https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L1048]),
 the alias can be derived and recorded into {{Frame}} only when the {{RelNode}} 
is a {{TableScan}} 
([https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L2754]).
 But when {{RelBuilder::scan}} from an expanded view, the node is not a 
{{TableScan}} and the alias is not kept.  Below test failed - we cannot 
reference a field by alias "MYVIEW".
{code:java}
  @Test public void testExpandViewShouldKeepAlias() throws SQLException {
try (Connection connection = DriverManager.getConnection("jdbc:calcite:")) {
  final Frameworks.ConfigBuilder configBuilder =
  expandingConfig(connection);
  final RelOptTable.ViewExpander viewExpander =
  (RelOptTable.ViewExpander) 
Frameworks.getPlanner(configBuilder.build());
  final RelFactories.TableScanFactory tableScanFactory =
  RelFactories.expandingScanFactory(viewExpander,
  RelFactories.DEFAULT_TABLE_SCAN_FACTORY);
  configBuilder.context(Contexts.of(tableScanFactory));
  final RelBuilder builder = RelBuilder.create(configBuilder.build());
  RelNode node =
  builder.scan("MYVIEW")
  .project(
  builder.field(1, "MYVIEW", "EMPNO"),
  builder.field(1, "MYVIEW", "ENAME"))
  .build();
  String expected =
  "LogicalProject(EMPNO=[$0], ENAME=[$1])\n"
  + "  LogicalFilter(condition=[=(1, 1)])\n"
  + "LogicalTableScan(table=[[scott, EMP]])\n";
  assertThat(node, hasTree(expected));
}
  }
{code}


> RelBuilder doesn't keep the alias when scan from an expanded view
> -
>
> Key: CALCITE-3226
> URL: https://issues.apache.org/jira/browse/CALCITE-3226
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Reporter: jin xing
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> In current implementation of {{RelBuilder::scan}} 
> ([https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L1048]),
>  the alias can be derived and recorded into {{Frame}} only when the 
> {{RelNode}} is a {{TableScan}} 
> ([https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L2754]).
>  But when {{RelBuilder::scan}} from an expanded view, the node is not a 
> {{TableScan}} and the alias is not kept.  Below test failed - we cannot 
> reference a field by alias "MYVIEW".
> {code:java}
>   @Test public void testExpandViewShouldKeepAlias() throws SQLException {
> try (Connection connection = 
> DriverManager.getConnection("jd

[jira] [Updated] (CALCITE-3228) Error while applying rule ProjectScanRule: interpreter

[jin xing (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing updated CALCITE-3228:
--
Attachment: CALCITE-3228.patch

> Error while applying rule ProjectScanRule: interpreter
> --
>
> Key: CALCITE-3228
> URL: https://issues.apache.org/jira/browse/CALCITE-3228
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Reporter: Chunwei Lei
>Priority: Critical
> Fix For: 1.21.0
>
> Attachments: CALCITE-3228.patch
>
>
> The following test can reproduce the issue.
>  
> {code:java}
> // FrameworksTest.java
>  @Test public void testMinMax() throws Exception {
>  Table table = new TableImpl();
>  final SchemaPlus rootSchema = Frameworks.createRootSchema(true);
>  SchemaPlus schema = rootSchema.add("x", new AbstractSchema());
>  schema.add("MYTABLE", table);
>  List traitDefs = new ArrayList<>();
>  traitDefs.add(ConventionTraitDef.INSTANCE);
>  traitDefs.add(RelDistributionTraitDef.INSTANCE);
>  SqlParser.Config parserConfig =
>  SqlParser.configBuilder(SqlParser.Config.DEFAULT)
>  .setCaseSensitive(false)
>  .build();
> final FrameworkConfig config = Frameworks.newConfigBuilder()
>  .parserConfig(parserConfig)
>  .defaultSchema(schema)
>  .traitDefs(traitDefs)
>  // define the rules you want to apply
>  .ruleSets(
>  RuleSets.ofList(AbstractConverter.ExpandConversionRule.INSTANCE, 
> ProjectTableScanRule.INSTANCE))
>  .programs(Programs.ofRules(Programs.RULE_SET))
>  .build();
> executeQuery(config, " select min(id) as mi, max(id) as ma from mytable where 
> id=1 group by id",
>  CalciteSystemProperty.DEBUG.value());
>  }
> {code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Commented] (CALCITE-3228) Error while applying rule ProjectScanRule: interpreter

[jin xing (JIRA)]

[ 
https://issues.apache.org/jira/browse/CALCITE-3228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16899594#comment-16899594
 ] 

jin xing commented on CALCITE-3228:
---

[^CALCITE-3228.patch]

Hi [~Chunwei Lei]
I try to dig and learn from this issue;

*Project::getMapping* try to return an inverse-surjection mapping. 
*SurjectionWithInverse* is defined as below:

 
{code:java}
/**
 * A surjection with inverse has precisely one source for each target.
 * (Whereas a general surjection has at least one source for each target.)
 * Every source has at most one target.
 */
{code}
That's why the test you provided throws

 
{code:java}
...
Caused by: java.lang.IllegalArgumentException: source #0 is already mapped to 
target #1
 at 
org.apache.calcite.util.mapping.Mappings$SurjectionWithInverse.set(Mappings.java:1326)
 at org.apache.calcite.rel.core.Project.getMapping(Project.java:285)
 at org.apache.calcite.rel.core.Project.getMapping(Project.java:256)
 at 
org.apache.calcite.rel.rules.ProjectTableScanRule.apply(ProjectTableScanRule.java:107)
 at 
org.apache.calcite.rel.rules.ProjectTableScanRule$2.onMatch(ProjectTableScanRule.java:83)
 at 
org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:208)
...
{code}
I have two ideas and it's great if you can confirm and rectify when you have 
time:

1. *ProjectTableScanRule* should not be limited by the Project::getMapping. I 
mean ProjectTableScanRule can handle to create a 

BindableTableScan as long as *project.getProjects().stream().allMatch(p -> p 
instanceof RexInputRef) == true,* and no need to be limited by a 
inverse-surjection mapping.

2. The definition of *Project::getMapping* is as below
{code:java}
/**
 * Returns a mapping of a set of project expressions.
 *
 * The mapping is an inverse surjection.
 * Every target has a source field, but
 * a source field may appear as zero, one, or more target fields.
 * Thus you can safely call
 * {@link 
org.apache.calcite.util.mapping.Mappings.TargetMapping#getTarget(int)}.
 *
{code}
But in my understanding, it's not safe to call *getTarget(int)* but safe to 
call *getSource(int);*

I try to file a fix for this issue and illustrate my idea. It's great if you 
can shepherd and tell if it's the right direction.

If you haven't started a PR on this issue, I can work on it under your shepherd.

But since you have the expertise, if you have already working on, I will learn 
and track your PR.

> Error while applying rule ProjectScanRule: interpreter
> --
>
> Key: CALCITE-3228
> URL: https://issues.apache.org/jira/browse/CALCITE-3228
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Reporter: Chunwei Lei
>Priority: Critical
> Fix For: 1.21.0
>
> Attachments: CALCITE-3228.patch
>
>
> The following test can reproduce the issue.
>  
> {code:java}
> // FrameworksTest.java
>  @Test public void testMinMax() throws Exception {
>  Table table = new TableImpl();
>  final SchemaPlus rootSchema = Frameworks.createRootSchema(true);
>  SchemaPlus schema = rootSchema.add("x", new AbstractSchema());
>  schema.add("MYTABLE", table);
>  List traitDefs = new ArrayList<>();
>  traitDefs.add(ConventionTraitDef.INSTANCE);
>  traitDefs.add(RelDistributionTraitDef.INSTANCE);
>  SqlParser.Config parserConfig =
>  SqlParser.configBuilder(SqlParser.Config.DEFAULT)
>  .setCaseSensitive(false)
>  .build();
> final FrameworkConfig config = Frameworks.newConfigBuilder()
>  .parserConfig(parserConfig)
>  .defaultSchema(schema)
>  .traitDefs(traitDefs)
>  // define the rules you want to apply
>  .ruleSets(
>  RuleSets.ofList(AbstractConverter.ExpandConversionRule.INSTANCE, 
> ProjectTableScanRule.INSTANCE))
>  .programs(Programs.ofRules(Programs.RULE_SET))
>  .build();
> executeQuery(config, " select min(id) as mi, max(id) as ma from mytable where 
> id=1 group by id",
>  CalciteSystemProperty.DEBUG.value());
>  }
> {code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Comment Edited] (CALCITE-3228) Error while applying rule ProjectScanRule: interpreter

[jin xing (JIRA)]

[ 
https://issues.apache.org/jira/browse/CALCITE-3228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16899594#comment-16899594
 ] 

jin xing edited comment on CALCITE-3228 at 8/4/19 9:28 AM:
---

[^CALCITE-3228.patch]

Hi [~Chunwei Lei]
 I try to dig and learn from this issue;

*Project::getMapping* try to return an inverse-surjection mapping. 
*SurjectionWithInverse* is defined as below:
{code:java}
/**
 * A surjection with inverse has precisely one source for each target.
 * (Whereas a general surjection has at least one source for each target.)
 * Every source has at most one target.
 */
{code}
That's why the test you provided throws
{code:java}
...
Caused by: java.lang.IllegalArgumentException: source #0 is already mapped to 
target #1
 at 
org.apache.calcite.util.mapping.Mappings$SurjectionWithInverse.set(Mappings.java:1326)
 at org.apache.calcite.rel.core.Project.getMapping(Project.java:285)
 at org.apache.calcite.rel.core.Project.getMapping(Project.java:256)
 at 
org.apache.calcite.rel.rules.ProjectTableScanRule.apply(ProjectTableScanRule.java:107)
 at 
org.apache.calcite.rel.rules.ProjectTableScanRule$2.onMatch(ProjectTableScanRule.java:83)
 at 
org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:208)
...
{code}
I have two ideas and it's great if you can confirm and rectify when you have 
time:

1. *ProjectTableScanRule* should not be limited by the Project::getMapping. I 
mean ProjectTableScanRule can handle to create a *BindableTableScan* as long as 
*project.getProjects().stream().allMatch(p -> p instanceof RexInputRef) == 
true,* and no need to be limited by a inverse-surjection mapping.

2. The definition of *Project::getMapping* is as below
{code:java}
/**
 * Returns a mapping of a set of project expressions.
 *
 * The mapping is an inverse surjection.
 * Every target has a source field, but
 * a source field may appear as zero, one, or more target fields.
 * Thus you can safely call
 * {@link 
org.apache.calcite.util.mapping.Mappings.TargetMapping#getTarget(int)}.
 *
{code}
But in my understanding, it's not safe to call *getTarget(int)* but safe to 
call *getSource(int);*

I try to file a fix for this issue and illustrate my idea. It's great if you 
can shepherd and tell if it's the right direction.

If you haven't started a PR on this issue, I can work on it under your shepherd.

But since you have the expertise, if you have already working on, I will learn 
and track your PR.


was (Author: jinxing6...@126.com):
[^CALCITE-3228.patch]

Hi [~Chunwei Lei]
I try to dig and learn from this issue;

*Project::getMapping* try to return an inverse-surjection mapping. 
*SurjectionWithInverse* is defined as below:

 
{code:java}
/**
 * A surjection with inverse has precisely one source for each target.
 * (Whereas a general surjection has at least one source for each target.)
 * Every source has at most one target.
 */
{code}
That's why the test you provided throws

 
{code:java}
...
Caused by: java.lang.IllegalArgumentException: source #0 is already mapped to 
target #1
 at 
org.apache.calcite.util.mapping.Mappings$SurjectionWithInverse.set(Mappings.java:1326)
 at org.apache.calcite.rel.core.Project.getMapping(Project.java:285)
 at org.apache.calcite.rel.core.Project.getMapping(Project.java:256)
 at 
org.apache.calcite.rel.rules.ProjectTableScanRule.apply(ProjectTableScanRule.java:107)
 at 
org.apache.calcite.rel.rules.ProjectTableScanRule$2.onMatch(ProjectTableScanRule.java:83)
 at 
org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:208)
...
{code}
I have two ideas and it's great if you can confirm and rectify when you have 
time:

1. *ProjectTableScanRule* should not be limited by the Project::getMapping. I 
mean ProjectTableScanRule can handle to create a 

BindableTableScan as long as *project.getProjects().stream().allMatch(p -> p 
instanceof RexInputRef) == true,* and no need to be limited by a 
inverse-surjection mapping.

2. The definition of *Project::getMapping* is as below
{code:java}
/**
 * Returns a mapping of a set of project expressions.
 *
 * The mapping is an inverse surjection.
 * Every target has a source field, but
 * a source field may appear as zero, one, or more target fields.
 * Thus you can safely call
 * {@link 
org.apache.calcite.util.mapping.Mappings.TargetMapping#getTarget(int)}.
 *
{code}
But in my understanding, it's not safe to call *getTarget(int)* but safe to 
call *getSource(int);*

I try to file a fix for this issue and illustrate my idea. It's great if you 
can shepherd and tell if it's the right direction.

If you haven't started a PR on this issue, I can work on it under your shepherd.

But since you have the expertise, if you have already working on, I will learn 
and track your PR.

> Error while applying rule ProjectScanRule: interpreter
> --
>
>   

[jira] [Comment Edited] (CALCITE-3228) Error while applying rule ProjectScanRule: interpreter

[jin xing (JIRA)]

[ 
https://issues.apache.org/jira/browse/CALCITE-3228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16899594#comment-16899594
 ] 

jin xing edited comment on CALCITE-3228 at 8/4/19 9:33 AM:
---

[^CALCITE-3228.patch]

Hi [~Chunwei Lei]
 I try to dig and learn from this issue;

*Project::getMapping* try to return an inverse-surjection mapping. 
*SurjectionWithInverse* is defined as below:
{code:java}
/**
 * A surjection with inverse has precisely one source for each target.
 * (Whereas a general surjection has at least one source for each target.)
 * Every source has at most one target.
 */
{code}
That's why the test you provided throws
{code:java}
...
Caused by: java.lang.IllegalArgumentException: source #0 is already mapped to 
target #1
 at 
org.apache.calcite.util.mapping.Mappings$SurjectionWithInverse.set(Mappings.java:1326)
 at org.apache.calcite.rel.core.Project.getMapping(Project.java:285)
 at org.apache.calcite.rel.core.Project.getMapping(Project.java:256)
 at 
org.apache.calcite.rel.rules.ProjectTableScanRule.apply(ProjectTableScanRule.java:107)
 at 
org.apache.calcite.rel.rules.ProjectTableScanRule$2.onMatch(ProjectTableScanRule.java:83)
 at 
org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:208)
...
{code}
I have two ideas and it's great if you can confirm and rectify when you have 
time:

1. *ProjectTableScanRule* should not be limited by the Project::getMapping. I 
mean ProjectTableScanRule can handle to create a *BindableTableScan* as long as 
*project.getProjects().stream().allMatch(p -> p instanceof RexInputRef) == 
true,* and no need to be limited by a inverse-surjection mapping.

2. The definition of *Project::getMapping* is as below
{code:java}
/**
 * Returns a mapping of a set of project expressions.
 *
 * The mapping is an inverse surjection.
 * Every target has a source field, but
 * a source field may appear as zero, one, or more target fields.
 * Thus you can safely call
 * {@link 
org.apache.calcite.util.mapping.Mappings.TargetMapping#getTarget(int)}.
 *
{code}
But in my understanding, it's not safe to call *getTarget(int)* but safe to 
call *getSource(int);*

I filed a fix for this issue as attached and try to illustrate my idea. It's 
great if you can shepherd and tell if it's the right direction.

If you haven't started a PR on this issue, I can work on it under your shepherd.

But since you have the expertise, if you have already working on, I will learn 
and track your PR.


was (Author: jinxing6...@126.com):
[^CALCITE-3228.patch]

Hi [~Chunwei Lei]
 I try to dig and learn from this issue;

*Project::getMapping* try to return an inverse-surjection mapping. 
*SurjectionWithInverse* is defined as below:
{code:java}
/**
 * A surjection with inverse has precisely one source for each target.
 * (Whereas a general surjection has at least one source for each target.)
 * Every source has at most one target.
 */
{code}
That's why the test you provided throws
{code:java}
...
Caused by: java.lang.IllegalArgumentException: source #0 is already mapped to 
target #1
 at 
org.apache.calcite.util.mapping.Mappings$SurjectionWithInverse.set(Mappings.java:1326)
 at org.apache.calcite.rel.core.Project.getMapping(Project.java:285)
 at org.apache.calcite.rel.core.Project.getMapping(Project.java:256)
 at 
org.apache.calcite.rel.rules.ProjectTableScanRule.apply(ProjectTableScanRule.java:107)
 at 
org.apache.calcite.rel.rules.ProjectTableScanRule$2.onMatch(ProjectTableScanRule.java:83)
 at 
org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:208)
...
{code}
I have two ideas and it's great if you can confirm and rectify when you have 
time:

1. *ProjectTableScanRule* should not be limited by the Project::getMapping. I 
mean ProjectTableScanRule can handle to create a *BindableTableScan* as long as 
*project.getProjects().stream().allMatch(p -> p instanceof RexInputRef) == 
true,* and no need to be limited by a inverse-surjection mapping.

2. The definition of *Project::getMapping* is as below
{code:java}
/**
 * Returns a mapping of a set of project expressions.
 *
 * The mapping is an inverse surjection.
 * Every target has a source field, but
 * a source field may appear as zero, one, or more target fields.
 * Thus you can safely call
 * {@link 
org.apache.calcite.util.mapping.Mappings.TargetMapping#getTarget(int)}.
 *
{code}
But in my understanding, it's not safe to call *getTarget(int)* but safe to 
call *getSource(int);*

I try to file a fix for this issue and illustrate my idea. It's great if you 
can shepherd and tell if it's the right direction.

If you haven't started a PR on this issue, I can work on it under your shepherd.

But since you have the expertise, if you have already working on, I will learn 
and track your PR.

> Error while applying rule ProjectScanRule: interpreter
> --
>
>   

[jira] [Commented] (CALCITE-3226) RelBuilder doesn't keep the alias when scan from an expanded view

[jin xing (JIRA)]

[ 
https://issues.apache.org/jira/browse/CALCITE-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900713#comment-16900713
 ] 

jin xing commented on CALCITE-3226:
---

Thanks a lot for looking into this and sorry for late reply (actually I was 
tracking update from Github but missed messages from JIRA);

I added a test for {{TableScan}}. But the {{TableScan}} is not from an expanded 
view but from a normal table.

I tried  the view {{“select * from t”}}, but turns out that it's always a 
Project+Scan 
(https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L3941
 ) , it's forced to create a {{Project}} no matter it's trival or not. As far 
as I can understand the code, {{convertSelectList}} will always return a 
{{Project}} when the child is a {{TableScan}}

> RelBuilder doesn't keep the alias when scan from an expanded view
> -
>
> Key: CALCITE-3226
> URL: https://issues.apache.org/jira/browse/CALCITE-3226
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Reporter: jin xing
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> In current implementation of {{RelBuilder::scan}} 
> ([https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L1048]),
>  the alias can be derived and recorded into {{Frame}} only when the 
> {{RelNode}} is a {{TableScan}} 
> ([https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L2754]).
>  But when {{RelBuilder::scan}} from an expanded view, the node is not a 
> {{TableScan}} and the alias is not kept.  Below test failed - we cannot 
> reference a field by alias "MYVIEW".
> {code:java}
>   @Test public void testExpandViewShouldKeepAlias() throws SQLException {
> try (Connection connection = 
> DriverManager.getConnection("jdbc:calcite:")) {
>   final Frameworks.ConfigBuilder configBuilder =
>   expandingConfig(connection);
>   final RelOptTable.ViewExpander viewExpander =
>   (RelOptTable.ViewExpander) 
> Frameworks.getPlanner(configBuilder.build());
>   final RelFactories.TableScanFactory tableScanFactory =
>   RelFactories.expandingScanFactory(viewExpander,
>   RelFactories.DEFAULT_TABLE_SCAN_FACTORY);
>   configBuilder.context(Contexts.of(tableScanFactory));
>   final RelBuilder builder = RelBuilder.create(configBuilder.build());
>   RelNode node =
>   builder.scan("MYVIEW")
>   .project(
>   builder.field(1, "MYVIEW", "EMPNO"),   // Exception 
> thrown from here.
>   builder.field(1, "MYVIEW", "ENAME"))
>   .build();
>   String expected =
>   "LogicalProject(EMPNO=[$0], ENAME=[$1])\n"
>   + "  LogicalFilter(condition=[=(1, 1)])\n"
>   + "LogicalTableScan(table=[[scott, EMP]])\n";
>   assertThat(node, hasTree(expected));
> }
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (CALCITE-3239) Calc#accept(RexShuttle shuttle) does not update rowType

[jin xing (JIRA)]

jin xing created CALCITE-3239:
-

 Summary: Calc#accept(RexShuttle shuttle) does not update rowType
 Key: CALCITE-3239
 URL: https://issues.apache.org/jira/browse/CALCITE-3239
 Project: Calcite
  Issue Type: Bug
  Components: core
Reporter: jin xing


In current code, {{Calc#accept(RexShuttle shuttle)}} keep the original 
{{rowType}}, even when the expression list is modified by the shuttle. 
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/rel/core/Calc.java#L175
 
It might be wrong and a new {{rowType}} should be derived.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Assigned] (CALCITE-3239) Calc#accept(RexShuttle shuttle) does not update rowType

[jin xing (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing reassigned CALCITE-3239:
-

Assignee: jin xing

> Calc#accept(RexShuttle shuttle) does not update rowType
> ---
>
> Key: CALCITE-3239
> URL: https://issues.apache.org/jira/browse/CALCITE-3239
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Reporter: jin xing
>Assignee: jin xing
>Priority: Major
>
> In current code, {{Calc#accept(RexShuttle shuttle)}} keep the original 
> {{rowType}}, even when the expression list is modified by the shuttle. 
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/rel/core/Calc.java#L175
>  
> It might be wrong and a new {{rowType}} should be derived.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Assigned] (CALCITE-3223) Non-RexInputRef may fails the matching of FilterToProjectUnifyRule during 'invert' by mistake.

[jin xing (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing reassigned CALCITE-3223:
-

Assignee: jin xing

> Non-RexInputRef may fails the matching of FilterToProjectUnifyRule during 
> 'invert' by mistake.
> --
>
> Key: CALCITE-3223
> URL: https://issues.apache.org/jira/browse/CALCITE-3223
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Reporter: jin xing
>Assignee: jin xing
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> In current code of 
> {{FilterToProjectUnifyRule::invert}}(https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/plan/SubstitutionVisitor.java#L1124),
>   the implementation 
> 1. Fails the matching when there is Non-RexInputRef  in the projects
> 2. Didn't check if all items of {{exprList}} has already been set correctly.
> As a result below tests  fails.
> {code:java}
>   @Test public void testFilterToProject0() {
> String union =
> "select * from \"emps\" where \"empid\" > 300\n"
> + "union all select * from \"emps\" where \"empid\" < 200";
> String mv = "select *, \"empid\" * 2 from (" + union + ")";
> String query = "select * from (" + union + ") where (\"empid\" * 2) > 3";
> checkMaterialize(mv, query);
>   }
>   @Test public void testFilterToProject1() {
> String agg =
> "select \"deptno\", count(*) as \"c\", sum(\"salary\") as \"s\"\n"
> + "from \"emps\" group by \"deptno\"";
> String mv = "select \"c\", \"s\", \"s\" from (" + agg + ")";
> String query = "select * from (" + agg + ") where (\"s\" * 0.8) > 1";
> checkNoMaterialize(mv, query, HR_FKUK_MODEL);
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Assigned] (CALCITE-3214) Add UnionToUnionRule for materialization matching

[jin xing (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing reassigned CALCITE-3214:
-

Assignee: jin xing

> Add UnionToUnionRule for materialization matching
> -
>
> Key: CALCITE-3214
> URL: https://issues.apache.org/jira/browse/CALCITE-3214
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Reporter: jin xing
>Assignee: jin xing
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Below materialization matching fails now
> {code:java}
>   @Test public void testDEV() {
> String sql0 = "select * from \"emps\" where \"empid\" < 300";
> String sql1 = "select * from \"emps\" where \"empid\" > 200";
> checkMaterialize(sql0 + " union all " + sql1, sql1 + " union all " + 
> sql0);
>   }
> {code}
> This issue proposes to add a rule for union matching



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Assigned] (CALCITE-3203) When matching materializations, match Project with child of Aggregate

[jin xing (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing reassigned CALCITE-3203:
-

Assignee: jin xing

> When matching materializations, match Project with child of Aggregate
> -
>
> Key: CALCITE-3203
> URL: https://issues.apache.org/jira/browse/CALCITE-3203
> Project: Calcite
>  Issue Type: Improvement
>  Components: core
>Reporter: jin xing
>Assignee: jin xing
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> In current code, {{SubstitutionVisitor}} & 
> {{MaterializedViewSubstitutionVisitor}} fail to support below matching:
> {code:java}
>query:   Project(projects: [$0, *(2, $1)])
>   Aggregate(groupSet: {0}, groupSets: [{0}], calls: [SUM($1)])
>  Scan(table: [hr, emps])
>target:  Project(projects: [$0, *(2, $1), *(2, $2)])
>   Aggregate(groupSet: {0}, groupSets: [{0}], calls: [SUM($1), 
> COUNT()])
>  Scan(table: [hr, emps])
> {code}
> And below test fails
> {code:java}
> // MaterializationTest.java
>   @Test public void testAggregate() {
> checkMaterialize(
>   "select \"deptno\", count(1), 2 * sum(\"empid\") from "
> + "(select * from \"emps\" union all select * from \"emps\")"
> + "group by \"deptno\"",
>   "select \"deptno\", 2 * sum(\"empid\") from "
> + "(select * from \"emps\" union all select * from \"emps\")"
> + "group by \"deptno\"");
>   }
> {code}
> The reason is that {{Project&Aggregate}} are not taken into consideration at 
> the same time in current matching rules.
> It might make sense to create a rule of 
> {{ProjectOnAggregateToProjectOnAggregateUnifyRule}} to handle such case.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Assigned] (CALCITE-3167) Remove overriding equals and hashCode methods in EnumerableTableScan

[jin xing (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing reassigned CALCITE-3167:
-

Assignee: jin xing

> Remove overriding equals and hashCode methods in EnumerableTableScan
> 
>
> Key: CALCITE-3167
> URL: https://issues.apache.org/jira/browse/CALCITE-3167
> Project: Calcite
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 1.19.0
>Reporter: jin xing
>Assignee: jin xing
>Priority: Minor
>  Labels: pull-request-available
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> In current code of {{EnumerableTableScan}}, methods of equals and hashCode 
> are overridden for matching of {{EnumerableTableScan}}s.
> While after optimizing with the same HEP planner, {{EnumerableTableScan}}s 
> from two plans but with the same digest will the share the same Java object. 
> See 
> [RelOptMaterializations|https://github.com/apache/calcite/blob/adf4cc4dc5cdb9f5e49c85d10f46a2fdcd831ccf/core/src/main/java/org/apache/calcite/plan/RelOptMaterializations.java#L192].
> I think it's ok to remove the redundant overriding methods in 
> {{EnumerableTableScan}}.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Commented] (CALCITE-3167) Remove overriding equals and hashCode methods in EnumerableTableScan

[jin xing (JIRA)]

[ 
https://issues.apache.org/jira/browse/CALCITE-3167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16906823#comment-16906823
 ] 

jin xing commented on CALCITE-3167:
---

[~julianhyde]
Gentle ping ~
In current change, I made equals/hashCode methods final in {{AbstractRelNode}} 
and passed all the tests.
And I'm not sure if [~amargoor] has some other ideas or comments on this one.
Could you please leave more comments? and it's great if there's chance this 
patch can be merged.

> Remove overriding equals and hashCode methods in EnumerableTableScan
> 
>
> Key: CALCITE-3167
> URL: https://issues.apache.org/jira/browse/CALCITE-3167
> Project: Calcite
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 1.19.0
>Reporter: jin xing
>Assignee: jin xing
>Priority: Minor
>  Labels: pull-request-available
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> In current code of {{EnumerableTableScan}}, methods of equals and hashCode 
> are overridden for matching of {{EnumerableTableScan}}s.
> While after optimizing with the same HEP planner, {{EnumerableTableScan}}s 
> from two plans but with the same digest will the share the same Java object. 
> See 
> [RelOptMaterializations|https://github.com/apache/calcite/blob/adf4cc4dc5cdb9f5e49c85d10f46a2fdcd831ccf/core/src/main/java/org/apache/calcite/plan/RelOptMaterializations.java#L192].
> I think it's ok to remove the redundant overriding methods in 
> {{EnumerableTableScan}}.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Commented] (CALCITE-3167) Remove overriding equals and hashCode methods in EnumerableTableScan

[jin xing (JIRA)]

[ 
https://issues.apache.org/jira/browse/CALCITE-3167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16906923#comment-16906923
 ] 

jin xing commented on CALCITE-3167:
---

[~julianhyde]
Thanks a lot for approve.
I updated the patch according to your comment and tests passed.
Please take another look when you have time :)

> Remove overriding equals and hashCode methods in EnumerableTableScan
> 
>
> Key: CALCITE-3167
> URL: https://issues.apache.org/jira/browse/CALCITE-3167
> Project: Calcite
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 1.19.0
>Reporter: jin xing
>Assignee: jin xing
>Priority: Minor
>  Labels: pull-request-available
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> In current code of {{EnumerableTableScan}}, methods of equals and hashCode 
> are overridden for matching of {{EnumerableTableScan}}s.
> While after optimizing with the same HEP planner, {{EnumerableTableScan}}s 
> from two plans but with the same digest will the share the same Java object. 
> See 
> [RelOptMaterializations|https://github.com/apache/calcite/blob/adf4cc4dc5cdb9f5e49c85d10f46a2fdcd831ccf/core/src/main/java/org/apache/calcite/plan/RelOptMaterializations.java#L192].
> I think it's ok to remove the redundant overriding methods in 
> {{EnumerableTableScan}}.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (CALCITE-3249) Substitution#getRexShuttle does not consider RexLiteral

[jin xing (JIRA)]

jin xing created CALCITE-3249:
-

 Summary: Substitution#getRexShuttle does not consider RexLiteral
 Key: CALCITE-3249
 URL: https://issues.apache.org/jira/browse/CALCITE-3249
 Project: Calcite
  Issue Type: Improvement
  Components: core
Reporter: jin xing
Assignee: jin xing






--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Updated] (CALCITE-3249) Substitution#getRexShuttle does not consider RexLiteral

[jin xing (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing updated CALCITE-3249:
--
Description: 
Current {{Substitution#getRexShuttle}} does not take {{RexLiteral}} into 
consideration.
Thus below query & mv fails matching:

{code:java}
MV:
select deptno, sum(salary), sum(commission), sum(k)
from
  (select deptno, salary, commission, 100 as k
  from emps)
group by deptno

Query:
select deptno, sum(salary), sum(k)
from
  (select deptno, salary, 100 as k
  from emps)
group by deptno
{code}

The root cause is that {{ProjectToProjectUnifyRule}} compensates a {{Project}} 
which contains {{RexLiteral}}, but {{AggregateOnProjectToAggregateUnifyRule}} 
works only when the {{Project}} in query is a mapping
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/plan/SubstitutionVisitor.java#L1357

  was:
Current {{Substitution#getRexShuttle}} does not take {{RexLiteral}} into 
consideration.
Thus below query & mv fails matching:

{code:java}
MV:
select deptno, sum(salary), sum(commission), sum(k)
from
  (select deptno, salary, commission, 100 as k
  from emps)
group by deptno

Query:
select deptno, sum(salary), sum(k)
from
  (select deptno, salary, 100 as k
  from emps)
group by deptno
{code}

The root cause is that {{ProjectToProjectUnifyRule}} compensate a {{Project}} 
which contains {{RexLiteral}}, but {{AggregateOnProjectToAggregateUnifyRule}} 
works only when the {{Project}} in query is a mapping
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/plan/SubstitutionVisitor.java#L1357


> Substitution#getRexShuttle does not consider RexLiteral
> ---
>
> Key: CALCITE-3249
> URL: https://issues.apache.org/jira/browse/CALCITE-3249
> Project: Calcite
>  Issue Type: Improvement
>  Components: core
>Reporter: jin xing
>Assignee: jin xing
>Priority: Minor
>
> Current {{Substitution#getRexShuttle}} does not take {{RexLiteral}} into 
> consideration.
> Thus below query & mv fails matching:
> {code:java}
> MV:
> select deptno, sum(salary), sum(commission), sum(k)
> from
>   (select deptno, salary, commission, 100 as k
>   from emps)
> group by deptno
> Query:
> select deptno, sum(salary), sum(k)
> from
>   (select deptno, salary, 100 as k
>   from emps)
> group by deptno
> {code}
> The root cause is that {{ProjectToProjectUnifyRule}} compensates a 
> {{Project}} which contains {{RexLiteral}}, but 
> {{AggregateOnProjectToAggregateUnifyRule}} works only when the {{Project}} in 
> query is a mapping
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/plan/SubstitutionVisitor.java#L1357



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Updated] (CALCITE-3249) Substitution#getRexShuttle does not consider RexLiteral

[jin xing (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing updated CALCITE-3249:
--
Description: 
Current {{Substitution#getRexShuttle}} does not take {{RexLiteral}} into 
consideration.
Thus below query & mv fails matching:

{code:java}
MV:
select deptno, sum(salary), sum(commission), sum(k)
from
  (select deptno, salary, commission, 100 as k
  from emps)
group by deptno

Query:
select deptno, sum(salary), sum(k)
from
  (select deptno, salary, 100 as k
  from emps)
group by deptno
{code}

The root cause is that {{ProjectToProjectUnifyRule}} compensate a {{Project}} 
which contains {{RexLiteral}}, but {{AggregateOnProjectToAggregateUnifyRule}} 
works only when the {{Project}} in query is a mapping
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/plan/SubstitutionVisitor.java#L1357

> Substitution#getRexShuttle does not consider RexLiteral
> ---
>
> Key: CALCITE-3249
> URL: https://issues.apache.org/jira/browse/CALCITE-3249
> Project: Calcite
>  Issue Type: Improvement
>  Components: core
>Reporter: jin xing
>Assignee: jin xing
>Priority: Minor
>
> Current {{Substitution#getRexShuttle}} does not take {{RexLiteral}} into 
> consideration.
> Thus below query & mv fails matching:
> {code:java}
> MV:
> select deptno, sum(salary), sum(commission), sum(k)
> from
>   (select deptno, salary, commission, 100 as k
>   from emps)
> group by deptno
> Query:
> select deptno, sum(salary), sum(k)
> from
>   (select deptno, salary, 100 as k
>   from emps)
> group by deptno
> {code}
> The root cause is that {{ProjectToProjectUnifyRule}} compensate a {{Project}} 
> which contains {{RexLiteral}}, but {{AggregateOnProjectToAggregateUnifyRule}} 
> works only when the {{Project}} in query is a mapping
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/plan/SubstitutionVisitor.java#L1357



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Assigned] (CALCITE-3251) BinaryExpression evaluate method support Long type.

[jin xing (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing reassigned CALCITE-3251:
-

Assignee: jin xing

> BinaryExpression evaluate method support Long type.
> ---
>
> Key: CALCITE-3251
> URL: https://issues.apache.org/jira/browse/CALCITE-3251
> Project: Calcite
>  Issue Type: Wish
>Reporter: xzh_dz
>Assignee: jin xing
>Priority: Minor
>  Labels: pull-request-available
>  Time Spent: 2h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Assigned] (CALCITE-3251) BinaryExpression evaluate method support Long type.

[jin xing (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing reassigned CALCITE-3251:
-

Assignee: (was: jin xing)

> BinaryExpression evaluate method support Long type.
> ---
>
> Key: CALCITE-3251
> URL: https://issues.apache.org/jira/browse/CALCITE-3251
> Project: Calcite
>  Issue Type: Wish
>Reporter: xzh_dz
>Priority: Minor
>  Labels: pull-request-available
>  Time Spent: 2h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Updated] (CALCITE-3251) BinaryExpression evaluate method support Long type.

[jin xing (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing updated CALCITE-3251:
--
Issue Type: Bug  (was: Wish)

> BinaryExpression evaluate method support Long type.
> ---
>
> Key: CALCITE-3251
> URL: https://issues.apache.org/jira/browse/CALCITE-3251
> Project: Calcite
>  Issue Type: Bug
>Reporter: xzh_dz
>Priority: Minor
>  Labels: pull-request-available
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (CALCITE-3256) Add ProjectOnProjectToProjectUnifyRule for materialization matching.

[jin xing (JIRA)]

jin xing created CALCITE-3256:
-

 Summary: Add ProjectOnProjectToProjectUnifyRule for 
materialization matching.
 Key: CALCITE-3256
 URL: https://issues.apache.org/jira/browse/CALCITE-3256
 Project: Calcite
  Issue Type: Improvement
  Components: core
Reporter: jin xing
Assignee: jin xing


In current implementation of rules in SubstitutionVisitor.java & 
MaterializedViewSubstitutionVisitor.java, it's quite common to add a 
compensating Project on top of child node of target(MV-rel) during matching. 
But afterwards the next round matching should be able to handle such a 
compensated Project and match upward along the plan tree. Otherwise we fail the 
matching. After all, the goal of matching is to transform the query and let a 
complete 'target' show up in the transformed query plan.

I found cases where the compensated Project cannot be properly handled.

{code:java}
MV:
select deptno, sum(salary) + 2, sum(commission)
from emps
group by deptno

Query:
select deptno, sum(salary) + 2
from emps
group by deptno
{code}

After matching of the Aggregates, a compensating Project is added, but 
afterwards matching fails to handle it.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Commented] (CALCITE-3226) RelBuilder doesn't keep the alias when scan from an expanded view

[jin xing (JIRA)]

[ 
https://issues.apache.org/jira/browse/CALCITE-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16909567#comment-16909567
 ] 

jin xing commented on CALCITE-3226:
---

Thanks a lot for clarify [~rubenql]
I think [~julianhyde] expects to add a test case where the view of “select * 
from t” expands as a Scan, but I tried and turned out that it always expands as 
a Project+Scan.
For a common table scan, the alias would have already been correctly set and I 
added a test case 
testProjectWithAliasFromScan(https://github.com/apache/calcite/pull/1351/files#diff-9f6a8f13d9ad70d7336340bf6b2e7b71R702)

Thanks a lot for review this issue [~julianhyde][~rubenql] and sorry for my 
late reply

> RelBuilder doesn't keep the alias when scan from an expanded view
> -
>
> Key: CALCITE-3226
> URL: https://issues.apache.org/jira/browse/CALCITE-3226
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Reporter: jin xing
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.21.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> In current implementation of {{RelBuilder::scan}} 
> ([https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L1048]),
>  the alias can be derived and recorded into {{Frame}} only when the 
> {{RelNode}} is a {{TableScan}} 
> ([https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L2754]).
>  But when {{RelBuilder::scan}} from an expanded view, the node is not a 
> {{TableScan}} and the alias is not kept.  Below test failed - we cannot 
> reference a field by alias "MYVIEW".
> {code:java}
>   @Test public void testExpandViewShouldKeepAlias() throws SQLException {
> try (Connection connection = 
> DriverManager.getConnection("jdbc:calcite:")) {
>   final Frameworks.ConfigBuilder configBuilder =
>   expandingConfig(connection);
>   final RelOptTable.ViewExpander viewExpander =
>   (RelOptTable.ViewExpander) 
> Frameworks.getPlanner(configBuilder.build());
>   final RelFactories.TableScanFactory tableScanFactory =
>   RelFactories.expandingScanFactory(viewExpander,
>   RelFactories.DEFAULT_TABLE_SCAN_FACTORY);
>   configBuilder.context(Contexts.of(tableScanFactory));
>   final RelBuilder builder = RelBuilder.create(configBuilder.build());
>   RelNode node =
>   builder.scan("MYVIEW")
>   .project(
>   builder.field(1, "MYVIEW", "EMPNO"),   // Exception 
> thrown from here.
>   builder.field(1, "MYVIEW", "ENAME"))
>   .build();
>   String expected =
>   "LogicalProject(EMPNO=[$0], ENAME=[$1])\n"
>   + "  LogicalFilter(condition=[=(1, 1)])\n"
>   + "LogicalTableScan(table=[[scott, EMP]])\n";
>   assertThat(node, hasTree(expected));
> }
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Comment Edited] (CALCITE-3226) RelBuilder doesn't keep the alias when scan from an expanded view

[jin xing (JIRA)]

[ 
https://issues.apache.org/jira/browse/CALCITE-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16909567#comment-16909567
 ] 

jin xing edited comment on CALCITE-3226 at 8/17/19 4:06 AM:


Thanks a lot for clarify [~rubenql]
I think [~julianhyde] expects to add a test case where the view of “select * 
from t” expands as a Scan, but I tried and turned out that it always expands as 
a Project+Scan.
For a common table scan, the alias would have already been correctly set and I 
added a test case 
testProjectWithAliasFromScan(https://github.com/apache/calcite/pull/1351/files#diff-9f6a8f13d9ad70d7336340bf6b2e7b71R702)

Thanks a lot for review this issue [~julianhyde] [~rubenql] and sorry for my 
late reply


was (Author: jinxing6...@126.com):
Thanks a lot for clarify [~rubenql]
I think [~julianhyde] expects to add a test case where the view of “select * 
from t” expands as a Scan, but I tried and turned out that it always expands as 
a Project+Scan.
For a common table scan, the alias would have already been correctly set and I 
added a test case 
testProjectWithAliasFromScan(https://github.com/apache/calcite/pull/1351/files#diff-9f6a8f13d9ad70d7336340bf6b2e7b71R702)

Thanks a lot for review this issue [~julianhyde][~rubenql] and sorry for my 
late reply

> RelBuilder doesn't keep the alias when scan from an expanded view
> -
>
> Key: CALCITE-3226
> URL: https://issues.apache.org/jira/browse/CALCITE-3226
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Reporter: jin xing
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.21.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> In current implementation of {{RelBuilder::scan}} 
> ([https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L1048]),
>  the alias can be derived and recorded into {{Frame}} only when the 
> {{RelNode}} is a {{TableScan}} 
> ([https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L2754]).
>  But when {{RelBuilder::scan}} from an expanded view, the node is not a 
> {{TableScan}} and the alias is not kept.  Below test failed - we cannot 
> reference a field by alias "MYVIEW".
> {code:java}
>   @Test public void testExpandViewShouldKeepAlias() throws SQLException {
> try (Connection connection = 
> DriverManager.getConnection("jdbc:calcite:")) {
>   final Frameworks.ConfigBuilder configBuilder =
>   expandingConfig(connection);
>   final RelOptTable.ViewExpander viewExpander =
>   (RelOptTable.ViewExpander) 
> Frameworks.getPlanner(configBuilder.build());
>   final RelFactories.TableScanFactory tableScanFactory =
>   RelFactories.expandingScanFactory(viewExpander,
>   RelFactories.DEFAULT_TABLE_SCAN_FACTORY);
>   configBuilder.context(Contexts.of(tableScanFactory));
>   final RelBuilder builder = RelBuilder.create(configBuilder.build());
>   RelNode node =
>   builder.scan("MYVIEW")
>   .project(
>   builder.field(1, "MYVIEW", "EMPNO"),   // Exception 
> thrown from here.
>   builder.field(1, "MYVIEW", "ENAME"))
>   .build();
>   String expected =
>   "LogicalProject(EMPNO=[$0], ENAME=[$1])\n"
>   + "  LogicalFilter(condition=[=(1, 1)])\n"
>   + "LogicalTableScan(table=[[scott, EMP]])\n";
>   assertThat(node, hasTree(expected));
> }
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Assigned] (CALCITE-3226) RelBuilder doesn't keep the alias when scan from an expanded view

[jin xing (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing reassigned CALCITE-3226:
-

Assignee: jin xing

> RelBuilder doesn't keep the alias when scan from an expanded view
> -
>
> Key: CALCITE-3226
> URL: https://issues.apache.org/jira/browse/CALCITE-3226
> Project: Calcite
>  Issue Type: Bug
>  Components: core
>Reporter: jin xing
>Assignee: jin xing
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.21.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> In current implementation of {{RelBuilder::scan}} 
> ([https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L1048]),
>  the alias can be derived and recorded into {{Frame}} only when the 
> {{RelNode}} is a {{TableScan}} 
> ([https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L2754]).
>  But when {{RelBuilder::scan}} from an expanded view, the node is not a 
> {{TableScan}} and the alias is not kept.  Below test failed - we cannot 
> reference a field by alias "MYVIEW".
> {code:java}
>   @Test public void testExpandViewShouldKeepAlias() throws SQLException {
> try (Connection connection = 
> DriverManager.getConnection("jdbc:calcite:")) {
>   final Frameworks.ConfigBuilder configBuilder =
>   expandingConfig(connection);
>   final RelOptTable.ViewExpander viewExpander =
>   (RelOptTable.ViewExpander) 
> Frameworks.getPlanner(configBuilder.build());
>   final RelFactories.TableScanFactory tableScanFactory =
>   RelFactories.expandingScanFactory(viewExpander,
>   RelFactories.DEFAULT_TABLE_SCAN_FACTORY);
>   configBuilder.context(Contexts.of(tableScanFactory));
>   final RelBuilder builder = RelBuilder.create(configBuilder.build());
>   RelNode node =
>   builder.scan("MYVIEW")
>   .project(
>   builder.field(1, "MYVIEW", "EMPNO"),   // Exception 
> thrown from here.
>   builder.field(1, "MYVIEW", "ENAME"))
>   .build();
>   String expected =
>   "LogicalProject(EMPNO=[$0], ENAME=[$1])\n"
>   + "  LogicalFilter(condition=[=(1, 1)])\n"
>   + "LogicalTableScan(table=[[scott, EMP]])\n";
>   assertThat(node, hasTree(expected));
> }
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (CALCITE-3262) Refine doc of SubstitutionVisitor.java

[jin xing (JIRA)]

jin xing created CALCITE-3262:
-

 Summary: Refine doc of SubstitutionVisitor.java
 Key: CALCITE-3262
 URL: https://issues.apache.org/jira/browse/CALCITE-3262
 Project: Calcite
  Issue Type: Improvement
  Components: core
Reporter: jin xing
Assignee: jin xing


Current doc of {{SubstitutionVisitor.java}} says the supported core relational 
operators are {{@link org.apache.calcite.rel.logical.LogicalTableScan}}, and so 
on.
But with {{convertTableAccess=true}} 
(https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L5636),
 it's a {{EnumerableTableScan}} below {{MutableScan}}, which is inconsistent 
with the doc. 
And what's more, {{MutableRels}} and {{SubstitutionVisitor}} supporting scope 
doesn't limit to be org.apache.calcite.rel.logical.LogicalXXX.
So I think it might make sense to update/refine the doc  to say that the 
supported core relational operators are 

{code:java}
 * {@link org.apache.calcite.rel.core.TableScan},
 * {@link org.apache.calcite.rel.core.Filter},
 * {@link org.apache.calcite.rel.core.Project},
 * {@link org.apache.calcite.rel.core.Join},
 * {@link org.apache.calcite.rel.core.Union},
 * {@link org.apache.calcite.rel.core.Aggregate}.
{code}




--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Updated] (CALCITE-3262) Refine doc of SubstitutionVisitor.java

[jin xing (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing updated CALCITE-3262:
--
Description: 
Current doc of {{SubstitutionVisitor.java}} says the supported core relational 
operators are {{@link org.apache.calcite.rel.logical.LogicalTableScan}}, and so 
on.
But with {{convertTableAccess=true}} 
(https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L5636),
 it's a {{EnumerableTableScan}} under {{MutableScan}}, which is inconsistent 
with the doc. 
And what's more, {{MutableRels}} and {{SubstitutionVisitor}} supporting scope 
doesn't limit to be org.apache.calcite.rel.logical.LogicalXXX.
So I think it might make sense to update/refine the doc  to say that the 
supported core relational operators are 

{code:java}
 * {@link org.apache.calcite.rel.core.TableScan},
 * {@link org.apache.calcite.rel.core.Filter},
 * {@link org.apache.calcite.rel.core.Project},
 * {@link org.apache.calcite.rel.core.Join},
 * {@link org.apache.calcite.rel.core.Union},
 * {@link org.apache.calcite.rel.core.Aggregate}.
{code}


  was:
Current doc of {{SubstitutionVisitor.java}} says the supported core relational 
operators are {{@link org.apache.calcite.rel.logical.LogicalTableScan}}, and so 
on.
But with {{convertTableAccess=true}} 
(https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L5636),
 it's a {{EnumerableTableScan}} below {{MutableScan}}, which is inconsistent 
with the doc. 
And what's more, {{MutableRels}} and {{SubstitutionVisitor}} supporting scope 
doesn't limit to be org.apache.calcite.rel.logical.LogicalXXX.
So I think it might make sense to update/refine the doc  to say that the 
supported core relational operators are 

{code:java}
 * {@link org.apache.calcite.rel.core.TableScan},
 * {@link org.apache.calcite.rel.core.Filter},
 * {@link org.apache.calcite.rel.core.Project},
 * {@link org.apache.calcite.rel.core.Join},
 * {@link org.apache.calcite.rel.core.Union},
 * {@link org.apache.calcite.rel.core.Aggregate}.
{code}



> Refine doc of SubstitutionVisitor.java
> --
>
> Key: CALCITE-3262
> URL: https://issues.apache.org/jira/browse/CALCITE-3262
> Project: Calcite
>  Issue Type: Improvement
>  Components: core
>Reporter: jin xing
>Assignee: jin xing
>Priority: Minor
>
> Current doc of {{SubstitutionVisitor.java}} says the supported core 
> relational operators are {{@link 
> org.apache.calcite.rel.logical.LogicalTableScan}}, and so on.
> But with {{convertTableAccess=true}} 
> (https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L5636),
>  it's a {{EnumerableTableScan}} under {{MutableScan}}, which is inconsistent 
> with the doc. 
> And what's more, {{MutableRels}} and {{SubstitutionVisitor}} supporting scope 
> doesn't limit to be org.apache.calcite.rel.logical.LogicalXXX.
> So I think it might make sense to update/refine the doc  to say that the 
> supported core relational operators are 
> {code:java}
>  * {@link org.apache.calcite.rel.core.TableScan},
>  * {@link org.apache.calcite.rel.core.Filter},
>  * {@link org.apache.calcite.rel.core.Project},
>  * {@link org.apache.calcite.rel.core.Join},
>  * {@link org.apache.calcite.rel.core.Union},
>  * {@link org.apache.calcite.rel.core.Aggregate}.
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Commented] (CALCITE-3167) Remove overriding equals and hashCode methods in EnumerableTableScan

[jin xing (Jira)]

[ 
https://issues.apache.org/jira/browse/CALCITE-3167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16911878#comment-16911878
 ] 

jin xing commented on CALCITE-3167:
---

Fixed in 
https://github.com/apache/calcite/commit/3f550c3f28d75d0f75b5815a22395dbecf992f03

> Remove overriding equals and hashCode methods in EnumerableTableScan
> 
>
> Key: CALCITE-3167
> URL: https://issues.apache.org/jira/browse/CALCITE-3167
> Project: Calcite
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 1.19.0
>Reporter: jin xing
>Assignee: Julian Hyde
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 1.21.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> In current code of {{EnumerableTableScan}}, methods of equals and hashCode 
> are overridden for matching of {{EnumerableTableScan}}s.
> While after optimizing with the same HEP planner, {{EnumerableTableScan}}s 
> from two plans but with the same digest will the share the same Java object. 
> See 
> [RelOptMaterializations|https://github.com/apache/calcite/blob/adf4cc4dc5cdb9f5e49c85d10f46a2fdcd831ccf/core/src/main/java/org/apache/calcite/plan/RelOptMaterializations.java#L192].
> I think it's ok to remove the redundant overriding methods in 
> {{EnumerableTableScan}}.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (CALCITE-3167) Remove overriding equals and hashCode methods in EnumerableTableScan

[jin xing (Jira)]

[ 
https://issues.apache.org/jira/browse/CALCITE-3167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16911877#comment-16911877
 ] 

jin xing commented on CALCITE-3167:
---

[~julianhyde]
Thanks a lot for merging !

> Remove overriding equals and hashCode methods in EnumerableTableScan
> 
>
> Key: CALCITE-3167
> URL: https://issues.apache.org/jira/browse/CALCITE-3167
> Project: Calcite
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 1.19.0
>Reporter: jin xing
>Assignee: Julian Hyde
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 1.21.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> In current code of {{EnumerableTableScan}}, methods of equals and hashCode 
> are overridden for matching of {{EnumerableTableScan}}s.
> While after optimizing with the same HEP planner, {{EnumerableTableScan}}s 
> from two plans but with the same digest will the share the same Java object. 
> See 
> [RelOptMaterializations|https://github.com/apache/calcite/blob/adf4cc4dc5cdb9f5e49c85d10f46a2fdcd831ccf/core/src/main/java/org/apache/calcite/plan/RelOptMaterializations.java#L192].
> I think it's ok to remove the redundant overriding methods in 
> {{EnumerableTableScan}}.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Resolved] (CALCITE-3167) Remove overriding equals and hashCode methods in EnumerableTableScan

[jin xing (Jira)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing resolved CALCITE-3167.
---
Resolution: Fixed

> Remove overriding equals and hashCode methods in EnumerableTableScan
> 
>
> Key: CALCITE-3167
> URL: https://issues.apache.org/jira/browse/CALCITE-3167
> Project: Calcite
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 1.19.0
>Reporter: jin xing
>Assignee: Julian Hyde
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 1.21.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> In current code of {{EnumerableTableScan}}, methods of equals and hashCode 
> are overridden for matching of {{EnumerableTableScan}}s.
> While after optimizing with the same HEP planner, {{EnumerableTableScan}}s 
> from two plans but with the same digest will the share the same Java object. 
> See 
> [RelOptMaterializations|https://github.com/apache/calcite/blob/adf4cc4dc5cdb9f5e49c85d10f46a2fdcd831ccf/core/src/main/java/org/apache/calcite/plan/RelOptMaterializations.java#L192].
> I think it's ok to remove the redundant overriding methods in 
> {{EnumerableTableScan}}.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (CALCITE-3273) Materialization matching rules that consider compensating MutableProject

[jin xing (Jira)]

jin xing created CALCITE-3273:
-

 Summary: Materialization matching rules that consider compensating 
MutableProject
 Key: CALCITE-3273
 URL: https://issues.apache.org/jira/browse/CALCITE-3273
 Project: Calcite
  Issue Type: Improvement
  Components: core
Reporter: jin xing






--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (CALCITE-3273) Materialization matching rules that consider compensating MutableProject

[jin xing (Jira)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing updated CALCITE-3273:
--
Description: 
In current implementation of rules in SubstitutionVisitor.java & 
MaterializedViewSubstitutionVisitor.java, it's quite common to add a 
compensating Project on top of child node of target(MV-rel) during matching. 
But afterwards the next round matching should be able to handle such a 
compensated Project and match upward along the plan tree. Otherwise we fail the 
matching. After all, the goal of matching is to transform the query and let a 
complete 'target' show up in the transformed query plan.

I opened this as a parent JIRA to add MV matching rules that consider 
compensating MutableProject.

> Materialization matching rules that consider compensating MutableProject
> 
>
> Key: CALCITE-3273
> URL: https://issues.apache.org/jira/browse/CALCITE-3273
> Project: Calcite
>  Issue Type: Improvement
>  Components: core
>Reporter: jin xing
>Priority: Major
>
> In current implementation of rules in SubstitutionVisitor.java & 
> MaterializedViewSubstitutionVisitor.java, it's quite common to add a 
> compensating Project on top of child node of target(MV-rel) during matching. 
> But afterwards the next round matching should be able to handle such a 
> compensated Project and match upward along the plan tree. Otherwise we fail 
> the matching. After all, the goal of matching is to transform the query and 
> let a complete 'target' show up in the transformed query plan.
> I opened this as a parent JIRA to add MV matching rules that consider 
> compensating MutableProject.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (CALCITE-3256) Add ProjectOnProjectToProjectUnifyRule for materialization matching.

[jin xing (Jira)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing updated CALCITE-3256:
--
Parent: CALCITE-3273
Issue Type: Sub-task  (was: Improvement)

> Add ProjectOnProjectToProjectUnifyRule for materialization matching.
> 
>
> Key: CALCITE-3256
> URL: https://issues.apache.org/jira/browse/CALCITE-3256
> Project: Calcite
>  Issue Type: Sub-task
>  Components: core
>Reporter: jin xing
>Assignee: jin xing
>Priority: Minor
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> In current implementation of rules in SubstitutionVisitor.java & 
> MaterializedViewSubstitutionVisitor.java, it's quite common to add a 
> compensating Project on top of child node of target(MV-rel) during matching. 
> But afterwards the next round matching should be able to handle such a 
> compensated Project and match upward along the plan tree. Otherwise we fail 
> the matching. After all, the goal of matching is to transform the query and 
> let a complete 'target' show up in the transformed query plan.
> I found cases where the compensated Project cannot be properly handled.
> {code:java}
> MV:
> select deptno, sum(salary) + 2, sum(commission)
> from emps
> group by deptno
> Query:
> select deptno, sum(salary) + 2
> from emps
> group by deptno
> {code}
> After matching of the Aggregates, a compensating Project is added, but 
> afterwards matching fails to handle it.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (CALCITE-3256) Add ProjectOnProjectToProjectUnifyRule for materialization matching.

[jin xing (Jira)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing updated CALCITE-3256:
--
Description: 
This issue proposes to add a rule to match a project
{code:java}
MV:
select deptno, sum(salary) + 2, sum(commission)
from emps
group by deptno

Query:
select deptno, sum(salary) + 2
from emps
group by deptno
{code}

After matching of the Aggregates, a compensating Project is added, but 
afterwards matching fails to handle it.

  was:
In current implementation of rules in SubstitutionVisitor.java & 
MaterializedViewSubstitutionVisitor.java, it's quite common to add a 
compensating Project on top of child node of target(MV-rel) during matching. 
But afterwards the next round matching should be able to handle such a 
compensated Project and match upward along the plan tree. Otherwise we fail the 
matching. After all, the goal of matching is to transform the query and let a 
complete 'target' show up in the transformed query plan.

I found cases where the compensated Project cannot be properly handled.

{code:java}
MV:
select deptno, sum(salary) + 2, sum(commission)
from emps
group by deptno

Query:
select deptno, sum(salary) + 2
from emps
group by deptno
{code}

After matching of the Aggregates, a compensating Project is added, but 
afterwards matching fails to handle it.


> Add ProjectOnProjectToProjectUnifyRule for materialization matching.
> 
>
> Key: CALCITE-3256
> URL: https://issues.apache.org/jira/browse/CALCITE-3256
> Project: Calcite
>  Issue Type: Sub-task
>  Components: core
>Reporter: jin xing
>Assignee: jin xing
>Priority: Minor
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> This issue proposes to add a rule to match a project
> {code:java}
> MV:
> select deptno, sum(salary) + 2, sum(commission)
> from emps
> group by deptno
> Query:
> select deptno, sum(salary) + 2
> from emps
> group by deptno
> {code}
> After matching of the Aggregates, a compensating Project is added, but 
> afterwards matching fails to handle it.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (CALCITE-3256) Add ProjectOnProjectToProjectUnifyRule for materialization matching.

[jin xing (Jira)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing updated CALCITE-3256:
--
Description: 
In current code, below matching fails:

{code:java}
MV:
select deptno, sum(salary) + 2, sum(commission)
from emps
group by deptno

Query:
select deptno, sum(salary) + 2
from emps
group by deptno
{code}

The reason is that --  after matching of the Aggregates, a compensating Project 
is added, but afterwards matching fails to handle it.

This issue proposes to add a rule that match when query and target are both 
Project and query has a compensating Project as a child. After this issue below 
case can be handled:
{code:java}
Query:
Project(projects: [$0, +($1, 2)])
  Project(projects: [$1, $3, $4])
Rel-A
Target:  
Project(projects: [$1, +($3, 2)])
  Rel-A
{code}


  was:
In current code, below matching fails:

{code:java}
MV:
select deptno, sum(salary) + 2, sum(commission)
from emps
group by deptno

Query:
select deptno, sum(salary) + 2
from emps
group by deptno
{code}

The reason is that --  after matching of the Aggregates, a compensating Project 
is added, but afterwards matching fails to handle it.

This issue proposes to add a rule that match when query and target are both 
Project and query has a compensating Project as a child. After this issue below 
case can be handled:
{code:java}
Query:
Project(projects: [$0, +($1, 2)])
  Project(projects: [$1, $3, $4])
Rel-A
 Target:  
Project(projects: [$1, +($3, 2)])
  Rel-A
{code}



> Add ProjectOnProjectToProjectUnifyRule for materialization matching.
> 
>
> Key: CALCITE-3256
> URL: https://issues.apache.org/jira/browse/CALCITE-3256
> Project: Calcite
>  Issue Type: Sub-task
>  Components: core
>Reporter: jin xing
>Assignee: jin xing
>Priority: Minor
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> In current code, below matching fails:
> {code:java}
> MV:
> select deptno, sum(salary) + 2, sum(commission)
> from emps
> group by deptno
> Query:
> select deptno, sum(salary) + 2
> from emps
> group by deptno
> {code}
> The reason is that --  after matching of the Aggregates, a compensating 
> Project is added, but afterwards matching fails to handle it.
> This issue proposes to add a rule that match when query and target are both 
> Project and query has a compensating Project as a child. After this issue 
> below case can be handled:
> {code:java}
> Query:
> Project(projects: [$0, +($1, 2)])
>   Project(projects: [$1, $3, $4])
> Rel-A
> Target:  
> Project(projects: [$1, +($3, 2)])
>   Rel-A
> {code}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (CALCITE-3256) Add ProjectOnProjectToProjectUnifyRule for materialization matching.

[jin xing (Jira)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing updated CALCITE-3256:
--
Description: 
In current code, below matching fails:

{code:java}
MV:
select deptno, sum(salary) + 2, sum(commission)
from emps
group by deptno

Query:
select deptno, sum(salary) + 2
from emps
group by deptno
{code}

The reason is that --  after matching of the Aggregates, a compensating Project 
is added, but afterwards matching fails to handle it.

This issue proposes to add a rule that match when query and target are both 
Project and query has a compensating Project as a child. After this issue below 
case can be handled:
{code:java}
Query:
Project(projects: [$0, +($1, 2)])
  Project(projects: [$1, $3, $4])
Rel-A
 Target:  
Project(projects: [$1, +($3, 2)])
  Rel-A
{code}


  was:
In current code, below matching fails:

{code:java}
MV:
select deptno, sum(salary) + 2, sum(commission)
from emps
group by deptno

Query:
select deptno, sum(salary) + 2
from emps
group by deptno
{code}

The reason is that --  after matching of the Aggregates, a compensating Project 
is added, but afterwards matching fails to handle it.

This issue proposes to add a rule that match when query and target are both 
Project and query has a compensating Project as a child. After this issue below 
case can be handled:
{code:java}
   query:   Project(projects: [$0, +($1, 2)])
  Project(projects: [$1, $3, $4])
   Rel-A
   target:  Project(projects: [$1, +($3, 2)])
 Rel-A
{code}



> Add ProjectOnProjectToProjectUnifyRule for materialization matching.
> 
>
> Key: CALCITE-3256
> URL: https://issues.apache.org/jira/browse/CALCITE-3256
> Project: Calcite
>  Issue Type: Sub-task
>  Components: core
>Reporter: jin xing
>Assignee: jin xing
>Priority: Minor
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> In current code, below matching fails:
> {code:java}
> MV:
> select deptno, sum(salary) + 2, sum(commission)
> from emps
> group by deptno
> Query:
> select deptno, sum(salary) + 2
> from emps
> group by deptno
> {code}
> The reason is that --  after matching of the Aggregates, a compensating 
> Project is added, but afterwards matching fails to handle it.
> This issue proposes to add a rule that match when query and target are both 
> Project and query has a compensating Project as a child. After this issue 
> below case can be handled:
> {code:java}
> Query:
> Project(projects: [$0, +($1, 2)])
>   Project(projects: [$1, $3, $4])
> Rel-A
>  Target:  
> Project(projects: [$1, +($3, 2)])
>   Rel-A
> {code}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (CALCITE-3256) Add ProjectOnProjectToProjectUnifyRule for materialization matching.

[jin xing (Jira)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing updated CALCITE-3256:
--
Description: 
In current code, below matching fails:

{code:java}
MV:
select deptno, sum(salary) + 2, sum(commission)
from emps
group by deptno

Query:
select deptno, sum(salary) + 2
from emps
group by deptno
{code}

The reason is that --  after matching of the Aggregates, a compensating Project 
is added, but afterwards matching fails to handle it.

This issue proposes to add a rule that match when query and target are both 
Project and query has a compensating Project as a child. After this issue below 
case can be handled:
{code:java}
   query:   Project(projects: [$0, +($1, 2)])
  Project(projects: [$1, $3, $4])
   Rel-A
   target:  Project(projects: [$1, +($3, 2)])
 Rel-A
{code}


  was:
This issue proposes to add a rule to match a project
{code:java}
MV:
select deptno, sum(salary) + 2, sum(commission)
from emps
group by deptno

Query:
select deptno, sum(salary) + 2
from emps
group by deptno
{code}

After matching of the Aggregates, a compensating Project is added, but 
afterwards matching fails to handle it.


> Add ProjectOnProjectToProjectUnifyRule for materialization matching.
> 
>
> Key: CALCITE-3256
> URL: https://issues.apache.org/jira/browse/CALCITE-3256
> Project: Calcite
>  Issue Type: Sub-task
>  Components: core
>Reporter: jin xing
>Assignee: jin xing
>Priority: Minor
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> In current code, below matching fails:
> {code:java}
> MV:
> select deptno, sum(salary) + 2, sum(commission)
> from emps
> group by deptno
> Query:
> select deptno, sum(salary) + 2
> from emps
> group by deptno
> {code}
> The reason is that --  after matching of the Aggregates, a compensating 
> Project is added, but afterwards matching fails to handle it.
> This issue proposes to add a rule that match when query and target are both 
> Project and query has a compensating Project as a child. After this issue 
> below case can be handled:
> {code:java}
>query:   Project(projects: [$0, +($1, 2)])
>   Project(projects: [$1, $3, $4])
>Rel-A
>target:  Project(projects: [$1, +($3, 2)])
>  Rel-A
> {code}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (CALCITE-3274) Add FilterOnProjectToFilterUnifyRule for materialization matching

[jin xing (Jira)]

jin xing created CALCITE-3274:
-

 Summary: Add FilterOnProjectToFilterUnifyRule for materialization 
matching
 Key: CALCITE-3274
 URL: https://issues.apache.org/jira/browse/CALCITE-3274
 Project: Calcite
  Issue Type: Sub-task
  Components: core
Reporter: jin xing






--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (CALCITE-3274) Add FilterOnProjectToFilterUnifyRule for materialization matching

[jin xing (Jira)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing updated CALCITE-3274:
--
Description: 
With current implementation of MV matching, below case fails

{code:java}
  @Test public void testFilterOnProject() {
String deduplicated =
"(select \"empid\", \"deptno\", \"name\", \"salary\", \"commission\"\n"
+ "from \"emps\"\n"
+ "group by \"empid\", \"deptno\", \"name\", \"salary\", 
\"commission\")";
String mv =
"select * from\n"
+ "(select \"deptno\", sum(\"salary\") as \"sum_salary\", 
sum(\"commission\")\n"
+ "from " + deduplicated + "\n"
+ "group by \"deptno\")\n"
+ "where \"sum_salary\" > 10";
String query =
"select * from\n"
+ "(select \"deptno\", sum(\"salary\") as \"sum_salary\"\n"
+ "from " + deduplicated + "\n"
+ "group by \"deptno\")\n"
+ "where \"sum_salary\" > 10";
checkMaterialize(mv, query);
  }
{code}

{{Reason:}}
After matching of the Aggregates, a compensating Project is added, but 
afterwards matching of filter fails to handle it.
This issue proposes to handle such case.


> Add FilterOnProjectToFilterUnifyRule for materialization matching
> -
>
> Key: CALCITE-3274
> URL: https://issues.apache.org/jira/browse/CALCITE-3274
> Project: Calcite
>  Issue Type: Sub-task
>  Components: core
>Reporter: jin xing
>Priority: Major
>
> With current implementation of MV matching, below case fails
> {code:java}
>   @Test public void testFilterOnProject() {
> String deduplicated =
> "(select \"empid\", \"deptno\", \"name\", \"salary\", 
> \"commission\"\n"
> + "from \"emps\"\n"
> + "group by \"empid\", \"deptno\", \"name\", \"salary\", 
> \"commission\")";
> String mv =
> "select * from\n"
> + "(select \"deptno\", sum(\"salary\") as \"sum_salary\", 
> sum(\"commission\")\n"
> + "from " + deduplicated + "\n"
> + "group by \"deptno\")\n"
> + "where \"sum_salary\" > 10";
> String query =
> "select * from\n"
> + "(select \"deptno\", sum(\"salary\") as \"sum_salary\"\n"
> + "from " + deduplicated + "\n"
> + "group by \"deptno\")\n"
> + "where \"sum_salary\" > 10";
> checkMaterialize(mv, query);
>   }
> {code}
> {{Reason:}}
> After matching of the Aggregates, a compensating Project is added, but 
> afterwards matching of filter fails to handle it.
> This issue proposes to handle such case.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (CALCITE-3274) Add FilterOnProjectToFilterUnifyRule for materialization matching

[jin xing (Jira)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing updated CALCITE-3274:
--
Description: 
With current implementation of MV matching, below case fails

{code:java}
  @Test public void testFilterOnProject() {
String deduplicated =
"(select \"empid\", \"deptno\", \"name\", \"salary\", \"commission\"\n"
+ "from \"emps\"\n"
+ "group by \"empid\", \"deptno\", \"name\", \"salary\", 
\"commission\")";
String mv =
"select * from\n"
+ "(select \"deptno\", sum(\"salary\") as \"sum_salary\", 
sum(\"commission\")\n"
+ "from " + deduplicated + "\n"
+ "group by \"deptno\")\n"
+ "where \"sum_salary\" > 10";
String query =
"select * from\n"
+ "(select \"deptno\", sum(\"salary\") as \"sum_salary\"\n"
+ "from " + deduplicated + "\n"
+ "group by \"deptno\")\n"
+ "where \"sum_salary\" > 10";
checkMaterialize(mv, query);
  }
{code}

{{Reason:}}
After matching of the Aggregates, a compensating Project is added, but 
afterwards matching of filter fails to handle it.
This issue proposes to handle such case where query and target are Filters and 
query has a compensating Project child node


  was:
With current implementation of MV matching, below case fails

{code:java}
  @Test public void testFilterOnProject() {
String deduplicated =
"(select \"empid\", \"deptno\", \"name\", \"salary\", \"commission\"\n"
+ "from \"emps\"\n"
+ "group by \"empid\", \"deptno\", \"name\", \"salary\", 
\"commission\")";
String mv =
"select * from\n"
+ "(select \"deptno\", sum(\"salary\") as \"sum_salary\", 
sum(\"commission\")\n"
+ "from " + deduplicated + "\n"
+ "group by \"deptno\")\n"
+ "where \"sum_salary\" > 10";
String query =
"select * from\n"
+ "(select \"deptno\", sum(\"salary\") as \"sum_salary\"\n"
+ "from " + deduplicated + "\n"
+ "group by \"deptno\")\n"
+ "where \"sum_salary\" > 10";
checkMaterialize(mv, query);
  }
{code}

{{Reason:}}
After matching of the Aggregates, a compensating Project is added, but 
afterwards matching of filter fails to handle it.
This issue proposes to handle such case.



> Add FilterOnProjectToFilterUnifyRule for materialization matching
> -
>
> Key: CALCITE-3274
> URL: https://issues.apache.org/jira/browse/CALCITE-3274
> Project: Calcite
>  Issue Type: Sub-task
>  Components: core
>Reporter: jin xing
>Priority: Major
>
> With current implementation of MV matching, below case fails
> {code:java}
>   @Test public void testFilterOnProject() {
> String deduplicated =
> "(select \"empid\", \"deptno\", \"name\", \"salary\", 
> \"commission\"\n"
> + "from \"emps\"\n"
> + "group by \"empid\", \"deptno\", \"name\", \"salary\", 
> \"commission\")";
> String mv =
> "select * from\n"
> + "(select \"deptno\", sum(\"salary\") as \"sum_salary\", 
> sum(\"commission\")\n"
> + "from " + deduplicated + "\n"
> + "group by \"deptno\")\n"
> + "where \"sum_salary\" > 10";
> String query =
> "select * from\n"
> + "(select \"deptno\", sum(\"salary\") as \"sum_salary\"\n"
> + "from " + deduplicated + "\n"
> + "group by \"deptno\")\n"
> + "where \"sum_salary\" > 10";
> checkMaterialize(mv, query);
>   }
> {code}
> {{Reason:}}
> After matching of the Aggregates, a compensating Project is added, but 
> afterwards matching of filter fails to handle it.
> This issue proposes to handle such case where query and target are Filters 
> and query has a compensating Project child node



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Assigned] (CALCITE-3274) Add FilterOnProjectToFilterUnifyRule for materialization matching

[jin xing (Jira)]

 [ 
https://issues.apache.org/jira/browse/CALCITE-3274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing reassigned CALCITE-3274:
-

Assignee: jin xing

> Add FilterOnProjectToFilterUnifyRule for materialization matching
> -
>
> Key: CALCITE-3274
> URL: https://issues.apache.org/jira/browse/CALCITE-3274
> Project: Calcite
>  Issue Type: Sub-task
>  Components: core
>Reporter: jin xing
>Assignee: jin xing
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> With current implementation of MV matching, below case fails
> {code:java}
>   @Test public void testFilterOnProject() {
> String deduplicated =
> "(select \"empid\", \"deptno\", \"name\", \"salary\", 
> \"commission\"\n"
> + "from \"emps\"\n"
> + "group by \"empid\", \"deptno\", \"name\", \"salary\", 
> \"commission\")";
> String mv =
> "select * from\n"
> + "(select \"deptno\", sum(\"salary\") as \"sum_salary\", 
> sum(\"commission\")\n"
> + "from " + deduplicated + "\n"
> + "group by \"deptno\")\n"
> + "where \"sum_salary\" > 10";
> String query =
> "select * from\n"
> + "(select \"deptno\", sum(\"salary\") as \"sum_salary\"\n"
> + "from " + deduplicated + "\n"
> + "group by \"deptno\")\n"
> + "where \"sum_salary\" > 10";
> checkMaterialize(mv, query);
>   }
> {code}
> {{Reason:}}
> After matching of the Aggregates, a compensating Project is added, but 
> afterwards matching of filter fails to handle it.
> This issue proposes to handle such case where query and target are Filters 
> and query has a compensating Project child node



--
This message was sent by Atlassian Jira
(v8.3.2#803003)