[jira] [Commented] (CALCITE-4360) Apply SubstitutionRule first in top-down driven rule apply
[ https://issues.apache.org/jira/browse/CALCITE-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17221928#comment-17221928 ] Jinpeng Wu commented on CALCITE-4360: - Yes. This should be a typo of previous commit. Thanks for fixing this. > Apply SubstitutionRule first in top-down driven rule apply > -- > > Key: CALCITE-4360 > URL: https://issues.apache.org/jira/browse/CALCITE-4360 > Project: Calcite > Issue Type: Improvement >Reporter: Chunwei Lei >Assignee: Chunwei Lei >Priority: Major > Labels: pull-request-available > Attachments: image-2020-10-27-21-55-55-155.png > > Time Spent: 20m > Remaining Estimate: 0h > > In the current implementation, TopDownRuleQueue adds substitute rules in the > end wrongly. The SubstitutionRule should be executed first. > !image-2020-10-27-21-55-55-155.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-4354) ITEM operator does not support synthetic struct type
[
https://issues.apache.org/jira/browse/CALCITE-4354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17221917#comment-17221917
]
Danny Chen edited comment on CALCITE-4354 at 10/28/20, 2:42 AM:
I tried this SQL in PostgreSQL 9.6
{code:sql}
create table t1(
f0 integer[],
f1 varchar(20)
);
insert into t1 values('{1, 2, 3}', 'abc');
select f0[1] from t1; -- outputs 1
{code}
I also found this access rules in SQL standard 2011 6.24
{code:java}
1) If the value of or is
the null value, then the result of is the null value.
2) Let i be the value of . Case:
a) If i is greater than zero and less than or equal to the cardinality of
, then the result of is the
value of the i-th element of the value of .
b) Otherwise, an exception condition is raised: data exception — array element
error.
{code}
So the array type is index 1-based. We can follow this for row type.
was (Author: danny0405):
I tried this SQL in PostgreSQL 9.6
{code:sql}
create table t1(
f0 integer[],
f1 varchar(20)
);
insert into t1 values('{1, 2, 3}', 'abc');
select f0[1] from t1; -- outputs 1
{code}
I also found this access rules in SQL standard 2011 6.24
{code:noformat}
1) If the value of or is
the null value, then the result of is the null value.
2) Let i be the value of . Case:
a) If i is greater than zero and less than or equal to the cardinality of
, then the result of is the
value of the i-th element of the value of .
b) Otherwise, an exception condition is raised: data exception — array element
error.
{code}
So the array type is index 1-based. We can follow this for row type.
> ITEM operator does not support synthetic struct type
>
>
> Key: CALCITE-4354
> URL: https://issues.apache.org/jira/browse/CALCITE-4354
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.27.0
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> The current implementation of the "ITEM" operator only supports struct/row
> type when this type if based on a class with named fields (that is, a
> non-synthetic type).
> For those non-synthetic struct types, the "ITEM" operator can be used to
> access named fields via a "string"-typed argument
> ([SqlItemOperator#96|https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql/fun/SqlItemOperator.java#L96]).
> The type checker accepts to apply "ITEM" on "ANY" type
> ([SqlItemOperator#49|https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql/fun/SqlItemOperator.java#L49]),
> and the operand checker only accepts "string"-based values
> ([SqlItemOperator.java#105|https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql/fun/SqlItemOperator.java#L105]).
> [SqlValidatorTest.java#L11933|https://github.com/apache/calcite/blob/master/core/src/test/java/org/apache/calcite/test/SqlValidatorTest.java#L11933]
> is an example of application of the "ITEM" operator over a struct with named
> fields.
> However, the “getAllowedSignatures” method
> ([SqlItemOperator.java#116|https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql/fun/SqlItemOperator.java#L116])
> does not reflect this, and asserts a signature as follows:
> "[]" or "[]".
> "ITEM" operator could be enriched with the support of synthetic struct/row
> type, with a positional access to the fields with a signature
> "[]".
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4354) ITEM operator does not support synthetic struct type
[
https://issues.apache.org/jira/browse/CALCITE-4354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17221917#comment-17221917
]
Danny Chen commented on CALCITE-4354:
-
I tried this SQL in PostgreSQL 9.6
{code:sql}
create table t1(
f0 integer[],
f1 varchar(20)
);
insert into t1 values('{1, 2, 3}', 'abc');
select f0[1] from t1; -- outputs 1
{code}
I also found this access rules in SQL standard 2011 6.24
{code:noformat}
1) If the value of or is
the null value, then the result of is the null value.
2) Let i be the value of . Case:
a) If i is greater than zero and less than or equal to the cardinality of
, then the result of is the
value of the i-th element of the value of .
b) Otherwise, an exception condition is raised: data exception — array element
error.
{code}
So the array type is index 1-based. We can follow this for row type.
> ITEM operator does not support synthetic struct type
>
>
> Key: CALCITE-4354
> URL: https://issues.apache.org/jira/browse/CALCITE-4354
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.27.0
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> The current implementation of the "ITEM" operator only supports struct/row
> type when this type if based on a class with named fields (that is, a
> non-synthetic type).
> For those non-synthetic struct types, the "ITEM" operator can be used to
> access named fields via a "string"-typed argument
> ([SqlItemOperator#96|https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql/fun/SqlItemOperator.java#L96]).
> The type checker accepts to apply "ITEM" on "ANY" type
> ([SqlItemOperator#49|https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql/fun/SqlItemOperator.java#L49]),
> and the operand checker only accepts "string"-based values
> ([SqlItemOperator.java#105|https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql/fun/SqlItemOperator.java#L105]).
> [SqlValidatorTest.java#L11933|https://github.com/apache/calcite/blob/master/core/src/test/java/org/apache/calcite/test/SqlValidatorTest.java#L11933]
> is an example of application of the "ITEM" operator over a struct with named
> fields.
> However, the “getAllowedSignatures” method
> ([SqlItemOperator.java#116|https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql/fun/SqlItemOperator.java#L116])
> does not reflect this, and asserts a signature as follows:
> "[]" or "[]".
> "ITEM" operator could be enriched with the support of synthetic struct/row
> type, with a positional access to the fields with a signature
> "[]".
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4345) SUM CASE WHEN xxx THEN 1 without "ELSE" throws NPE in AggregateCaseToFilterRule
[
https://issues.apache.org/jira/browse/CALCITE-4345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17221911#comment-17221911
]
Jiatao Tao commented on CALCITE-4345:
-
Sure~
> SUM CASE WHEN xxx THEN 1 without "ELSE" throws NPE in
> AggregateCaseToFilterRule
> ---
>
> Key: CALCITE-4345
> URL: https://issues.apache.org/jira/browse/CALCITE-4345
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Jiatao Tao
>Assignee: Jiatao Tao
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2020-10-19-16-59-44-900.png
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> Add a test in CsvTest:
>
> {code:java}
> @Test void testSumCaseWhenWithoutElse() throws SQLException {
> sql("model", "select sum( CASE WHEN empno=10 THEN 1 END ) from EMPS").ok();
> }
> {code}
> Will throw NPE
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4358) Potential memory leak in VolcanoPlanner#materializations due to circular reference
[ https://issues.apache.org/jira/browse/CALCITE-4358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17221909#comment-17221909 ] Jiatao Tao commented on CALCITE-4358: - [~zabetak] In fact, no, the RelOptMaterialization is only used in the volcano planner's code, we don't change that. We just get the MV string SQL from HMS, use calcite to transform the String SQL to RelNode, and call VolcanoPlanner#addMaterialization every query, then it's volcano planner's ori process. > Potential memory leak in VolcanoPlanner#materializations due to circular > reference > -- > > Key: CALCITE-4358 > URL: https://issues.apache.org/jira/browse/CALCITE-4358 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: Jiatao Tao >Assignee: Jiatao Tao >Priority: Major > > Hi fellows > We use Calcite's materialization view in our product env, and it OOM times, I > dig the memory dump and found there are all of RelOptMaterialization instance. > > We will call Planner#clear before every query and will new a new Planner just > as org.apache.calcite.prepare.CalcitePrepareImpl did, and inside the dump, > there are lots of planner instance, so I suspect it is a circular reference > that JVM can not GC them, here's the chain: > > 1. org.apache.calcite.plan.volcano.VolcanoPlanner#materializations: > List 2. > org.apache.calcite.plan.RelOptMaterialization#queryRel: AbstractRelNode 3. > org.apache.calcite.rel.AbstractRelNode#cluster: RelOptCluster 4. > org.apache.calcite.plan.RelOptCluster#planner: VolcanoPlanner 5. > org.apache.calcite.plan.volcano.VolcanoPlanner#materializations: > List -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4354) ITEM operator does not support synthetic struct type
[
https://issues.apache.org/jira/browse/CALCITE-4354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17221692#comment-17221692
]
Julian Hyde commented on CALCITE-4354:
--
Makes sense:
* Please update reference.md.
* Is the index 0-based or 1-based?
* Change the JIRA case summary and commit message. Frame it as a feature not a
bug. Explain it in terms of ROW rather than 'synthetic struct type', and give a
simple SQL example.
* Add one or two tests to {{operator.iq}}.
We are also going to want the ITEM operator to apply to record/struct types,
e.g. {{address[2]}} and {{address['ZIPCODE']}}, where {{address}} is a struct
or record type that has fields STREET, ZIPCODE, CITY, STATE. That could be a
different JIRA case.
> ITEM operator does not support synthetic struct type
>
>
> Key: CALCITE-4354
> URL: https://issues.apache.org/jira/browse/CALCITE-4354
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.27.0
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> The current implementation of the "ITEM" operator only supports struct/row
> type when this type if based on a class with named fields (that is, a
> non-synthetic type).
> For those non-synthetic struct types, the "ITEM" operator can be used to
> access named fields via a "string"-typed argument
> ([SqlItemOperator#96|https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql/fun/SqlItemOperator.java#L96]).
> The type checker accepts to apply "ITEM" on "ANY" type
> ([SqlItemOperator#49|https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql/fun/SqlItemOperator.java#L49]),
> and the operand checker only accepts "string"-based values
> ([SqlItemOperator.java#105|https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql/fun/SqlItemOperator.java#L105]).
> [SqlValidatorTest.java#L11933|https://github.com/apache/calcite/blob/master/core/src/test/java/org/apache/calcite/test/SqlValidatorTest.java#L11933]
> is an example of application of the "ITEM" operator over a struct with named
> fields.
> However, the “getAllowedSignatures” method
> ([SqlItemOperator.java#116|https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql/fun/SqlItemOperator.java#L116])
> does not reflect this, and asserts a signature as follows:
> "[]" or "[]".
> "ITEM" operator could be enriched with the support of synthetic struct/row
> type, with a positional access to the fields with a signature
> "[]".
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4352) RelBuilder/RexSimplify/Sarg incorrectly transforms complex expressions with IS NULL/IS NOT NULL
[
https://issues.apache.org/jira/browse/CALCITE-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17221679#comment-17221679
]
Julian Hyde commented on CALCITE-4352:
--
Thanks for checking, [~aigor].
> RelBuilder/RexSimplify/Sarg incorrectly transforms complex expressions with
> IS NULL/IS NOT NULL
> ---
>
> Key: CALCITE-4352
> URL: https://issues.apache.org/jira/browse/CALCITE-4352
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.27.0
> Environment: All environments
>Reporter: Igor Lozynskyi
>Priority: Major
> Labels: pull-request-available
> Attachments: rex-simplify-issue-tests-3.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Now, the following query with IS NOT NULL:
> {code:java}
> SELECT *
> FROM emp
> WHERE deptno > 20 AND deptno < 30 AND mgr IS NOT NULL{code}
> Is simplified to:
> {code:java}
> SELECT *
> FROM emp
> WHERE deptno > 20 AND deptno < 30{code}
> It seems that the simplification of complex `AND` filters incorrectly
> simplify `IS NOT NULL` to `TRUE`.
> During simplification, `IS NOT NULL` is transformed to `SEARCH($3,
> Sarg[(-∞..+∞), null])` instead of `SEARCH($3, Sarg[(-∞..+∞)])`.
> Also, the following query with IS NULL:
> {code:java}
> SELECT *
> FROM emp
> WHERE deptno > 20 AND deptno < 30 AND mgr IS NULL
> {code}
> Is simplified to the following query:
> {code:java}
> SELECT *
> FROM emp
> WHERE deptno > 20 AND deptno < 30
> {code}
> However, now it is correct when transformed into a Sarg, but incorrect after
> `RexUtil.expandSearch()` expansion.
>
> A simpler query does not show this issue (stays the same):
> {code:java}
> SELECT *
> FROM emp
> WHERE mgr IS NOT NULL{code}
> The patch with the corresponding test cases is in attachments.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4345) SUM CASE WHEN xxx THEN 1 without "ELSE" throws NPE in AggregateCaseToFilterRule
[
https://issues.apache.org/jira/browse/CALCITE-4345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17221671#comment-17221671
]
Julian Hyde commented on CALCITE-4345:
--
We have been going round and round discussing whether a NULL of type INTEGER is
or it not an int literal. This discussion is pointless, because
{{isIntLiteral}} is a private method. You can should replace it with something
else that works better. "Better" means simpler, more intuitive code.
> SUM CASE WHEN xxx THEN 1 without "ELSE" throws NPE in
> AggregateCaseToFilterRule
> ---
>
> Key: CALCITE-4345
> URL: https://issues.apache.org/jira/browse/CALCITE-4345
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Jiatao Tao
>Assignee: Jiatao Tao
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2020-10-19-16-59-44-900.png
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> Add a test in CsvTest:
>
> {code:java}
> @Test void testSumCaseWhenWithoutElse() throws SQLException {
> sql("model", "select sum( CASE WHEN empno=10 THEN 1 END ) from EMPS").ok();
> }
> {code}
> Will throw NPE
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4360) Apply SubstitutionRule first in top-down driven rule apply
[ https://issues.apache.org/jira/browse/CALCITE-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17221667#comment-17221667 ] Julian Hyde commented on CALCITE-4360: -- Do you have a test case that justifies your assertion? > Apply SubstitutionRule first in top-down driven rule apply > -- > > Key: CALCITE-4360 > URL: https://issues.apache.org/jira/browse/CALCITE-4360 > Project: Calcite > Issue Type: Improvement >Reporter: Chunwei Lei >Assignee: Chunwei Lei >Priority: Major > Labels: pull-request-available > Attachments: image-2020-10-27-21-55-55-155.png > > Time Spent: 20m > Remaining Estimate: 0h > > In the current implementation, TopDownRuleQueue adds substitute rules in the > end wrongly. The SubstitutionRule should be executed first. > !image-2020-10-27-21-55-55-155.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4352) RelBuilder/RexSimplify/Sarg incorrectly transforms complex expressions with IS NULL/IS NOT NULL
[
https://issues.apache.org/jira/browse/CALCITE-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17221661#comment-17221661
]
Igor Lozynskyi commented on CALCITE-4352:
-
[~julianhyde], I executed our test suites over your branch, and all suites are
green. Currently, I cannot detect any filter-related issues.
Thanks for this fix!
> RelBuilder/RexSimplify/Sarg incorrectly transforms complex expressions with
> IS NULL/IS NOT NULL
> ---
>
> Key: CALCITE-4352
> URL: https://issues.apache.org/jira/browse/CALCITE-4352
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.27.0
> Environment: All environments
>Reporter: Igor Lozynskyi
>Priority: Major
> Labels: pull-request-available
> Attachments: rex-simplify-issue-tests-3.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Now, the following query with IS NOT NULL:
> {code:java}
> SELECT *
> FROM emp
> WHERE deptno > 20 AND deptno < 30 AND mgr IS NOT NULL{code}
> Is simplified to:
> {code:java}
> SELECT *
> FROM emp
> WHERE deptno > 20 AND deptno < 30{code}
> It seems that the simplification of complex `AND` filters incorrectly
> simplify `IS NOT NULL` to `TRUE`.
> During simplification, `IS NOT NULL` is transformed to `SEARCH($3,
> Sarg[(-∞..+∞), null])` instead of `SEARCH($3, Sarg[(-∞..+∞)])`.
> Also, the following query with IS NULL:
> {code:java}
> SELECT *
> FROM emp
> WHERE deptno > 20 AND deptno < 30 AND mgr IS NULL
> {code}
> Is simplified to the following query:
> {code:java}
> SELECT *
> FROM emp
> WHERE deptno > 20 AND deptno < 30
> {code}
> However, now it is correct when transformed into a Sarg, but incorrect after
> `RexUtil.expandSearch()` expansion.
>
> A simpler query does not show this issue (stays the same):
> {code:java}
> SELECT *
> FROM emp
> WHERE mgr IS NOT NULL{code}
> The patch with the corresponding test cases is in attachments.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-4360) Apply SubstitutionRule first in top-down driven rule apply
[ https://issues.apache.org/jira/browse/CALCITE-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated CALCITE-4360: Labels: pull-request-available (was: ) > Apply SubstitutionRule first in top-down driven rule apply > -- > > Key: CALCITE-4360 > URL: https://issues.apache.org/jira/browse/CALCITE-4360 > Project: Calcite > Issue Type: Improvement >Reporter: Chunwei Lei >Assignee: Chunwei Lei >Priority: Major > Labels: pull-request-available > Attachments: image-2020-10-27-21-55-55-155.png > > Time Spent: 10m > Remaining Estimate: 0h > > In the current implementation, TopDownRuleQueue adds substitute rules in the > end wrongly. The SubstitutionRule should be executed first. > !image-2020-10-27-21-55-55-155.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-4360) Apply SubstitutionRule first in top-down driven rule apply
[ https://issues.apache.org/jira/browse/CALCITE-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chunwei Lei updated CALCITE-4360: - Summary: Apply SubstitutionRule first in top-down driven rule apply (was: Should apply SubstitutionRule first in top-down driven rule apply) > Apply SubstitutionRule first in top-down driven rule apply > -- > > Key: CALCITE-4360 > URL: https://issues.apache.org/jira/browse/CALCITE-4360 > Project: Calcite > Issue Type: Improvement >Reporter: Chunwei Lei >Assignee: Chunwei Lei >Priority: Major > Attachments: image-2020-10-27-21-55-55-155.png > > > In the current implementation, TopDownRuleQueue adds substitute rules in the > end wrongly. The SubstitutionRule should be executed first. > !image-2020-10-27-21-55-55-155.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-4360) Should apply SubstitutionRule first in top-down driven rule apply
[ https://issues.apache.org/jira/browse/CALCITE-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chunwei Lei updated CALCITE-4360: - Description: In the current implementation, TopDownRuleQueue adds substitute rules in the end wrongly. The SubstitutionRule should be executed first. !image-2020-10-27-21-55-55-155.png! was: In the current implementation, TopDownRuleQueue adds substitute rules in the end wrongly. The !image-2020-10-27-21-55-55-155.png! > Should apply SubstitutionRule first in top-down driven rule apply > - > > Key: CALCITE-4360 > URL: https://issues.apache.org/jira/browse/CALCITE-4360 > Project: Calcite > Issue Type: Improvement >Reporter: Chunwei Lei >Assignee: Chunwei Lei >Priority: Major > Attachments: image-2020-10-27-21-55-55-155.png > > > In the current implementation, TopDownRuleQueue adds substitute rules in the > end wrongly. The SubstitutionRule should be executed first. > !image-2020-10-27-21-55-55-155.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-4360) Should apply SubstitutionRule first in top-down driven rule apply
[ https://issues.apache.org/jira/browse/CALCITE-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chunwei Lei updated CALCITE-4360: - Summary: Should apply SubstitutionRule first in top-down driven rule apply (was: Should apply SubstituteRule first in top-down driven rule apply) > Should apply SubstitutionRule first in top-down driven rule apply > - > > Key: CALCITE-4360 > URL: https://issues.apache.org/jira/browse/CALCITE-4360 > Project: Calcite > Issue Type: Improvement >Reporter: Chunwei Lei >Assignee: Chunwei Lei >Priority: Major > Attachments: image-2020-10-27-21-55-55-155.png > > > In the current implementation, TopDownRuleQueue adds substitute rules in the > end wrongly. The > !image-2020-10-27-21-55-55-155.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-4360) Should apply SubstituteRule first in top-down driven rule apply
[ https://issues.apache.org/jira/browse/CALCITE-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chunwei Lei updated CALCITE-4360: - Description: In the current implementation, TopDownRuleQueue adds substitute rules in the end wrongly. The !image-2020-10-27-21-55-55-155.png! was:!image-2020-10-27-21-55-55-155.png! > Should apply SubstituteRule first in top-down driven rule apply > --- > > Key: CALCITE-4360 > URL: https://issues.apache.org/jira/browse/CALCITE-4360 > Project: Calcite > Issue Type: Improvement >Reporter: Chunwei Lei >Assignee: Chunwei Lei >Priority: Major > Attachments: image-2020-10-27-21-55-55-155.png > > > In the current implementation, TopDownRuleQueue adds substitute rules in the > end wrongly. The > !image-2020-10-27-21-55-55-155.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-4360) Should apply SubstituteRule first in top-down driven rule apply
Chunwei Lei created CALCITE-4360: Summary: Should apply SubstituteRule first in top-down driven rule apply Key: CALCITE-4360 URL: https://issues.apache.org/jira/browse/CALCITE-4360 Project: Calcite Issue Type: Improvement Reporter: Chunwei Lei Assignee: Chunwei Lei Attachments: image-2020-10-27-21-55-55-155.png !image-2020-10-27-21-55-55-155.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (CALCITE-4359) EnumerableProject.create didn't consider input's traitSet
[
https://issues.apache.org/jira/browse/CALCITE-4359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiatao Tao closed CALCITE-4359.
---
Resolution: Not A Problem
> EnumerableProject.create didn't consider input's traitSet
> -
>
> Key: CALCITE-4359
> URL: https://issues.apache.org/jira/browse/CALCITE-4359
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Jiatao Tao
>Assignee: Jiatao Tao
>Priority: Major
>
> {code:java}
> /** Creates an EnumerableProject, specifying row type rather than field
> * names. */
> public static EnumerableProject create(final RelNode input,
> final List projects, RelDataType rowType) {
> final RelOptCluster cluster = input.getCluster();
> final RelMetadataQuery mq = cluster.getMetadataQuery();
> final RelTraitSet traitSet =
> cluster.traitSet().replace(EnumerableConvention.INSTANCE)
> .replaceIfs(RelCollationTraitDef.INSTANCE,
> () -> RelMdCollation.project(mq, input, projects));
> return new EnumerableProject(cluster, traitSet, input, projects, rowType);
> }
> {code}
> Code uses "cluster.traitSet()", but RelOptCluster#traitSet returns
> emptyTraitSet, should use input#getTraitSet, please tell me if i am wrong
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4359) EnumerableProject.create didn't consider input's traitSet
[
https://issues.apache.org/jira/browse/CALCITE-4359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17221421#comment-17221421
]
Jiatao Tao commented on CALCITE-4359:
-
Seems not an issue, close this one.
> EnumerableProject.create didn't consider input's traitSet
> -
>
> Key: CALCITE-4359
> URL: https://issues.apache.org/jira/browse/CALCITE-4359
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Jiatao Tao
>Assignee: Jiatao Tao
>Priority: Major
>
> {code:java}
> /** Creates an EnumerableProject, specifying row type rather than field
> * names. */
> public static EnumerableProject create(final RelNode input,
> final List projects, RelDataType rowType) {
> final RelOptCluster cluster = input.getCluster();
> final RelMetadataQuery mq = cluster.getMetadataQuery();
> final RelTraitSet traitSet =
> cluster.traitSet().replace(EnumerableConvention.INSTANCE)
> .replaceIfs(RelCollationTraitDef.INSTANCE,
> () -> RelMdCollation.project(mq, input, projects));
> return new EnumerableProject(cluster, traitSet, input, projects, rowType);
> }
> {code}
> Code uses "cluster.traitSet()", but RelOptCluster#traitSet returns
> emptyTraitSet, should use input#getTraitSet, please tell me if i am wrong
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4359) EnumerableProject.create didn't consider input's traitSet
Jiatao Tao created CALCITE-4359:
---
Summary: EnumerableProject.create didn't consider input's traitSet
Key: CALCITE-4359
URL: https://issues.apache.org/jira/browse/CALCITE-4359
Project: Calcite
Issue Type: Bug
Components: core
Reporter: Jiatao Tao
Assignee: Jiatao Tao
{code:java}
/** Creates an EnumerableProject, specifying row type rather than field
* names. */
public static EnumerableProject create(final RelNode input,
final List projects, RelDataType rowType) {
final RelOptCluster cluster = input.getCluster();
final RelMetadataQuery mq = cluster.getMetadataQuery();
final RelTraitSet traitSet =
cluster.traitSet().replace(EnumerableConvention.INSTANCE)
.replaceIfs(RelCollationTraitDef.INSTANCE,
() -> RelMdCollation.project(mq, input, projects));
return new EnumerableProject(cluster, traitSet, input, projects, rowType);
}
{code}
Code uses "cluster.traitSet()", but RelOptCluster#traitSet returns
emptyTraitSet, should use input#getTraitSet, please tell me if i am wrong
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4358) Potential memory leak in VolcanoPlanner#materializations due to circular reference
[ https://issues.apache.org/jira/browse/CALCITE-4358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17221377#comment-17221377 ] Stamatis Zampetakis commented on CALCITE-4358: -- One question is do you keep RelOptMaterialization somewhere else outside of Calcite? I don't think circular reference is a problem unless one part of the circle is referenced and kept by your code. > Potential memory leak in VolcanoPlanner#materializations due to circular > reference > -- > > Key: CALCITE-4358 > URL: https://issues.apache.org/jira/browse/CALCITE-4358 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: Jiatao Tao >Assignee: Jiatao Tao >Priority: Major > > Hi fellows > We use Calcite's materialization view in our product env, and it OOM times, I > dig the memory dump and found there are all of RelOptMaterialization instance. > > We will call Planner#clear before every query and will new a new Planner just > as org.apache.calcite.prepare.CalcitePrepareImpl did, and inside the dump, > there are lots of planner instance, so I suspect it is a circular reference > that JVM can not GC them, here's the chain: > > 1. org.apache.calcite.plan.volcano.VolcanoPlanner#materializations: > List 2. > org.apache.calcite.plan.RelOptMaterialization#queryRel: AbstractRelNode 3. > org.apache.calcite.rel.AbstractRelNode#cluster: RelOptCluster 4. > org.apache.calcite.plan.RelOptCluster#planner: VolcanoPlanner 5. > org.apache.calcite.plan.volcano.VolcanoPlanner#materializations: > List -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4354) ITEM operator does not support synthetic struct type
[ https://issues.apache.org/jira/browse/CALCITE-4354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17221365#comment-17221365 ] Stamatis Zampetakis commented on CALCITE-4354: -- I think the change makes sense. I left some small comments in the PR. Other than that it would be nice to update the description of the case with some SQL examples of what is not working now and what will be working after the change. > ITEM operator does not support synthetic struct type > > > Key: CALCITE-4354 > URL: https://issues.apache.org/jira/browse/CALCITE-4354 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.27.0 >Reporter: Alessandro Solimando >Assignee: Alessandro Solimando >Priority: Minor > Labels: pull-request-available > Time Spent: 1h 50m > Remaining Estimate: 0h > > The current implementation of the "ITEM" operator only supports struct/row > type when this type if based on a class with named fields (that is, a > non-synthetic type). > For those non-synthetic struct types, the "ITEM" operator can be used to > access named fields via a "string"-typed argument > ([SqlItemOperator#96|https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql/fun/SqlItemOperator.java#L96]). > The type checker accepts to apply "ITEM" on "ANY" type > ([SqlItemOperator#49|https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql/fun/SqlItemOperator.java#L49]), > and the operand checker only accepts "string"-based values > ([SqlItemOperator.java#105|https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql/fun/SqlItemOperator.java#L105]). > [SqlValidatorTest.java#L11933|https://github.com/apache/calcite/blob/master/core/src/test/java/org/apache/calcite/test/SqlValidatorTest.java#L11933] > is an example of application of the "ITEM" operator over a struct with named > fields. > However, the “getAllowedSignatures” method > ([SqlItemOperator.java#116|https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql/fun/SqlItemOperator.java#L116]) > does not reflect this, and asserts a signature as follows: > "[]" or "[]". > "ITEM" operator could be enriched with the support of synthetic struct/row > type, with a positional access to the fields with a signature > "[]". -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-4345) SUM CASE WHEN xxx THEN 1 without "ELSE" throws NPE in AggregateCaseToFilterRule
[
https://issues.apache.org/jira/browse/CALCITE-4345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17221362#comment-17221362
]
Jiatao Tao edited comment on CALCITE-4345 at 10/27/20, 12:12 PM:
-
"isNumericLiteralEqualTo" seems to expand the scope, we need to consider all
the type, I can combine "isIntLiteral" with "intValue" as a new method like
"isIntLiteralEqual(RexNode, int)".
was (Author: aron.tao):
As I said in PR:
If we think "null" is not a reasonable "int literal", I can add this condition
to "isIntLiteral"
The [@vlsi|https://github.com/vlsi] mentioned (overflow case) is not the same,
if isIntLiteral(decimal) returns false, we already know it is not an int, why
do we still use intValue(decimal)? Please tell me if I am wrong
"isNumericLiteralEqualTo" seems to expand the scope, we need to consider all
the type, I can combine "isIntLiteral" with "intValue" as a new method like
"isIntLiteralEqualTo(RexNode, int)".
> SUM CASE WHEN xxx THEN 1 without "ELSE" throws NPE in
> AggregateCaseToFilterRule
> ---
>
> Key: CALCITE-4345
> URL: https://issues.apache.org/jira/browse/CALCITE-4345
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Jiatao Tao
>Assignee: Jiatao Tao
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2020-10-19-16-59-44-900.png
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> Add a test in CsvTest:
>
> {code:java}
> @Test void testSumCaseWhenWithoutElse() throws SQLException {
> sql("model", "select sum( CASE WHEN empno=10 THEN 1 END ) from EMPS").ok();
> }
> {code}
> Will throw NPE
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4345) SUM CASE WHEN xxx THEN 1 without "ELSE" throws NPE in AggregateCaseToFilterRule
[
https://issues.apache.org/jira/browse/CALCITE-4345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17221362#comment-17221362
]
Jiatao Tao commented on CALCITE-4345:
-
As I said in PR:
If we think "null" is not a reasonable "int literal", I can add this condition
to "isIntLiteral"
The [@vlsi|https://github.com/vlsi] mentioned (overflow case) is not the same,
if isIntLiteral(decimal) returns false, we already know it is not an int, why
do we still use intValue(decimal)? Please tell me if I am wrong
"isNumericLiteralEqualTo" seems to expand the scope, we need to consider all
the type, I can combine "isIntLiteral" with "intValue" as a new method like
"isIntLiteralEqualTo(RexNode, int)".
> SUM CASE WHEN xxx THEN 1 without "ELSE" throws NPE in
> AggregateCaseToFilterRule
> ---
>
> Key: CALCITE-4345
> URL: https://issues.apache.org/jira/browse/CALCITE-4345
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Jiatao Tao
>Assignee: Jiatao Tao
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2020-10-19-16-59-44-900.png
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> Add a test in CsvTest:
>
> {code:java}
> @Test void testSumCaseWhenWithoutElse() throws SQLException {
> sql("model", "select sum( CASE WHEN empno=10 THEN 1 END ) from EMPS").ok();
> }
> {code}
> Will throw NPE
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4358) Potential memory leak in VolcanoPlanner#materializations due to circular reference
Jiatao Tao created CALCITE-4358: --- Summary: Potential memory leak in VolcanoPlanner#materializations due to circular reference Key: CALCITE-4358 URL: https://issues.apache.org/jira/browse/CALCITE-4358 Project: Calcite Issue Type: Bug Components: core Reporter: Jiatao Tao Assignee: Jiatao Tao Hi fellows We use Calcite's materialization view in our product env, and it OOM times, I dig the memory dump and found there are all of RelOptMaterialization instance. We will call Planner#clear before every query and will new a new Planner just as org.apache.calcite.prepare.CalcitePrepareImpl did, and inside the dump, there are lots of planner instance, so I suspect it is a circular reference that JVM can not GC them, here's the chain: 1. org.apache.calcite.plan.volcano.VolcanoPlanner#materializations: List 2. org.apache.calcite.plan.RelOptMaterialization#queryRel: AbstractRelNode 3. org.apache.calcite.rel.AbstractRelNode#cluster: RelOptCluster 4. org.apache.calcite.plan.RelOptCluster#planner: VolcanoPlanner 5. org.apache.calcite.plan.volcano.VolcanoPlanner#materializations: List -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4354) ITEM operator does not support synthetic struct type
[ https://issues.apache.org/jira/browse/CALCITE-4354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17221216#comment-17221216 ] Danny Chen commented on CALCITE-4354: - I'm fine with the change. What do you think [~zabetak] and [~julianhyde] ? > ITEM operator does not support synthetic struct type > > > Key: CALCITE-4354 > URL: https://issues.apache.org/jira/browse/CALCITE-4354 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.27.0 >Reporter: Alessandro Solimando >Assignee: Alessandro Solimando >Priority: Minor > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > The current implementation of the "ITEM" operator only supports struct/row > type when this type if based on a class with named fields (that is, a > non-synthetic type). > For those non-synthetic struct types, the "ITEM" operator can be used to > access named fields via a "string"-typed argument > ([SqlItemOperator#96|https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql/fun/SqlItemOperator.java#L96]). > The type checker accepts to apply "ITEM" on "ANY" type > ([SqlItemOperator#49|https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql/fun/SqlItemOperator.java#L49]), > and the operand checker only accepts "string"-based values > ([SqlItemOperator.java#105|https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql/fun/SqlItemOperator.java#L105]). > [SqlValidatorTest.java#L11933|https://github.com/apache/calcite/blob/master/core/src/test/java/org/apache/calcite/test/SqlValidatorTest.java#L11933] > is an example of application of the "ITEM" operator over a struct with named > fields. > However, the “getAllowedSignatures” method > ([SqlItemOperator.java#116|https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql/fun/SqlItemOperator.java#L116]) > does not reflect this, and asserts a signature as follows: > "[]" or "[]". > "ITEM" operator could be enriched with the support of synthetic struct/row > type, with a positional access to the fields with a signature > "[]". -- This message was sent by Atlassian Jira (v8.3.4#803005)
