[jira] [Updated] (CALCITE-3909) RelMdMinRowCount doesn't take into account UNION DISTINCT
[
https://issues.apache.org/jira/browse/CALCITE-3909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chunwei Lei updated CALCITE-3909:
-
Description:
When computing minRowCount for Union, it doesn't task into account UNION
DISTINCT.
{code:java}
public Double getMinRowCount(Union rel, RelMetadataQuery mq) {
double rowCount = 0.0;
for (RelNode input : rel.getInputs()) {
Double partialRowCount = mq.getMinRowCount(input);
if (partialRowCount != null) {
rowCount += partialRowCount;
}
}
return rowCount;
}
{code}
> RelMdMinRowCount doesn't take into account UNION DISTINCT
> --
>
> Key: CALCITE-3909
> URL: https://issues.apache.org/jira/browse/CALCITE-3909
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.22.0
>Reporter: Chunwei Lei
>Assignee: Chunwei Lei
>Priority: Major
>
> When computing minRowCount for Union, it doesn't task into account UNION
> DISTINCT.
> {code:java}
> public Double getMinRowCount(Union rel, RelMetadataQuery mq) {
> double rowCount = 0.0;
> for (RelNode input : rel.getInputs()) {
> Double partialRowCount = mq.getMinRowCount(input);
> if (partialRowCount != null) {
> rowCount += partialRowCount;
> }
> }
> return rowCount;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3910) Enhance ProjectJoinTransposeRule to support SemiJoin and AntiJoin
Chunwei Lei created CALCITE-3910: Summary: Enhance ProjectJoinTransposeRule to support SemiJoin and AntiJoin Key: CALCITE-3910 URL: https://issues.apache.org/jira/browse/CALCITE-3910 Project: Calcite Issue Type: Improvement Components: core Affects Versions: 1.22.0 Reporter: Chunwei Lei -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-3910) Enhance ProjectJoinTransposeRule to support SemiJoin and AntiJoin
[
https://issues.apache.org/jira/browse/CALCITE-3910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chunwei Lei updated CALCITE-3910:
-
Description:
Currently, ProjectJoinTransposeRule does not support push project pass SemiJoin
and AntiJoin.
{code:java}
public void onMatch(RelOptRuleCall call) {
Project origProj = call.rel(0);
final Join join = call.rel(1);
if (!join.getJoinType().projectsRight()) {
return; // TODO: support SemiJoin / AntiJoin
}
...
...
}{code}
> Enhance ProjectJoinTransposeRule to support SemiJoin and AntiJoin
> -
>
> Key: CALCITE-3910
> URL: https://issues.apache.org/jira/browse/CALCITE-3910
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.22.0
>Reporter: Chunwei Lei
>Priority: Major
>
> Currently, ProjectJoinTransposeRule does not support push project pass
> SemiJoin and AntiJoin.
> {code:java}
> public void onMatch(RelOptRuleCall call) {
> Project origProj = call.rel(0);
> final Join join = call.rel(1);
> if (!join.getJoinType().projectsRight()) {
> return; // TODO: support SemiJoin / AntiJoin
> }
> ...
> ...
> }{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3909) RelMdMinRowCount doesn't take into account UNION DISTINCT
Chunwei Lei created CALCITE-3909: Summary: RelMdMinRowCount doesn't take into account UNION DISTINCT Key: CALCITE-3909 URL: https://issues.apache.org/jira/browse/CALCITE-3909 Project: Calcite Issue Type: Bug Components: core Affects Versions: 1.22.0 Reporter: Chunwei Lei Assignee: Chunwei Lei -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-3909) RelMdMinRowCount doesn't take into account UNION DISTINCT
[
https://issues.apache.org/jira/browse/CALCITE-3909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated CALCITE-3909:
Labels: pull-request-available (was: )
> RelMdMinRowCount doesn't take into account UNION DISTINCT
> --
>
> Key: CALCITE-3909
> URL: https://issues.apache.org/jira/browse/CALCITE-3909
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.22.0
>Reporter: Chunwei Lei
>Assignee: Chunwei Lei
>Priority: Major
> Labels: pull-request-available
>
> When computing minRowCount for Union, it doesn't task into account UNION
> DISTINCT.
> {code:java}
> public Double getMinRowCount(Union rel, RelMetadataQuery mq) {
> double rowCount = 0.0;
> for (RelNode input : rel.getInputs()) {
> Double partialRowCount = mq.getMinRowCount(input);
> if (partialRowCount != null) {
> rowCount += partialRowCount;
> }
> }
> return rowCount;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-3833) Support SemiJoin in EnumerableMergeJoin
[
https://issues.apache.org/jira/browse/CALCITE-3833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Q L updated CALCITE-3833:
---
Fix Version/s: 1.23.0
> Support SemiJoin in EnumerableMergeJoin
> ---
>
> Key: CALCITE-3833
> URL: https://issues.apache.org/jira/browse/CALCITE-3833
> Project: Calcite
> Issue Type: New Feature
> Components: core
>Affects Versions: 1.21.0
>Reporter: Ruben Q L
>Assignee: Ruben Q L
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.23.0
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Currently, {{EnumerableMergeJoin}} only supports INNER joins. The goal of
> this ticket is to improve this join operator in order to support SEMI joins
> too.
> Note: {{RelMdCollation#mergeJoin}} will need to be modified too: in case of
> SEMI join, it must return just the {{leftCollations}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-3833) Support SemiJoin in EnumerableMergeJoin
[
https://issues.apache.org/jira/browse/CALCITE-3833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Q L resolved CALCITE-3833.
Resolution: Fixed
> Support SemiJoin in EnumerableMergeJoin
> ---
>
> Key: CALCITE-3833
> URL: https://issues.apache.org/jira/browse/CALCITE-3833
> Project: Calcite
> Issue Type: New Feature
> Components: core
>Affects Versions: 1.21.0
>Reporter: Ruben Q L
>Assignee: Ruben Q L
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.23.0
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Currently, {{EnumerableMergeJoin}} only supports INNER joins. The goal of
> this ticket is to improve this join operator in order to support SEMI joins
> too.
> Note: {{RelMdCollation#mergeJoin}} will need to be modified too: in case of
> SEMI join, it must return just the {{leftCollations}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3833) Support SemiJoin in EnumerableMergeJoin
[
https://issues.apache.org/jira/browse/CALCITE-3833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17080370#comment-17080370
]
Ruben Q L commented on CALCITE-3833:
Resolved via
https://github.com/apache/calcite/commit/f93fc1db4ed9fbdae01022f29e8178dc58fd68f9
> Support SemiJoin in EnumerableMergeJoin
> ---
>
> Key: CALCITE-3833
> URL: https://issues.apache.org/jira/browse/CALCITE-3833
> Project: Calcite
> Issue Type: New Feature
> Components: core
>Affects Versions: 1.21.0
>Reporter: Ruben Q L
>Assignee: Ruben Q L
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Currently, {{EnumerableMergeJoin}} only supports INNER joins. The goal of
> this ticket is to improve this join operator in order to support SEMI joins
> too.
> Note: {{RelMdCollation#mergeJoin}} will need to be modified too: in case of
> SEMI join, it must return just the {{leftCollations}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3896) Pass through parent trait requests to child operators
[
https://issues.apache.org/jira/browse/CALCITE-3896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17080614#comment-17080614
]
Haisheng Yuan commented on CALCITE-3896:
1. As described in the issue, the redundancy *can't* be avoided given the
current design that physical operator must decide its children's required
traits when it is created. Solving it requires overhaul of planner, and the
implementation rules, which is a big task. This JIRA is not intended to solve
this problem.
2. All the logical and physical rules must be finished before enforcing. All
the alternatives generated by implementation rule are valid.
3. Given a specific required traitSet, passThough generates 1 or none. Can you
give an example to generate multiple candidates?
> Pass through parent trait requests to child operators
> -
>
> Key: CALCITE-3896
> URL: https://issues.apache.org/jira/browse/CALCITE-3896
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
>
> This is not on-demand trait requests as described in [mailing
> list|http://mail-archives.apache.org/mod_mbox/calcite-dev/201910.mbox/%3cd75b20f4-542a-4a73-897e-66ab426494c1.h.y...@alibaba-inc.com%3e],
> which requires the overhaul of the core planner. This ticket tries to enable
> VolcanoPlanner with basic and minimal ability to pass through parent trait
> request to child operators without rules, though may not be flexible or
> powerful, but should be able to work with current Calcite application with
> minimal changes.
> The method for physical operators to implement would be:
> {code:java}
> interface RelNode {
> RelNode passThrough(RelTraitSet required);
> }
> {code}
> Given that Calcite's physical operators decides its child operators' traits
> when the physical operator is created in physical implementation rule, there
> are some drawback that can't be avoided. e.g., given the following plan:
> {code:java}
> StreamAgg on [a]
>+-- MergeJoin on [a, b, c]
>|--- TableScan foo
>+--- TableScan bar
> {code}
> Suppose the MergeJoin implementation rule generates several mergejoins that
> distributes by [a], [a,b], [a,b,c] separately. Then we pass parent operator
> StreamAgg's trait request to MergeJoin. Since MergeJoin[a] satisfies parent's
> request, nothing to do. Next pass request to MergeJoin[a,b], we get
> MergeJoin[a], then pass request to MergeJoin[a,b,c], we get MergeJoin[a]
> again. We know they are redundant and there is no need to pass through parent
> operator's trait request, but these MergeJoin operators are independent and
> agnostic of each other's existence.
> The ideal way is that in physical implementation rule, during the creation of
> physical operator, it should not care about itself and its child operators'
> physical traits. But this is another different topic.
> Anyway, better than nothing, once it is done, we can provide the option to
> obsolete or disable {{AbstractConverter}}, but still be able to do property
> enforcement.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3912) Incorrect mapping parsing when properties have same name as reserved keywords in ElasticSearch
Andrei Sereda created CALCITE-3912:
--
Summary: Incorrect mapping parsing when properties have same name
as reserved keywords in ElasticSearch
Key: CALCITE-3912
URL: https://issues.apache.org/jira/browse/CALCITE-3912
Project: Calcite
Issue Type: Improvement
Components: elasticsearch-adapter
Reporter: Andrei Sereda
Assignee: Andrei Sereda
If a property has name {{type}} or {{properties}} ES adapter doesn't process
them correctly
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3896) Pass through parent trait requests to child operators
[
https://issues.apache.org/jira/browse/CALCITE-3896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17080615#comment-17080615
]
Haisheng Yuan commented on CALCITE-3896:
This will not replace abstract converter, but will make it possible to move on
to remove abstract converter.
> Pass through parent trait requests to child operators
> -
>
> Key: CALCITE-3896
> URL: https://issues.apache.org/jira/browse/CALCITE-3896
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
>
> This is not on-demand trait requests as described in [mailing
> list|http://mail-archives.apache.org/mod_mbox/calcite-dev/201910.mbox/%3cd75b20f4-542a-4a73-897e-66ab426494c1.h.y...@alibaba-inc.com%3e],
> which requires the overhaul of the core planner. This ticket tries to enable
> VolcanoPlanner with basic and minimal ability to pass through parent trait
> request to child operators without rules, though may not be flexible or
> powerful, but should be able to work with current Calcite application with
> minimal changes.
> The method for physical operators to implement would be:
> {code:java}
> interface RelNode {
> RelNode passThrough(RelTraitSet required);
> }
> {code}
> Given that Calcite's physical operators decides its child operators' traits
> when the physical operator is created in physical implementation rule, there
> are some drawback that can't be avoided. e.g., given the following plan:
> {code:java}
> StreamAgg on [a]
>+-- MergeJoin on [a, b, c]
>|--- TableScan foo
>+--- TableScan bar
> {code}
> Suppose the MergeJoin implementation rule generates several mergejoins that
> distributes by [a], [a,b], [a,b,c] separately. Then we pass parent operator
> StreamAgg's trait request to MergeJoin. Since MergeJoin[a] satisfies parent's
> request, nothing to do. Next pass request to MergeJoin[a,b], we get
> MergeJoin[a], then pass request to MergeJoin[a,b,c], we get MergeJoin[a]
> again. We know they are redundant and there is no need to pass through parent
> operator's trait request, but these MergeJoin operators are independent and
> agnostic of each other's existence.
> The ideal way is that in physical implementation rule, during the creation of
> physical operator, it should not care about itself and its child operators'
> physical traits. But this is another different topic.
> Anyway, better than nothing, once it is done, we can provide the option to
> obsolete or disable {{AbstractConverter}}, but still be able to do property
> enforcement.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-3912) Incorrect mapping parsing when properties have same name as reserved keywords in ElasticSearch
[
https://issues.apache.org/jira/browse/CALCITE-3912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated CALCITE-3912:
Labels: pull-request-available (was: )
> Incorrect mapping parsing when properties have same name as reserved keywords
> in ElasticSearch
> --
>
> Key: CALCITE-3912
> URL: https://issues.apache.org/jira/browse/CALCITE-3912
> Project: Calcite
> Issue Type: Improvement
> Components: elasticsearch-adapter
>Reporter: Andrei Sereda
>Assignee: Andrei Sereda
>Priority: Major
> Labels: pull-request-available
>
> If a property has name {{type}} or {{properties}} ES adapter doesn't
> correctly process the
> [mapping|https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html]
> structure.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-3912) Incorrect mapping parsing when properties have same name as reserved keywords in ElasticSearch
[
https://issues.apache.org/jira/browse/CALCITE-3912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrei Sereda updated CALCITE-3912:
---
Description: If a property has name {{type}} or {{properties}} ES adapter
doesn't correctly process the
[mapping|https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html]
structure. (was: If a property has name {{type}} or {{properties}} ES adapter
doesn't process them correctly)
> Incorrect mapping parsing when properties have same name as reserved keywords
> in ElasticSearch
> --
>
> Key: CALCITE-3912
> URL: https://issues.apache.org/jira/browse/CALCITE-3912
> Project: Calcite
> Issue Type: Improvement
> Components: elasticsearch-adapter
>Reporter: Andrei Sereda
>Assignee: Andrei Sereda
>Priority: Major
>
> If a property has name {{type}} or {{properties}} ES adapter doesn't
> correctly process the
> [mapping|https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html]
> structure.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3911) JoinCommuteRule may generate wrong plan for SEMI/ANTI join
Haisheng Yuan created CALCITE-3911: -- Summary: JoinCommuteRule may generate wrong plan for SEMI/ANTI join Key: CALCITE-3911 URL: https://issues.apache.org/jira/browse/CALCITE-3911 Project: Calcite Issue Type: Bug Components: core Reporter: Haisheng Yuan JoinCommuteRule generates wrong plan for SEMI/ANTI join when swapOuter is true. Semi / Anti joins are not swappable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-3913) A formal verification techniques for testing correctness in calcite
Qi Zhou created CALCITE-3913: Summary: A formal verification techniques for testing correctness in calcite Key: CALCITE-3913 URL: https://issues.apache.org/jira/browse/CALCITE-3913 Project: Calcite Issue Type: Wish Reporter: Qi Zhou Hi All, We have developed a technique that can formally be verified if two logical plans in calcite are indeed semantically equivalent. We published this paper in VLDB 2019. Here is the link to the paper +https://www.vldb.org/pvldb/vol12/p1276-zhou.pdf+. This technique converts two logical plan into their symbolic representations and using an SMT (Satisfiability modulo theories) solver to verify the relationship between two symbolic representations to verify the equivalence. We are wondering if it is possible that we can integrate this tool into calcite, as a way to help the correctness testing process in calcite. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-3914) Improve SubsitutionVisitor to consider RexCall of type PLUS and TIMES for canonicalization
Vineet Garg created CALCITE-3914: Summary: Improve SubsitutionVisitor to consider RexCall of type PLUS and TIMES for canonicalization Key: CALCITE-3914 URL: https://issues.apache.org/jira/browse/CALCITE-3914 Project: Calcite Issue Type: Improvement Components: core Reporter: Vineet Garg Assignee: Vineet Garg -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3913) A formal verification techniques for testing correctness in calcite
[ https://issues.apache.org/jira/browse/CALCITE-3913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17081006#comment-17081006 ] Haisheng Yuan commented on CALCITE-3913: Interesting. Looking forward to the contribution. > A formal verification techniques for testing correctness in calcite > --- > > Key: CALCITE-3913 > URL: https://issues.apache.org/jira/browse/CALCITE-3913 > Project: Calcite > Issue Type: Wish >Reporter: Qi Zhou >Priority: Major > > We have developed a technique that can formally be verified if two logical > plans in calcite are indeed semantically equivalent. We published this paper > in VLDB 2019. Here is the [link to the > paper|https://www.vldb.org/pvldb/vol12/p1276-zhou.pdf]. > This technique converts two logical plan into their symbolic representations > and using an SMT (Satisfiability modulo theories) solver to verify the > relationship between two symbolic representations to verify the equivalence. > We are wondering if it is possible that we can integrate this tool into > calcite, as a way to help the correctness testing process in calcite. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-3914) Improve SubsitutionVisitor to consider RexCall of type PLUS and TIMES for canonicalization
[ https://issues.apache.org/jira/browse/CALCITE-3914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated CALCITE-3914: Labels: pull-request-available (was: ) > Improve SubsitutionVisitor to consider RexCall of type PLUS and TIMES for > canonicalization > --- > > Key: CALCITE-3914 > URL: https://issues.apache.org/jira/browse/CALCITE-3914 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-3913) A formal verification techniques for testing correctness in calcite
[ https://issues.apache.org/jira/browse/CALCITE-3913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Hyde updated CALCITE-3913: - Description: We have developed a technique that can formally be verified if two logical plans in calcite are indeed semantically equivalent. We published this paper in VLDB 2019. Here is the [link to the paper|https://www.vldb.org/pvldb/vol12/p1276-zhou.pdf]. This technique converts two logical plan into their symbolic representations and using an SMT (Satisfiability modulo theories) solver to verify the relationship between two symbolic representations to verify the equivalence. We are wondering if it is possible that we can integrate this tool into calcite, as a way to help the correctness testing process in calcite. was: Hi All, We have developed a technique that can formally be verified if two logical plans in calcite are indeed semantically equivalent. We published this paper in VLDB 2019. Here is the link to the paper +https://www.vldb.org/pvldb/vol12/p1276-zhou.pdf+. This technique converts two logical plan into their symbolic representations and using an SMT (Satisfiability modulo theories) solver to verify the relationship between two symbolic representations to verify the equivalence. We are wondering if it is possible that we can integrate this tool into calcite, as a way to help the correctness testing process in calcite. > A formal verification techniques for testing correctness in calcite > --- > > Key: CALCITE-3913 > URL: https://issues.apache.org/jira/browse/CALCITE-3913 > Project: Calcite > Issue Type: Wish >Reporter: Qi Zhou >Priority: Major > > We have developed a technique that can formally be verified if two logical > plans in calcite are indeed semantically equivalent. We published this paper > in VLDB 2019. Here is the [link to the > paper|https://www.vldb.org/pvldb/vol12/p1276-zhou.pdf]. > This technique converts two logical plan into their symbolic representations > and using an SMT (Satisfiability modulo theories) solver to verify the > relationship between two symbolic representations to verify the equivalence. > We are wondering if it is possible that we can integrate this tool into > calcite, as a way to help the correctness testing process in calcite. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3913) A formal verification techniques for testing correctness in calcite
[
https://issues.apache.org/jira/browse/CALCITE-3913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17081007#comment-17081007
]
Julian Hyde commented on CALCITE-3913:
--
What form would the contribution take? Would you add a new test (and test
utilities to support it)? Would you add extra checks to existing tests (say
{{RelOptRulesTest}})?
I am excited to bring some formal methods into Calcite. Even if they are just
used for testing at first, people will be able to see them in use and may be
inspired to apply them in other ways.
> A formal verification techniques for testing correctness in calcite
> ---
>
> Key: CALCITE-3913
> URL: https://issues.apache.org/jira/browse/CALCITE-3913
> Project: Calcite
> Issue Type: Wish
>Reporter: Qi Zhou
>Priority: Major
>
> We have developed a technique that can formally be verified if two logical
> plans in calcite are indeed semantically equivalent. We published this paper
> in VLDB 2019. Here is the [link to the
> paper|https://www.vldb.org/pvldb/vol12/p1276-zhou.pdf].
> This technique converts two logical plan into their symbolic representations
> and using an SMT (Satisfiability modulo theories) solver to verify the
> relationship between two symbolic representations to verify the equivalence.
> We are wondering if it is possible that we can integrate this tool into
> calcite, as a way to help the correctness testing process in calcite.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-3913) Test correctness using formal verification techniques
[ https://issues.apache.org/jira/browse/CALCITE-3913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Hyde updated CALCITE-3913: - Summary: Test correctness using formal verification techniques (was: A formal verification techniques for testing correctness in calcite) > Test correctness using formal verification techniques > - > > Key: CALCITE-3913 > URL: https://issues.apache.org/jira/browse/CALCITE-3913 > Project: Calcite > Issue Type: Wish >Reporter: Qi Zhou >Priority: Major > > We have developed a technique that can formally be verified if two logical > plans in calcite are indeed semantically equivalent. We published this paper > in VLDB 2019. Here is the [link to the > paper|https://www.vldb.org/pvldb/vol12/p1276-zhou.pdf]. > This technique converts two logical plan into their symbolic representations > and using an SMT (Satisfiability modulo theories) solver to verify the > relationship between two symbolic representations to verify the equivalence. > We are wondering if it is possible that we can integrate this tool into > calcite, as a way to help the correctness testing process in calcite. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3893) SQL with GROUP_ID may generate wrong plan
[
https://issues.apache.org/jira/browse/CALCITE-3893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17080789#comment-17080789
]
Julian Hyde commented on CALCITE-3893:
--
bq. But if we allow this pattern, we need to fix the logic of RelToSqlConverter.
Yes.
bq. The runtime behevior of P1 and P2 are different.
But I think you know what I meant.
> SQL with GROUP_ID may generate wrong plan
> -
>
> Key: CALCITE-3893
> URL: https://issues.apache.org/jira/browse/CALCITE-3893
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.22.0
>Reporter: Shuo Cheng
>Assignee: Feng Zhu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.23.0
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Consider the following SQL:
> {code:java}
> select
> deptno, group_id() as g, count(*) as c
> from emp
> group by grouping sets (deptno, (), ())
> {code}
> the plan after SqlToRel is:
> {code:java}
> LogicalUnion(all=[true])
> LogicalProject(DEPTNO=[$0], G=[0:BIGINT], C=[$1])
> LogicalAggregate(group=[{0}], groups=[[{0}, {}]], C=[COUNT()])
> LogicalProject(DEPTNO=[$7])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> LogicalProject(DEPTNO=[$0], G=[1:BIGINT], C=[$1])
> LogicalAggregate(group=[{0}], groups=[[{}]], C=[COUNT()])
> LogicalProject(DEPTNO=[$7])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {code}
> I'm afraid there's some semantic problems here. As `groups` of the second
> Aggregate is empty, then what is `$0` meaning in the Project above it. Maybe
> that we want is:
> {code:java}
> LogicalUnion(all=[true])
> LogicalProject(DEPTNO=[$0], G=[0:BIGINT], C=[$1])
> LogicalAggregate(group=[{0}], groups=[[{0}, {}]], C=[COUNT()])
> LogicalProject(DEPTNO=[$7])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> LogicalProject(DEPTNO=[$0], G=[1:BIGINT], C=[$1])
> LogicalAggregate(group=[{0}], groups=[[{0}]], C=[COUNT()])
> LogicalProject(DEPTNO=[null])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]]){code}
> I noticed this is introduced by CALCITE-1824, cc [~donnyzone],
> [~vladimirsitnikov].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3913) Test correctness using formal verification techniques
[ https://issues.apache.org/jira/browse/CALCITE-3913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17081011#comment-17081011 ] Haisheng Yuan commented on CALCITE-3913: I think one use case is that there are tens of thousands of test queries to test the query optimizer, and it is still increasing. Detecting query equality and duplicate might help reduce tests. > Test correctness using formal verification techniques > - > > Key: CALCITE-3913 > URL: https://issues.apache.org/jira/browse/CALCITE-3913 > Project: Calcite > Issue Type: Wish >Reporter: Qi Zhou >Priority: Major > > We have developed a technique that can formally be verified if two logical > plans in calcite are indeed semantically equivalent. We published this paper > in VLDB 2019. Here is the [link to the > paper|https://www.vldb.org/pvldb/vol12/p1276-zhou.pdf]. > This technique converts two logical plan into their symbolic representations > and using an SMT (Satisfiability modulo theories) solver to verify the > relationship between two symbolic representations to verify the equivalence. > We are wondering if it is possible that we can integrate this tool into > calcite, as a way to help the correctness testing process in calcite. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3913) Test correctness using formal verification techniques
[ https://issues.apache.org/jira/browse/CALCITE-3913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17081040#comment-17081040 ] Qi Zhou commented on CALCITE-3913: -- The first thing we considering to do is to bring this tool to formally verify some of the transformation rules are correct. Basically formally verify the output optimized query plan is equivalent to the original query plan. > Test correctness using formal verification techniques > - > > Key: CALCITE-3913 > URL: https://issues.apache.org/jira/browse/CALCITE-3913 > Project: Calcite > Issue Type: Wish >Reporter: Qi Zhou >Priority: Major > > We have developed a technique that can formally be verified if two logical > plans in calcite are indeed semantically equivalent. We published this paper > in VLDB 2019. Here is the [link to the > paper|https://www.vldb.org/pvldb/vol12/p1276-zhou.pdf]. > This technique converts two logical plan into their symbolic representations > and using an SMT (Satisfiability modulo theories) solver to verify the > relationship between two symbolic representations to verify the equivalence. > We are wondering if it is possible that we can integrate this tool into > calcite, as a way to help the correctness testing process in calcite. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-3915) Add rule listener to report rule attempts and time at DEBUG log level
[
https://issues.apache.org/jira/browse/CALCITE-3915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haisheng Yuan updated CALCITE-3915:
---
Description:
We can add a rule listener to the VolcanoPlanner for recording rule attempts
and the accumulative time for the rule firing. This would be very helpful to
debug planner performance issue (such as CALCITE-2970). The listener will only
be added under DEBUG log level.
It will produce a rule attempt summary like below -
{noformat}
2020-04-10 18:03:24,621 [main] DEBUG -
Rules
Attempts Time (us)
ProjectMergeRule:force_mode
1,203 1,124,242
EnumerableProjectRule(in:NONE,out:ENUMERABLE)
129 226,475
AggregatePullUpConstantsRule
84 1,949
AggregateProjectMergeRule
81 128,575
ProjectFilterTransposeRule
76 8,734
JoinPushExpressionsRule
71 10,314
FilterJoinRule:FilterJoinRule:no-filter
71 1,306
JoinPushThroughJoinRule:right
47 676
JoinPushThroughJoinRule:left
45 175,696
FilterJoinRule:FilterJoinRule:filter
41 89,381
ReduceExpressionsRule(Filter)
24 38,128
EnumerableFilterRule(in:NONE,out:ENUMERABLE)
24 52,457
EnumerableJoinRule(in:NONE,out:ENUMERABLE)
24 71,752
EnumerableMergeJoinRule(in:NONE,out:ENUMERABLE)
24 47,987
FilterProjectTransposeRule
22 55,177
JoinCommuteRule
20 70,240
EnumerableAggregateRule(in:NONE,out:ENUMERABLE)
10 26,522
AggregateExpandDistinctAggregatesRule
10 37
EnumerableTableScanRule(in:NONE,out:ENUMERABLE)
1 527
EnumerableInterpreterRule(in:BINDABLE,out:ENUMERABLE)
1 764
BindableTableScanRule
1 1,710
ExpandConversionRule
1 175
{noformat}
was:
We can add a rule listener to the VolcanoPlanner for recording rule attempts
and the accumulative time for the rule firing. This would be very helpful to
debug planner performance issue (such as CALCITE-2970). The listener will only
be added under DEBUG log level.
It will produce a rule attempt summary like below -
{code:java}
2020-04-10 18:03:24,621 [main] DEBUG -
Rules
Attempts Time (us)
ProjectMergeRule:force_mode
1,203 1,124,242
EnumerableProjectRule(in:NONE,out:ENUMERABLE)
129 226,475
AggregatePullUpConstantsRule
84 1,949
AggregateProjectMergeRule
81 128,575
ProjectFilterTransposeRule
76 8,734
JoinPushExpressionsRule
71 10,314
FilterJoinRule:FilterJoinRule:no-filter
71 1,306
JoinPushThroughJoinRule:right
47 676
JoinPushThroughJoinRule:left
45 175,696
FilterJoinRule:FilterJoinRule:filter
41 89,381
ReduceExpressionsRule(Filter)
24 38,128
EnumerableFilterRule(in:NONE,out:ENUMERABLE)
24 52,457
EnumerableJoinRule(in:NONE,out:ENUMERABLE)
24 71,752
EnumerableMergeJoinRule(in:NONE,out:ENUMERABLE)
24 47,987
FilterProjectTransposeRule
22 55,177
JoinCommuteRule
[jira] [Updated] (CALCITE-3915) Add rule listener to report rule attempts and time at DEBUG log level
[
https://issues.apache.org/jira/browse/CALCITE-3915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haisheng Yuan updated CALCITE-3915:
---
Description:
We can add a rule listener to the VolcanoPlanner for recording rule attempts
and the accumulative time for the rule firing. This would be very helpful to
debug planner performance issue (such as CALCITE-2970). The listener will only
be added under DEBUG log level.
It will produce a rule attempt summary like below -
{code:java}
2020-04-10 18:03:24,621 [main] DEBUG -
Rules
Attempts Time (us)
ProjectMergeRule:force_mode
1,203 1,124,242
EnumerableProjectRule(in:NONE,out:ENUMERABLE)
129 226,475
AggregatePullUpConstantsRule
84 1,949
AggregateProjectMergeRule
81 128,575
ProjectFilterTransposeRule
76 8,734
JoinPushExpressionsRule
71 10,314
FilterJoinRule:FilterJoinRule:no-filter
71 1,306
JoinPushThroughJoinRule:right
47 676
JoinPushThroughJoinRule:left
45 175,696
FilterJoinRule:FilterJoinRule:filter
41 89,381
ReduceExpressionsRule(Filter)
24 38,128
EnumerableFilterRule(in:NONE,out:ENUMERABLE)
24 52,457
EnumerableJoinRule(in:NONE,out:ENUMERABLE)
24 71,752
EnumerableMergeJoinRule(in:NONE,out:ENUMERABLE)
24 47,987
FilterProjectTransposeRule
22 55,177
JoinCommuteRule
20 70,240
EnumerableAggregateRule(in:NONE,out:ENUMERABLE)
10 26,522
AggregateExpandDistinctAggregatesRule
10 37
EnumerableTableScanRule(in:NONE,out:ENUMERABLE)
1 527
EnumerableInterpreterRule(in:BINDABLE,out:ENUMERABLE)
1 764
BindableTableScanRule
1 1,710
ExpandConversionRule
1 175
{code}
was:
We can add a rule listener to the VolcanoPlanner for recording rule attempts
and the accumulative time for the rule firing. This would be very helpful to
debug planner performance issue (such as CALCITE-2970). The listener will only
be added under DEBUG log level.
It will produce a rule attempt summary like below -
2020-04-10 18:03:24,621 [main] DEBUG -
Rules
Attempts Time (us)
ProjectMergeRule:force_mode
1,203 1,124,242
EnumerableProjectRule(in:NONE,out:ENUMERABLE)
129 226,475
AggregatePullUpConstantsRule
84 1,949
AggregateProjectMergeRule
81 128,575
ProjectFilterTransposeRule
76 8,734
JoinPushExpressionsRule
71 10,314
FilterJoinRule:FilterJoinRule:no-filter
71 1,306
JoinPushThroughJoinRule:right
47 676
JoinPushThroughJoinRule:left
45 175,696
FilterJoinRule:FilterJoinRule:filter
41 89,381
ReduceExpressionsRule(Filter)
24 38,128
EnumerableFilterRule(in:NONE,out:ENUMERABLE)
24 52,457
EnumerableJoinRule(in:NONE,out:ENUMERABLE)
24 71,752
EnumerableMergeJoinRule(in:NONE,out:ENUMERABLE)
24 47,987
FilterProjectTransposeRule
22 55,177
JoinCommuteRule
[jira] [Commented] (CALCITE-3915) Add rule listener to report rule attempts and time at DEBUG log level
[
https://issues.apache.org/jira/browse/CALCITE-3915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17081075#comment-17081075
]
Xiening Dai commented on CALCITE-3915:
--
The example in the description is messed up due to the indents. We will have a
pretty output in console.
> Add rule listener to report rule attempts and time at DEBUG log level
> -
>
> Key: CALCITE-3915
> URL: https://issues.apache.org/jira/browse/CALCITE-3915
> Project: Calcite
> Issue Type: Bug
>Reporter: Xiening Dai
>Priority: Minor
>
> We can add a rule listener to the VolcanoPlanner for recording rule attempts
> and the accumulative time for the rule firing. This would be very helpful to
> debug planner performance issue (such as CALCITE-2970). The listener will
> only be added under DEBUG log level.
> It will produce a rule attempt summary like below -
> {noformat}
> 2020-04-10 18:03:24,621 [main] DEBUG -
> Rules
> Attempts Time (us)
> ProjectMergeRule:force_mode
> 1,203 1,124,242
> EnumerableProjectRule(in:NONE,out:ENUMERABLE)
> 129 226,475
> AggregatePullUpConstantsRule
> 84 1,949
> AggregateProjectMergeRule
> 81 128,575
> ProjectFilterTransposeRule
> 76 8,734
> JoinPushExpressionsRule
> 71 10,314
> FilterJoinRule:FilterJoinRule:no-filter
> 71 1,306
> JoinPushThroughJoinRule:right
> 47 676
> JoinPushThroughJoinRule:left
> 45 175,696
> FilterJoinRule:FilterJoinRule:filter
> 41 89,381
> ReduceExpressionsRule(Filter)
> 24 38,128
> EnumerableFilterRule(in:NONE,out:ENUMERABLE)
> 24 52,457
> EnumerableJoinRule(in:NONE,out:ENUMERABLE)
> 24 71,752
> EnumerableMergeJoinRule(in:NONE,out:ENUMERABLE)
> 24 47,987
> FilterProjectTransposeRule
> 22 55,177
> JoinCommuteRule
> 20 70,240
> EnumerableAggregateRule(in:NONE,out:ENUMERABLE)
> 10 26,522
> AggregateExpandDistinctAggregatesRule
> 10 37
> EnumerableTableScanRule(in:NONE,out:ENUMERABLE)
> 1 527
> EnumerableInterpreterRule(in:BINDABLE,out:ENUMERABLE)
> 1 764
> BindableTableScanRule
> 1 1,710
> ExpandConversionRule
> 1 175
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3915) Add rule listener to report rule attempts and time at DEBUG log level
[
https://issues.apache.org/jira/browse/CALCITE-3915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17081109#comment-17081109
]
Chunwei Lei commented on CALCITE-3915:
--
+1 for adding such a listener.
> Add rule listener to report rule attempts and time at DEBUG log level
> -
>
> Key: CALCITE-3915
> URL: https://issues.apache.org/jira/browse/CALCITE-3915
> Project: Calcite
> Issue Type: Bug
>Reporter: Xiening Dai
>Assignee: Xiening Dai
>Priority: Minor
>
> We can add a rule listener to the VolcanoPlanner for recording rule attempts
> and the accumulative time for the rule firing. This would be very helpful to
> debug planner performance issue (such as CALCITE-2970). The listener will
> only be added under DEBUG log level.
> It will produce a rule attempt summary like below -
> {noformat}
> 2020-04-10 18:03:24,621 [main] DEBUG -
> Rules
> Attempts Time (us)
> ProjectMergeRule:force_mode
> 1,203 1,124,242
> EnumerableProjectRule(in:NONE,out:ENUMERABLE)
> 129 226,475
> AggregatePullUpConstantsRule
> 84 1,949
> AggregateProjectMergeRule
> 81 128,575
> ProjectFilterTransposeRule
> 76 8,734
> JoinPushExpressionsRule
> 71 10,314
> FilterJoinRule:FilterJoinRule:no-filter
> 71 1,306
> JoinPushThroughJoinRule:right
> 47 676
> JoinPushThroughJoinRule:left
> 45 175,696
> FilterJoinRule:FilterJoinRule:filter
> 41 89,381
> ReduceExpressionsRule(Filter)
> 24 38,128
> EnumerableFilterRule(in:NONE,out:ENUMERABLE)
> 24 52,457
> EnumerableJoinRule(in:NONE,out:ENUMERABLE)
> 24 71,752
> EnumerableMergeJoinRule(in:NONE,out:ENUMERABLE)
> 24 47,987
> FilterProjectTransposeRule
> 22 55,177
> JoinCommuteRule
> 20 70,240
> EnumerableAggregateRule(in:NONE,out:ENUMERABLE)
> 10 26,522
> AggregateExpandDistinctAggregatesRule
> 10 37
> EnumerableTableScanRule(in:NONE,out:ENUMERABLE)
> 1 527
> EnumerableInterpreterRule(in:BINDABLE,out:ENUMERABLE)
> 1 764
> BindableTableScanRule
> 1 1,710
> ExpandConversionRule
> 1 175
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3913) Test correctness using formal verification techniques
[ https://issues.apache.org/jira/browse/CALCITE-3913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17081108#comment-17081108 ] Chunwei Lei commented on CALCITE-3913: -- Interesting work! I think checking the rule correctness is a good use case for this tool. So maybe we can add a new test to do it. > Test correctness using formal verification techniques > - > > Key: CALCITE-3913 > URL: https://issues.apache.org/jira/browse/CALCITE-3913 > Project: Calcite > Issue Type: Wish >Reporter: Qi Zhou >Priority: Major > > We have developed a technique that can formally be verified if two logical > plans in calcite are indeed semantically equivalent. We published this paper > in VLDB 2019. Here is the [link to the > paper|https://www.vldb.org/pvldb/vol12/p1276-zhou.pdf]. > This technique converts two logical plan into their symbolic representations > and using an SMT (Satisfiability modulo theories) solver to verify the > relationship between two symbolic representations to verify the equivalence. > We are wondering if it is possible that we can integrate this tool into > calcite, as a way to help the correctness testing process in calcite. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3890) Infer IS NOT NULL predicate from join
[
https://issues.apache.org/jira/browse/CALCITE-3890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17081110#comment-17081110
]
Chunwei Lei commented on CALCITE-3890:
--
[~vgarg], [~kgyrtkirk] Instead of using a rule to add ISNOTNULL predicate which
is the way in Hive, I am wondering if we can do it when creating the JOIN
operator. Maybe we can add a configuration to indicate whether to add ISNOTNULL
predicate when creating the JOIN operator using RelBuilder.
> Infer IS NOT NULL predicate from join
> -
>
> Key: CALCITE-3890
> URL: https://issues.apache.org/jira/browse/CALCITE-3890
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Chunwei Lei
>Assignee: Vineet Garg
>Priority: Major
>
> We can infer IS NOT NULL predicate from join which implies some columns may
> not be null. For instance,
>
> {code:java}
> select * from a join b on a.id = b.id;
> {code}
> we can infer a.id is not null and b.id is not null.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Issue Comment Deleted] (CALCITE-3915) Add rule listener to report rule attempts and time at DEBUG log level
[
https://issues.apache.org/jira/browse/CALCITE-3915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiening Dai updated CALCITE-3915:
-
Comment: was deleted
(was: The example in the description is messed up due to the indents. We will
have a pretty output in console.)
> Add rule listener to report rule attempts and time at DEBUG log level
> -
>
> Key: CALCITE-3915
> URL: https://issues.apache.org/jira/browse/CALCITE-3915
> Project: Calcite
> Issue Type: Bug
>Reporter: Xiening Dai
>Priority: Minor
>
> We can add a rule listener to the VolcanoPlanner for recording rule attempts
> and the accumulative time for the rule firing. This would be very helpful to
> debug planner performance issue (such as CALCITE-2970). The listener will
> only be added under DEBUG log level.
> It will produce a rule attempt summary like below -
> {noformat}
> 2020-04-10 18:03:24,621 [main] DEBUG -
> Rules
> Attempts Time (us)
> ProjectMergeRule:force_mode
> 1,203 1,124,242
> EnumerableProjectRule(in:NONE,out:ENUMERABLE)
> 129 226,475
> AggregatePullUpConstantsRule
> 84 1,949
> AggregateProjectMergeRule
> 81 128,575
> ProjectFilterTransposeRule
> 76 8,734
> JoinPushExpressionsRule
> 71 10,314
> FilterJoinRule:FilterJoinRule:no-filter
> 71 1,306
> JoinPushThroughJoinRule:right
> 47 676
> JoinPushThroughJoinRule:left
> 45 175,696
> FilterJoinRule:FilterJoinRule:filter
> 41 89,381
> ReduceExpressionsRule(Filter)
> 24 38,128
> EnumerableFilterRule(in:NONE,out:ENUMERABLE)
> 24 52,457
> EnumerableJoinRule(in:NONE,out:ENUMERABLE)
> 24 71,752
> EnumerableMergeJoinRule(in:NONE,out:ENUMERABLE)
> 24 47,987
> FilterProjectTransposeRule
> 22 55,177
> JoinCommuteRule
> 20 70,240
> EnumerableAggregateRule(in:NONE,out:ENUMERABLE)
> 10 26,522
> AggregateExpandDistinctAggregatesRule
> 10 37
> EnumerableTableScanRule(in:NONE,out:ENUMERABLE)
> 1 527
> EnumerableInterpreterRule(in:BINDABLE,out:ENUMERABLE)
> 1 764
> BindableTableScanRule
> 1 1,710
> ExpandConversionRule
> 1 175
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3915) Add rule listener to report rule attempts and time at DEBUG log level
Xiening Dai created CALCITE-3915: Summary: Add rule listener to report rule attempts and time at DEBUG log level Key: CALCITE-3915 URL: https://issues.apache.org/jira/browse/CALCITE-3915 Project: Calcite Issue Type: Bug Reporter: Xiening Dai We can add a rule listener to the VolcanoPlanner for recording rule attempts and the accumulative time for the rule firing. This would be very helpful to debug planner performance issue (such as CALCITE-2970). The listener will only be added under DEBUG log level. It will produce a rule attempt summary like below - 2020-04-10 18:03:24,621 [main] DEBUG - Rules Attempts Time (us) ProjectMergeRule:force_mode 1,203 1,124,242 EnumerableProjectRule(in:NONE,out:ENUMERABLE) 129 226,475 AggregatePullUpConstantsRule 84 1,949 AggregateProjectMergeRule 81 128,575 ProjectFilterTransposeRule 76 8,734 JoinPushExpressionsRule 71 10,314 FilterJoinRule:FilterJoinRule:no-filter 71 1,306 JoinPushThroughJoinRule:right 47 676 JoinPushThroughJoinRule:left 45 175,696 FilterJoinRule:FilterJoinRule:filter 41 89,381 ReduceExpressionsRule(Filter) 24 38,128 EnumerableFilterRule(in:NONE,out:ENUMERABLE) 24 52,457 EnumerableJoinRule(in:NONE,out:ENUMERABLE) 24 71,752 EnumerableMergeJoinRule(in:NONE,out:ENUMERABLE) 24 47,987 FilterProjectTransposeRule 22 55,177 JoinCommuteRule 20 70,240 EnumerableAggregateRule(in:NONE,out:ENUMERABLE) 10 26,522 AggregateExpandDistinctAggregatesRule 10 37 EnumerableTableScanRule(in:NONE,out:ENUMERABLE) 1 527 EnumerableInterpreterRule(in:BINDABLE,out:ENUMERABLE) 1 764 BindableTableScanRule 1 1,710 ExpandConversionRule 1 175 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (CALCITE-3915) Add rule listener to report rule attempts and time at DEBUG log level
[
https://issues.apache.org/jira/browse/CALCITE-3915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiening Dai reassigned CALCITE-3915:
Assignee: Xiening Dai
> Add rule listener to report rule attempts and time at DEBUG log level
> -
>
> Key: CALCITE-3915
> URL: https://issues.apache.org/jira/browse/CALCITE-3915
> Project: Calcite
> Issue Type: Bug
>Reporter: Xiening Dai
>Assignee: Xiening Dai
>Priority: Minor
>
> We can add a rule listener to the VolcanoPlanner for recording rule attempts
> and the accumulative time for the rule firing. This would be very helpful to
> debug planner performance issue (such as CALCITE-2970). The listener will
> only be added under DEBUG log level.
> It will produce a rule attempt summary like below -
> {noformat}
> 2020-04-10 18:03:24,621 [main] DEBUG -
> Rules
> Attempts Time (us)
> ProjectMergeRule:force_mode
> 1,203 1,124,242
> EnumerableProjectRule(in:NONE,out:ENUMERABLE)
> 129 226,475
> AggregatePullUpConstantsRule
> 84 1,949
> AggregateProjectMergeRule
> 81 128,575
> ProjectFilterTransposeRule
> 76 8,734
> JoinPushExpressionsRule
> 71 10,314
> FilterJoinRule:FilterJoinRule:no-filter
> 71 1,306
> JoinPushThroughJoinRule:right
> 47 676
> JoinPushThroughJoinRule:left
> 45 175,696
> FilterJoinRule:FilterJoinRule:filter
> 41 89,381
> ReduceExpressionsRule(Filter)
> 24 38,128
> EnumerableFilterRule(in:NONE,out:ENUMERABLE)
> 24 52,457
> EnumerableJoinRule(in:NONE,out:ENUMERABLE)
> 24 71,752
> EnumerableMergeJoinRule(in:NONE,out:ENUMERABLE)
> 24 47,987
> FilterProjectTransposeRule
> 22 55,177
> JoinCommuteRule
> 20 70,240
> EnumerableAggregateRule(in:NONE,out:ENUMERABLE)
> 10 26,522
> AggregateExpandDistinctAggregatesRule
> 10 37
> EnumerableTableScanRule(in:NONE,out:ENUMERABLE)
> 1 527
> EnumerableInterpreterRule(in:BINDABLE,out:ENUMERABLE)
> 1 764
> BindableTableScanRule
> 1 1,710
> ExpandConversionRule
> 1 175
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-3914) Improve SubstitutionVisitor to consider RexCall of type PLUS and TIMES for canonicalization
[ https://issues.apache.org/jira/browse/CALCITE-3914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated CALCITE-3914: - Summary: Improve SubstitutionVisitor to consider RexCall of type PLUS and TIMES for canonicalization (was: Improve SubsitutionVisitor to consider RexCall of type PLUS and TIMES for canonicalization ) > Improve SubstitutionVisitor to consider RexCall of type PLUS and TIMES for > canonicalization > > > Key: CALCITE-3914 > URL: https://issues.apache.org/jira/browse/CALCITE-3914 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-3915) Add rule listener to report rule attempts and time at DEBUG log level
[
https://issues.apache.org/jira/browse/CALCITE-3915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated CALCITE-3915:
Labels: pull-request-available (was: )
> Add rule listener to report rule attempts and time at DEBUG log level
> -
>
> Key: CALCITE-3915
> URL: https://issues.apache.org/jira/browse/CALCITE-3915
> Project: Calcite
> Issue Type: Bug
>Reporter: Xiening Dai
>Assignee: Xiening Dai
>Priority: Minor
> Labels: pull-request-available
>
> We can add a rule listener to the VolcanoPlanner for recording rule attempts
> and the accumulative time for the rule firing. This would be very helpful to
> debug planner performance issue (such as CALCITE-2970). The listener will
> only be added under DEBUG log level.
> It will produce a rule attempt summary like below -
> {noformat}
> 2020-04-10 18:03:24,621 [main] DEBUG -
> Rules
> Attempts Time (us)
> ProjectMergeRule:force_mode
> 1,203 1,124,242
> EnumerableProjectRule(in:NONE,out:ENUMERABLE)
> 129 226,475
> AggregatePullUpConstantsRule
> 84 1,949
> AggregateProjectMergeRule
> 81 128,575
> ProjectFilterTransposeRule
> 76 8,734
> JoinPushExpressionsRule
> 71 10,314
> FilterJoinRule:FilterJoinRule:no-filter
> 71 1,306
> JoinPushThroughJoinRule:right
> 47 676
> JoinPushThroughJoinRule:left
> 45 175,696
> FilterJoinRule:FilterJoinRule:filter
> 41 89,381
> ReduceExpressionsRule(Filter)
> 24 38,128
> EnumerableFilterRule(in:NONE,out:ENUMERABLE)
> 24 52,457
> EnumerableJoinRule(in:NONE,out:ENUMERABLE)
> 24 71,752
> EnumerableMergeJoinRule(in:NONE,out:ENUMERABLE)
> 24 47,987
> FilterProjectTransposeRule
> 22 55,177
> JoinCommuteRule
> 20 70,240
> EnumerableAggregateRule(in:NONE,out:ENUMERABLE)
> 10 26,522
> AggregateExpandDistinctAggregatesRule
> 10 37
> EnumerableTableScanRule(in:NONE,out:ENUMERABLE)
> 1 527
> EnumerableInterpreterRule(in:BINDABLE,out:ENUMERABLE)
> 1 764
> BindableTableScanRule
> 1 1,710
> ExpandConversionRule
> 1 175
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3890) Infer IS NOT NULL predicate from join
[
https://issues.apache.org/jira/browse/CALCITE-3890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17081169#comment-17081169
]
Julian Hyde commented on CALCITE-3890:
--
I'm worried about adding a configuration parameter. We don't want to end up
with hundreds of them. If it makes sense, let's just do it.
> Infer IS NOT NULL predicate from join
> -
>
> Key: CALCITE-3890
> URL: https://issues.apache.org/jira/browse/CALCITE-3890
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Chunwei Lei
>Assignee: Vineet Garg
>Priority: Major
>
> We can infer IS NOT NULL predicate from join which implies some columns may
> not be null. For instance,
>
> {code:java}
> select * from a join b on a.id = b.id;
> {code}
> we can infer a.id is not null and b.id is not null.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
