[jira] [Created] (CALCITE-4367) Incorrect documentation for Avatica JSON request/response signatures
John Bodley created CALCITE-4367:
Summary: Incorrect documentation for Avatica JSON request/response
signatures
Key: CALCITE-4367
URL: https://issues.apache.org/jira/browse/CALCITE-4367
Project: Calcite
Issue Type: Bug
Components: avatica
Reporter: John Bodley
I noticed a few inconsistencies between what is documented in the [Avatica JSON
Reference|https://calcite.apache.org/avatica/docs/json_reference.html] and what
the Avatica JDBC driver provides, specifically:
# The {{DatabasePropertyRequest}} was missing the {{connection_id}} field in
the example signature.
# `RpcMetadata` is actually a response as opposed to a miscellaneous type per
[here|https://github.com/apache/calcite-avatica/blob/4b7eee5bf430b916c7c07897b6f60d2b6b6dabb7/core/src/main/protobuf/responses.proto#L114-L116]
and thus requires a {{response}} field. Note I'm not certain if this was
intentional, i.e., it being a response, however it it is it seems that it
should be renamed to {{RpcMetadataResponse}} for consistency.
# The supplied {{ConnectionProperties}} contains an undocumented {{dirty}}
field ({{is_dirty}} for protobuf).
# For the {{SchemasRequest}} the {{catalog}} and {{schemaPattern}} are optional
rather than required.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (CALCITE-4365) SEARCHes in ANDs and ORs cannot be simplified if the complexity of its SARG can actually be reduced to 1
[
https://issues.apache.org/jira/browse/CALCITE-4365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen closed CALCITE-4365.
---
Fix Version/s: 1.27.0
Assignee: Danny Chen
Resolution: Duplicate
Close because it is duplicated.
> SEARCHes in ANDs and ORs cannot be simplified if the complexity of its SARG
> can actually be reduced to 1
>
>
> Key: CALCITE-4365
> URL: https://issues.apache.org/jira/browse/CALCITE-4365
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Caizhi Weng
>Assignee: Danny Chen
>Priority: Major
> Fix For: 1.27.0
>
>
> Consider the following test case in {{RexProgramTest}}.
> {code:java}
> @Test void testSimplifySearch() {
> // (deptno <> 20 OR deptno IS NULL) AND deptno = 10
> // ==>
> // deptno = 10
> final RexNode e =
> and(
> in(vInt(), literal(10), literal(20)),
> eq(vInt(), literal(10)));
> checkSimplify2(e, "=(?0.int0, 10)", "=(?0.int0, 10)");
> }
> {code}
> We expect that {{AND(SEARCH(?0, [10, 20], =(?0, 10)))}} can be simplified to
> {{=(?0, 10)}} but fails. This is because {{RexSimplify#simplifyAnd}} and
> {{RexSimplify#simplifyOrs}} does not deal with the case when the complexity
> of {{SargCollector}} is 1.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4364) "a in (1, 2) and a = 1" should be simplified to "a=1"
[
https://issues.apache.org/jira/browse/CALCITE-4364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17223329#comment-17223329
]
Danny Chen commented on CALCITE-4364:
-
Tested, not fixed by CALCITE-4352 yet.
> "a in (1, 2) and a = 1" should be simplified to "a=1"
> -
>
> Key: CALCITE-4364
> URL: https://issues.apache.org/jira/browse/CALCITE-4364
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.26.0
>Reporter: Danny Chen
>Priority: Major
> Fix For: 1.27.0
>
>
> Check this test in {{RexProgramTest}}:
> {code:java}
> @Test void testSimplifyInAnd() {
> // deptno in (20, 10) and deptno = 10
> // ==>
> // deptno = 10
> final RexNode e =
> and(
> in(vInt(), literal(20), literal(10)),
> eq(vInt(), literal(10)));
> checkSimplify(e, "=(?0.int0, 10)");
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3913) Test correctness using formal verification techniques
[ https://issues.apache.org/jira/browse/CALCITE-3913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17223278#comment-17223278 ] Qi Zhou commented on CALCITE-3913: -- The commit tree in the previous pull request becomes too complicated. I clean up the commit tree and create another pull request with it. https://github.com/apache/calcite/pull/2236 > Test correctness using formal verification techniques > - > > Key: CALCITE-3913 > URL: https://issues.apache.org/jira/browse/CALCITE-3913 > Project: Calcite > Issue Type: Wish >Reporter: Qi Zhou >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > We have developed a technique that can formally be verified if two logical > plans in calcite are indeed semantically equivalent. We published this paper > in VLDB 2019. Here is the [link to the > paper|https://www.vldb.org/pvldb/vol12/p1276-zhou.pdf]. > This technique converts two logical plan into their symbolic representations > and using an SMT (Satisfiability modulo theories) solver to verify the > relationship between two symbolic representations to verify the equivalence. > We are wondering if it is possible that we can integrate this tool into > calcite, as a way to help the correctness testing process in calcite. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (CALCITE-4366) LatticeSuggester should treat UNION / VALUES sub-expressions as tables
[
https://issues.apache.org/jira/browse/CALCITE-4366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Hyde reassigned CALCITE-4366:
Assignee: Julian Hyde
> LatticeSuggester should treat UNION / VALUES sub-expressions as tables
> --
>
> Key: CALCITE-4366
> URL: https://issues.apache.org/jira/browse/CALCITE-4366
> Project: Calcite
> Issue Type: Bug
>Reporter: Julian Hyde
>Assignee: Julian Hyde
>Priority: Major
>
> LatticeSuggester should treat UNION / VALUES sub-expressions as tables, and
> generate a LatticeTable for each.
> If those expressions occur as a common-table expression (CTE; i.e. in the
> WITH clause) the LatticeTable should have the name and column names of that
> CTE.
> For example, given the query
> {code}
> WITH SubDept (id, name) AS (SELECT deptno, dname FROM Dept)
> SELECT *
> FROM Emp AS e
> JOIN SubDept AS d ON e.deptno = d.id
> {code}
> {{LatticeSuggester}} should generate {{LatticeTable(Emp (empno, deptno,
> ename, sal))}} and {{LatticeTable(SubDept (id, name))}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4366) LatticeSuggester should treat UNION / VALUES sub-expressions as tables
Julian Hyde created CALCITE-4366:
Summary: LatticeSuggester should treat UNION / VALUES
sub-expressions as tables
Key: CALCITE-4366
URL: https://issues.apache.org/jira/browse/CALCITE-4366
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
LatticeSuggester should treat UNION / VALUES sub-expressions as tables, and
generate a LatticeTable for each.
If those expressions occur as a common-table expression (CTE; i.e. in the WITH
clause) the LatticeTable should have the name and column names of that CTE.
For example, given the query
{code}
WITH SubDept (id, name) AS (SELECT deptno, dname FROM Dept)
SELECT *
FROM Emp AS e
JOIN SubDept AS d ON e.deptno = d.id
{code}
{{LatticeSuggester}} should generate {{LatticeTable(Emp (empno, deptno, ename,
sal))}} and {{LatticeTable(SubDept (id, name))}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4348) Materialization recognition fails when query has calculation expression
[
https://issues.apache.org/jira/browse/CALCITE-4348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17223175#comment-17223175
]
Julian Hyde commented on CALCITE-4348:
--
bq. By the way ...
Yes, let's get CALCITE-3409 reviewed.
> Materialization recognition fails when query has calculation expression
> ---
>
> Key: CALCITE-4348
> URL: https://issues.apache.org/jira/browse/CALCITE-4348
> Project: Calcite
> Issue Type: Wish
>Reporter: xzh_dz
>Priority: Major
> Attachments: image-2020-10-21-11-40-08-353.png
>
>
> The following example should be recognized by materialized views, but failed.
> {code:java}
> // code placeholder
> org.apache.calcite.test.MaterializedViewSubstitutionVisitorTest
> @Test void testAggregateComplexExpression() {
> String mv = ""
> + "select \"deptno\", \"salary\", count(\"commission\")\n"
> + "from \"emps\"\n"
> + "group by \"deptno\", \"salary\"";
> String query = ""
> + "select \"deptno\" + \"salary\", count(\"commission\")\n"
> + "from \"emps\"\n"
> + "group by \"deptno\" + \"salary\"";
> sql(mv, query).ok();
> }
> {code}
> {code:java}
> Query:
> Aggregate(groupSet: {0}, groupSets: [{0}], calls: [COUNT($1)])
> Calc(program: (expr#0..4=[{inputs}], expr#5=[+($t1, $t3)], $f0=[$t5],
> commission=[$t4]))
> Scan(table: [hr, emps])
> mv:
> Aggregate(groupSet: {0, 1}, groupSets: [{0, 1}], calls: [COUNT($2)])
> Calc(program: (expr#0..4=[{inputs}], deptno=[$t1], salary=[$t3],
> commission=[$t4]))
> Scan(table: [hr, emps]){code}
> Exception:
> !image-2020-10-21-11-40-08-353.png|width=606,height=223!
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3409) Add an interface in MaterializedViewSubstitutionVisitor to allow registering UnifyRule
[
https://issues.apache.org/jira/browse/CALCITE-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17223174#comment-17223174
]
Julian Hyde commented on CALCITE-3409:
--
[~jcamachorodriguez] Can you please review this PR?
> Add an interface in MaterializedViewSubstitutionVisitor to allow registering
> UnifyRule
> --
>
> Key: CALCITE-3409
> URL: https://issues.apache.org/jira/browse/CALCITE-3409
> Project: Calcite
> Issue Type: Improvement
>Reporter: Jin Xing
>Assignee: Jin Xing
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> In current code of MaterializedViewSubstitutionVisitor, all matching rules
> are internal defined. The existing rules support the most popular scenarios.
> But my customers sometimes ask for the ability to self define some matching
> rules, thus to support some special scenarios.
> I take below example as an illustration:
> {code:java}
> Query:
> select * from table
> where from_unixtime(_EVENT_TIME_, "mmdd hh") >= "20190909 00"
> and from_unixtime(_EVENT_TIME_, "mmdd hh") <= "20190909 23" ;
> Materialized View:
> select * from table
> where from_unixtime(_EVENT_TIME_, "mmdd") = "20190909";{code}
> It's hard to enumerate the matching pattern for different functions in
> internal matching rules. We can expose a method to register new UnifyRules
> and allow user to extend the ability of MV matching
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4348) Materialization recognition fails when query has calculation expression
[
https://issues.apache.org/jira/browse/CALCITE-4348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17223171#comment-17223171
]
Julian Hyde commented on CALCITE-4348:
--
{quote}It seems like that can not preserves line breaks
{quote}
That is not true. Maybe you need to be in 'Text' mode (rather than 'Visual')
when you paste. Here is a case that has an error stack: CALCITE-3929.
> Materialization recognition fails when query has calculation expression
> ---
>
> Key: CALCITE-4348
> URL: https://issues.apache.org/jira/browse/CALCITE-4348
> Project: Calcite
> Issue Type: Wish
>Reporter: xzh_dz
>Priority: Major
> Attachments: image-2020-10-21-11-40-08-353.png
>
>
> The following example should be recognized by materialized views, but failed.
> {code:java}
> // code placeholder
> org.apache.calcite.test.MaterializedViewSubstitutionVisitorTest
> @Test void testAggregateComplexExpression() {
> String mv = ""
> + "select \"deptno\", \"salary\", count(\"commission\")\n"
> + "from \"emps\"\n"
> + "group by \"deptno\", \"salary\"";
> String query = ""
> + "select \"deptno\" + \"salary\", count(\"commission\")\n"
> + "from \"emps\"\n"
> + "group by \"deptno\" + \"salary\"";
> sql(mv, query).ok();
> }
> {code}
> {code:java}
> Query:
> Aggregate(groupSet: {0}, groupSets: [{0}], calls: [COUNT($1)])
> Calc(program: (expr#0..4=[{inputs}], expr#5=[+($t1, $t3)], $f0=[$t5],
> commission=[$t4]))
> Scan(table: [hr, emps])
> mv:
> Aggregate(groupSet: {0, 1}, groupSets: [{0, 1}], calls: [COUNT($2)])
> Calc(program: (expr#0..4=[{inputs}], deptno=[$t1], salary=[$t3],
> commission=[$t4]))
> Scan(table: [hr, emps]){code}
> Exception:
> !image-2020-10-21-11-40-08-353.png|width=606,height=223!
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4361) BigQuery comments starting with # cause SqlParseException
[
https://issues.apache.org/jira/browse/CALCITE-4361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17223166#comment-17223166
]
Julian Hyde commented on CALCITE-4361:
--
Can you please check whether '#' starts a comment if it is not followed by
space? E.g.
{code:java}
select 1, #no space after pound
2, # space after pound
3 ### multiple pounds followed by space{code}
> BigQuery comments starting with # cause SqlParseException
> -
>
> Key: CALCITE-4361
> URL: https://issues.apache.org/jira/browse/CALCITE-4361
> Project: Calcite
> Issue Type: Bug
> Components: babel
>Reporter: Justin Swett
>Priority: Minor
>
> Unable to parse big query sql statements that use #.
>
> {code:java}
> @Test void testBigQueryHashtagComments(){
> final String s = "select * from \n" +
> " (select 1 as foo)\n" +
> " union all\n" +
> " # (select 2)\n" +
> " (select 3) # something useful to be said about this line";
> sql(s).withConformance(SqlConformanceEnum.BIG_QUERY).ok();
> }
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (CALCITE-4362) Unable to parse queries that use BigQuery pseudo columns _PARTITIONDATE/_PARTITIONTIME
[
https://issues.apache.org/jira/browse/CALCITE-4362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Hyde reassigned CALCITE-4362:
Assignee: Julian Hyde
> Unable to parse queries that use BigQuery pseudo columns
> _PARTITIONDATE/_PARTITIONTIME
> --
>
> Key: CALCITE-4362
> URL: https://issues.apache.org/jira/browse/CALCITE-4362
> Project: Calcite
> Issue Type: Bug
> Components: babel
>Reporter: Justin Swett
>Assignee: Julian Hyde
>Priority: Major
>
> BigQuery supports partitioned tables using pseudo columns "_PARTITIONDATE"
> e.g. The following table has two columns you can select from but also
> partitions the data by creation date
> {code:java}
> CREATE or REPLACE TABLE #{table} (
> transaction_id STRING,
> order_date DATE )
> PARTITION BY _PARTITIONDATE
> {code}
>
> The following queries cannot be validated
> {code:java}
> select * from #{table} where _PARTITIONDATE = '2020-11-03' ;
> -- OR
> select * from #{table} where _PARTITIONTIME = '2020-11-03' ;{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4362) Unable to parse queries that use BigQuery pseudo columns _PARTITIONDATE/_PARTITIONTIME
[
https://issues.apache.org/jira/browse/CALCITE-4362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17223154#comment-17223154
]
Julian Hyde commented on CALCITE-4362:
--
The solution is probably to make {{_PARTITIONDATE}} and {{_PARTITIONTIME}} into
system columns. Specifically:
* When you ask a table 'Is {{_PARTITIONDATE}} one of your columns', the table
will say 'yes'.
* When you ask a table 'tell me your columns' (e.g. when expanding '*') it
will not mention {{_PARTITIONDATE}}.
Oracle's {{ROWID}} is similar.
> Unable to parse queries that use BigQuery pseudo columns
> _PARTITIONDATE/_PARTITIONTIME
> --
>
> Key: CALCITE-4362
> URL: https://issues.apache.org/jira/browse/CALCITE-4362
> Project: Calcite
> Issue Type: Bug
> Components: babel
>Reporter: Justin Swett
>Priority: Major
>
> BigQuery supports partitioned tables using pseudo columns "_PARTITIONDATE"
> e.g. The following table has two columns you can select from but also
> partitions the data by creation date
> {code:java}
> CREATE or REPLACE TABLE #{table} (
> transaction_id STRING,
> order_date DATE )
> PARTITION BY _PARTITIONDATE
> {code}
>
> The following queries cannot be validated
> {code:java}
> select * from #{table} where _PARTITIONDATE = '2020-11-03' ;
> -- OR
> select * from #{table} where _PARTITIONTIME = '2020-11-03' ;{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4363) Need a utility to check if a SQL operator is standard
[ https://issues.apache.org/jira/browse/CALCITE-4363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17223122#comment-17223122 ] Julian Hyde commented on CALCITE-4363: -- Can you clarify what you mean by 'standard'? We have the notion of 'function tables' (e.g. a connection might use the built-in functions plus the Oracle functions) and so it would make sense to check whether a function is in the current function table. Somewhat related: SqlStdOperatorTable contains 'internal functions' (that cannot be represented in SQL, and are therefore never created by the parser). I have been thinking for a while that these should be moved out. > Need a utility to check if a SQL operator is standard > - > > Key: CALCITE-4363 > URL: https://issues.apache.org/jira/browse/CALCITE-4363 > Project: Calcite > Issue Type: Wish > Components: core >Reporter: Liya Fan >Assignee: Liya Fan >Priority: Major > > Currently, all standard SQL operators are defined in SqlStdOperatorTable.java > as static members, not as enums. This makes it difficult to check if a SQL > function is a standard one. > In many scenarios, we need such a check. For example, some optimizations can > be performed, if we know the function is standard (not user-defined), because > user-defined functions may have some unknown side-effects. > To support the check, we only need to maintain a set that contains all > standard operators. > Do you think such a utility should be supported? > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4364) "a in (1, 2) and a = 1" should be simplified to "a=1"
[
https://issues.apache.org/jira/browse/CALCITE-4364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17223109#comment-17223109
]
Julian Hyde commented on CALCITE-4364:
--
Might be fixed by CALCITE-4352. Can you test against the PR?
> "a in (1, 2) and a = 1" should be simplified to "a=1"
> -
>
> Key: CALCITE-4364
> URL: https://issues.apache.org/jira/browse/CALCITE-4364
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.26.0
>Reporter: Danny Chen
>Priority: Major
> Fix For: 1.27.0
>
>
> Check this test in {{RexProgramTest}}:
> {code:java}
> @Test void testSimplifyInAnd() {
> // deptno in (20, 10) and deptno = 10
> // ==>
> // deptno = 10
> final RexNode e =
> and(
> in(vInt(), literal(20), literal(10)),
> eq(vInt(), literal(10)));
> checkSimplify(e, "=(?0.int0, 10)");
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4365) SEARCHes in ANDs and ORs cannot be simplified if the complexity of its SARG can actually be reduced to 1
[
https://issues.apache.org/jira/browse/CALCITE-4365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17223108#comment-17223108
]
Julian Hyde commented on CALCITE-4365:
--
Can you check my PR for CALCITE-4352 and see whether it fixes it? Like you, I
observed that we should keep even simple Sargs as Sargs during simplification,
because it allows them to combine with other (perhaps more complex) Sargs.
> SEARCHes in ANDs and ORs cannot be simplified if the complexity of its SARG
> can actually be reduced to 1
>
>
> Key: CALCITE-4365
> URL: https://issues.apache.org/jira/browse/CALCITE-4365
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Caizhi Weng
>Priority: Major
>
> Consider the following test case in {{RexProgramTest}}.
> {code:java}
> @Test void testSimplifySearch() {
> // (deptno <> 20 OR deptno IS NULL) AND deptno = 10
> // ==>
> // deptno = 10
> final RexNode e =
> and(
> in(vInt(), literal(10), literal(20)),
> eq(vInt(), literal(10)));
> checkSimplify2(e, "=(?0.int0, 10)", "=(?0.int0, 10)");
> }
> {code}
> We expect that {{AND(SEARCH(?0, [10, 20], =(?0, 10)))}} can be simplified to
> {{=(?0, 10)}} but fails. This is because {{RexSimplify#simplifyAnd}} and
> {{RexSimplify#simplifyOrs}} does not deal with the case when the complexity
> of {{SargCollector}} is 1.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4365) SEARCHes in ANDs and ORs cannot be simplified if the complexity of its SARG can actually be reduced to 1
Caizhi Weng created CALCITE-4365:
Summary: SEARCHes in ANDs and ORs cannot be simplified if the
complexity of its SARG can actually be reduced to 1
Key: CALCITE-4365
URL: https://issues.apache.org/jira/browse/CALCITE-4365
Project: Calcite
Issue Type: Bug
Components: core
Reporter: Caizhi Weng
Consider the following test case in {{RexProgramTest}}.
{code:java}
@Test void testSimplifySearch() {
// (deptno <> 20 OR deptno IS NULL) AND deptno = 10
// ==>
// deptno = 10
final RexNode e =
and(
in(vInt(), literal(10), literal(20)),
eq(vInt(), literal(10)));
checkSimplify2(e, "=(?0.int0, 10)", "=(?0.int0, 10)");
}
{code}
We expect that {{AND(SEARCH(?0, [10, 20], =(?0, 10)))}} can be simplified to
{{=(?0, 10)}} but fails. This is because {{RexSimplify#simplifyAnd}} and
{{RexSimplify#simplifyOrs}} does not deal with the case when the complexity of
{{SargCollector}} is 1.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4358) Potential memory leak in VolcanoPlanner#materializations due to circular reference
[ https://issues.apache.org/jira/browse/CALCITE-4358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17222810#comment-17222810 ] Stamatis Zampetakis commented on CALCITE-4358: -- I don't doubt there is a circular reference; there are many of those in Java APPs in general. The question is which is the part (GC root) that prevents these objects from garbage collection. I am not sure if there is something visible in the screenshot; what is this thread? why does it hold a RelOptMaterialization? > Potential memory leak in VolcanoPlanner#materializations due to circular > reference > -- > > Key: CALCITE-4358 > URL: https://issues.apache.org/jira/browse/CALCITE-4358 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: Jiatao Tao >Assignee: Jiatao Tao >Priority: Major > Attachments: screenshot-1.png > > > Hi fellows > We use Calcite's materialization view in our product env, and it OOM times, > I dig the memory dump and found there are all of RelOptMaterialization > instance. > > We will call Planner#clear before every query and will new a new Planner > just as org.apache.calcite.prepare.CalcitePrepareImpl did, and inside the > dump, there are lots of planner instance, so I suspect it is a circular > reference that JVM can not GC them, here's the chain: > > 1. org.apache.calcite.plan.volcano.VolcanoPlanner#materializations: > List > 2. org.apache.calcite.plan.RelOptMaterialization#queryRel: AbstractRelNode > 3. org.apache.calcite.rel.AbstractRelNode#cluster: RelOptCluster > 4. org.apache.calcite.plan.RelOptCluster#planner: VolcanoPlanner > 5. org.apache.calcite.plan.volcano.VolcanoPlanner#materializations: > List -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-4364) "a in (1, 2) and a = 1" should be simplified to "a=1"
[
https://issues.apache.org/jira/browse/CALCITE-4364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen updated CALCITE-4364:
Description:
Check this test in {{RexProgramTest}}:
{code:java}
@Test void testSimplifyInAnd() {
// deptno in (20, 10) and deptno = 10
// ==>
// deptno = 10
final RexNode e =
and(
in(vInt(), literal(20), literal(10)),
eq(vInt(), literal(10)));
checkSimplify(e, "=(?0.int0, 10)");
}
{code}
was:
Check this test in {{RexProgramTest}}:
```java
@Test void testSimplifyInAnd() {
// deptno in (20, 10) and deptno = 10
// ==>
// deptno = 10
final RexNode e =
and(
in(vInt(), literal(20), literal(10)),
eq(vInt(), literal(10)));
checkSimplify(e, "=(?0.int0, 10)");
}
```
> "a in (1, 2) and a = 1" should be simplified to "a=1"
> -
>
> Key: CALCITE-4364
> URL: https://issues.apache.org/jira/browse/CALCITE-4364
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.26.0
>Reporter: Danny Chen
>Priority: Major
> Fix For: 1.27.0
>
>
> Check this test in {{RexProgramTest}}:
> {code:java}
> @Test void testSimplifyInAnd() {
> // deptno in (20, 10) and deptno = 10
> // ==>
> // deptno = 10
> final RexNode e =
> and(
> in(vInt(), literal(20), literal(10)),
> eq(vInt(), literal(10)));
> checkSimplify(e, "=(?0.int0, 10)");
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4364) "a in (1, 2) and a = 1" should be simplified to "a=1"
Danny Chen created CALCITE-4364:
---
Summary: "a in (1, 2) and a = 1" should be simplified to "a=1"
Key: CALCITE-4364
URL: https://issues.apache.org/jira/browse/CALCITE-4364
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.26.0
Reporter: Danny Chen
Fix For: 1.27.0
Check this test in {{RexProgramTest}}:
```java
@Test void testSimplifyInAnd() {
// deptno in (20, 10) and deptno = 10
// ==>
// deptno = 10
final RexNode e =
and(
in(vInt(), literal(20), literal(10)),
eq(vInt(), literal(10)));
checkSimplify(e, "=(?0.int0, 10)");
}
```
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-4354) Allow ITEM operator on ROW/STRUCT data types
[
https://issues.apache.org/jira/browse/CALCITE-4354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stamatis Zampetakis updated CALCITE-4354:
-
Summary: Allow ITEM operator on ROW/STRUCT data types (was: add support
for ROW/STRUCT data types to ITEM operator)
> Allow ITEM operator on ROW/STRUCT data types
>
>
> Key: CALCITE-4354
> URL: https://issues.apache.org/jira/browse/CALCITE-4354
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.27.0
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.27.0
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> The current implementation of the "ITEM" operator cannot be apply to
> row/struct type.
> The idea is to extend "ITEM" to support a signature
> "[]", allowing to access elements by their position
> (1-based index), or their name (if available).
>
> The current implementation supports an SQL query as follows:
> {code:java}
> select au."birthPlace"."city" as city from "bookstore"."authors" au;{code}
>
> The extended implementation will also support the following equivalent query:
> {code:java}
> select au."birthPlace"['city'] as city from "bookstore"."authors" au;
> {code}
>
> And since we have that "birthPlace(coords, city, country)", we can also
> access "city" with its index 2:
> {code:java}
> select au."birthPlace"[2] as city from "bookstore"."authors" au;{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-4354) add support for ROW/STRUCT data types to ITEM operator
[
https://issues.apache.org/jira/browse/CALCITE-4354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stamatis Zampetakis updated CALCITE-4354:
-
Fix Version/s: 1.27.0
> add support for ROW/STRUCT data types to ITEM operator
> --
>
> Key: CALCITE-4354
> URL: https://issues.apache.org/jira/browse/CALCITE-4354
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.27.0
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.27.0
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> The current implementation of the "ITEM" operator cannot be apply to
> row/struct type.
> The idea is to extend "ITEM" to support a signature
> "[]", allowing to access elements by their position
> (1-based index), or their name (if available).
>
> The current implementation supports an SQL query as follows:
> {code:java}
> select au."birthPlace"."city" as city from "bookstore"."authors" au;{code}
>
> The extended implementation will also support the following equivalent query:
> {code:java}
> select au."birthPlace"['city'] as city from "bookstore"."authors" au;
> {code}
>
> And since we have that "birthPlace(coords, city, country)", we can also
> access "city" with its index 2:
> {code:java}
> select au."birthPlace"[2] as city from "bookstore"."authors" au;{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-4360) Apply SubstitutionRule first in top-down driven rule apply
[ https://issues.apache.org/jira/browse/CALCITE-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17222756#comment-17222756 ] Chunwei Lei edited comment on CALCITE-4360 at 10/29/20, 7:51 AM: - BTW, I found all the test cases in TopDownOptTest failed if applying the substitution rule last in the investigation and It's supposed not to be. I would continue to investigate it and file another issue if there is something wrong. was (Author: chunwei lei): BTW, I found all the test cases in TopDownOptTest failed if applying the substitution rule last in the investigation and It's supposed not to be. I would continue to investigate it. > Apply SubstitutionRule first in top-down driven rule apply > -- > > Key: CALCITE-4360 > URL: https://issues.apache.org/jira/browse/CALCITE-4360 > Project: Calcite > Issue Type: Improvement >Reporter: Chunwei Lei >Assignee: Chunwei Lei >Priority: Major > Labels: pull-request-available > Attachments: image-2020-10-27-21-55-55-155.png > > Time Spent: 1h 40m > Remaining Estimate: 0h > > In the current implementation, TopDownRuleQueue adds substitute rules in the > end wrongly. The SubstitutionRule should be executed first. > !image-2020-10-27-21-55-55-155.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4360) Apply SubstitutionRule first in top-down driven rule apply
[ https://issues.apache.org/jira/browse/CALCITE-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17222756#comment-17222756 ] Chunwei Lei commented on CALCITE-4360: -- BTW, I found all the test cases in TopDownOptTest failed if applying the substitution rule last in the investigation and It's supposed not to be. I would continue to investigate it. > Apply SubstitutionRule first in top-down driven rule apply > -- > > Key: CALCITE-4360 > URL: https://issues.apache.org/jira/browse/CALCITE-4360 > Project: Calcite > Issue Type: Improvement >Reporter: Chunwei Lei >Assignee: Chunwei Lei >Priority: Major > Labels: pull-request-available > Attachments: image-2020-10-27-21-55-55-155.png > > Time Spent: 1h 40m > Remaining Estimate: 0h > > In the current implementation, TopDownRuleQueue adds substitute rules in the > end wrongly. The SubstitutionRule should be executed first. > !image-2020-10-27-21-55-55-155.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4360) Apply SubstitutionRule first in top-down driven rule apply
[ https://issues.apache.org/jira/browse/CALCITE-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17222749#comment-17222749 ] Chunwei Lei commented on CALCITE-4360: -- [~fan_li_ya] Makes sense. > Apply SubstitutionRule first in top-down driven rule apply > -- > > Key: CALCITE-4360 > URL: https://issues.apache.org/jira/browse/CALCITE-4360 > Project: Calcite > Issue Type: Improvement >Reporter: Chunwei Lei >Assignee: Chunwei Lei >Priority: Major > Labels: pull-request-available > Attachments: image-2020-10-27-21-55-55-155.png > > Time Spent: 1h 40m > Remaining Estimate: 0h > > In the current implementation, TopDownRuleQueue adds substitute rules in the > end wrongly. The SubstitutionRule should be executed first. > !image-2020-10-27-21-55-55-155.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-4358) Potential memory leak in VolcanoPlanner#materializations due to circular reference
[ https://issues.apache.org/jira/browse/CALCITE-4358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiatao Tao updated CALCITE-4358: Description: Hi fellows We use Calcite's materialization view in our product env, and it OOM times, I dig the memory dump and found there are all of RelOptMaterialization instance. We will call Planner#clear before every query and will new a new Planner just as org.apache.calcite.prepare.CalcitePrepareImpl did, and inside the dump, there are lots of planner instance, so I suspect it is a circular reference that JVM can not GC them, here's the chain: 1. org.apache.calcite.plan.volcano.VolcanoPlanner#materializations: List 2. org.apache.calcite.plan.RelOptMaterialization#queryRel: AbstractRelNode 3. org.apache.calcite.rel.AbstractRelNode#cluster: RelOptCluster 4. org.apache.calcite.plan.RelOptCluster#planner: VolcanoPlanner 5. org.apache.calcite.plan.volcano.VolcanoPlanner#materializations: List was: Hi fellows We use Calcite's materialization view in our product env, and it OOM times, I dig the memory dump and found there are all of RelOptMaterialization instance. We will call Planner#clear before every query and will new a new Planner just as org.apache.calcite.prepare.CalcitePrepareImpl did, and inside the dump, there are lots of planner instance, so I suspect it is a circular reference that JVM can not GC them, here's the chain: 1. org.apache.calcite.plan.volcano.VolcanoPlanner#materializations: List 2. org.apache.calcite.plan.RelOptMaterialization#queryRel: AbstractRelNode 3. org.apache.calcite.rel.AbstractRelNode#cluster: RelOptCluster 4. org.apache.calcite.plan.RelOptCluster#planner: VolcanoPlanner 5. org.apache.calcite.plan.volcano.VolcanoPlanner#materializations: List > Potential memory leak in VolcanoPlanner#materializations due to circular > reference > -- > > Key: CALCITE-4358 > URL: https://issues.apache.org/jira/browse/CALCITE-4358 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: Jiatao Tao >Assignee: Jiatao Tao >Priority: Major > Attachments: screenshot-1.png > > > Hi fellows > We use Calcite's materialization view in our product env, and it OOM times, > I dig the memory dump and found there are all of RelOptMaterialization > instance. > > We will call Planner#clear before every query and will new a new Planner > just as org.apache.calcite.prepare.CalcitePrepareImpl did, and inside the > dump, there are lots of planner instance, so I suspect it is a circular > reference that JVM can not GC them, here's the chain: > > 1. org.apache.calcite.plan.volcano.VolcanoPlanner#materializations: > List > 2. org.apache.calcite.plan.RelOptMaterialization#queryRel: AbstractRelNode > 3. org.apache.calcite.rel.AbstractRelNode#cluster: RelOptCluster > 4. org.apache.calcite.plan.RelOptCluster#planner: VolcanoPlanner > 5. org.apache.calcite.plan.volcano.VolcanoPlanner#materializations: > List -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4358) Potential memory leak in VolcanoPlanner#materializations due to circular reference
[ https://issues.apache.org/jira/browse/CALCITE-4358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17222746#comment-17222746 ] Jiatao Tao commented on CALCITE-4358: - Can this be the evidence? I show the incoming objects, the call trace is: 1. org.apache.calcite.plan.volcano.VolcanoPlanner#materializations: List 2. org.apache.calcite.plan.RelOptMaterialization#queryRel: AbstractRelNode 3. org.apache.calcite.rel.AbstractRelNode#cluster: RelOptCluster 4. org.apache.calcite.plan.RelOptCluster#planner: VolcanoPlanner 5. org.apache.calcite.plan.volcano.VolcanoPlanner#materializations: List !screenshot-1.png|width=735,height=573! > Potential memory leak in VolcanoPlanner#materializations due to circular > reference > -- > > Key: CALCITE-4358 > URL: https://issues.apache.org/jira/browse/CALCITE-4358 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: Jiatao Tao >Assignee: Jiatao Tao >Priority: Major > Attachments: screenshot-1.png > > > Hi fellows > We use Calcite's materialization view in our product env, and it OOM times, > I dig the memory dump and found there are all of RelOptMaterialization > instance. > > We will call Planner#clear before every query and will new a new Planner > just as org.apache.calcite.prepare.CalcitePrepareImpl did, and inside the > dump, there are lots of planner instance, so I suspect it is a circular > reference that JVM can not GC them, here's the chain: > > 1. org.apache.calcite.plan.volcano.VolcanoPlanner#materializations: > List > 2. org.apache.calcite.plan.RelOptMaterialization#queryRel: AbstractRelNode > 3. org.apache.calcite.rel.AbstractRelNode#cluster: RelOptCluster > 4. org.apache.calcite.plan.RelOptCluster#planner: VolcanoPlanner > 5. org.apache.calcite.plan.volcano.VolcanoPlanner#materializations: > List -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-4358) Potential memory leak in VolcanoPlanner#materializations due to circular reference
[ https://issues.apache.org/jira/browse/CALCITE-4358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiatao Tao updated CALCITE-4358: Attachment: screenshot-1.png > Potential memory leak in VolcanoPlanner#materializations due to circular > reference > -- > > Key: CALCITE-4358 > URL: https://issues.apache.org/jira/browse/CALCITE-4358 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: Jiatao Tao >Assignee: Jiatao Tao >Priority: Major > Attachments: screenshot-1.png > > > Hi fellows > We use Calcite's materialization view in our product env, and it OOM times, I > dig the memory dump and found there are all of RelOptMaterialization instance. > > We will call Planner#clear before every query and will new a new Planner just > as org.apache.calcite.prepare.CalcitePrepareImpl did, and inside the dump, > there are lots of planner instance, so I suspect it is a circular reference > that JVM can not GC them, here's the chain: > > 1. org.apache.calcite.plan.volcano.VolcanoPlanner#materializations: > List 2. > org.apache.calcite.plan.RelOptMaterialization#queryRel: AbstractRelNode 3. > org.apache.calcite.rel.AbstractRelNode#cluster: RelOptCluster 4. > org.apache.calcite.plan.RelOptCluster#planner: VolcanoPlanner 5. > org.apache.calcite.plan.volcano.VolcanoPlanner#materializations: > List -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4360) Apply SubstitutionRule first in top-down driven rule apply
[ https://issues.apache.org/jira/browse/CALCITE-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17222742#comment-17222742 ] Liya Fan commented on CALCITE-4360: --- [~Chunwei Lei] Thanks for the investigation. The process is a little tricky. However, I wish we could add some comments to the related code, so the code reader will no longer be confused. > Apply SubstitutionRule first in top-down driven rule apply > -- > > Key: CALCITE-4360 > URL: https://issues.apache.org/jira/browse/CALCITE-4360 > Project: Calcite > Issue Type: Improvement >Reporter: Chunwei Lei >Assignee: Chunwei Lei >Priority: Major > Labels: pull-request-available > Attachments: image-2020-10-27-21-55-55-155.png > > Time Spent: 1h 40m > Remaining Estimate: 0h > > In the current implementation, TopDownRuleQueue adds substitute rules in the > end wrongly. The SubstitutionRule should be executed first. > !image-2020-10-27-21-55-55-155.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CALCITE-4360) Apply SubstitutionRule first in top-down driven rule apply
[ https://issues.apache.org/jira/browse/CALCITE-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chunwei Lei resolved CALCITE-4360. -- Resolution: Invalid > Apply SubstitutionRule first in top-down driven rule apply > -- > > Key: CALCITE-4360 > URL: https://issues.apache.org/jira/browse/CALCITE-4360 > Project: Calcite > Issue Type: Improvement >Reporter: Chunwei Lei >Assignee: Chunwei Lei >Priority: Major > Labels: pull-request-available > Attachments: image-2020-10-27-21-55-55-155.png > > Time Spent: 1.5h > Remaining Estimate: 0h > > In the current implementation, TopDownRuleQueue adds substitute rules in the > end wrongly. The SubstitutionRule should be executed first. > !image-2020-10-27-21-55-55-155.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4360) Apply SubstitutionRule first in top-down driven rule apply
[ https://issues.apache.org/jira/browse/CALCITE-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17222730#comment-17222730 ] Chunwei Lei commented on CALCITE-4360: -- I am sorry that I misunderstood the codes. Currently, the substitution rule is already applied first though it is not so straightforward. The process looks like: 1.put the non-substitution rule at the front and substitution rule at the end of the queue 2.get each rule from the queue in order from first to last and generate an ApplyRule task 3.push the ApplyRule task into the stack As you can see, the ApplyRule(substitution) task is poped earlier than the ApplyRule(non-substitution) task from the stack. Thus substitution rule is executed first. > Apply SubstitutionRule first in top-down driven rule apply > -- > > Key: CALCITE-4360 > URL: https://issues.apache.org/jira/browse/CALCITE-4360 > Project: Calcite > Issue Type: Improvement >Reporter: Chunwei Lei >Assignee: Chunwei Lei >Priority: Major > Labels: pull-request-available > Attachments: image-2020-10-27-21-55-55-155.png > > Time Spent: 1h 20m > Remaining Estimate: 0h > > In the current implementation, TopDownRuleQueue adds substitute rules in the > end wrongly. The SubstitutionRule should be executed first. > !image-2020-10-27-21-55-55-155.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-4277) When rel has been removed from its subset, skip the origin rule match (Jiatao Tao)
[ https://issues.apache.org/jira/browse/CALCITE-4277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiatao Tao updated CALCITE-4277: Summary: When rel has been removed from its subset, skip the origin rule match (Jiatao Tao) (was: skip rule match when rel's relset is obsolete) > When rel has been removed from its subset, skip the origin rule match (Jiatao > Tao) > -- > > Key: CALCITE-4277 > URL: https://issues.apache.org/jira/browse/CALCITE-4277 > Project: Calcite > Issue Type: Improvement >Reporter: Jiatao Tao >Assignee: Jiatao Tao >Priority: Major > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-4363) Need a utility to check if a SQL operator is standard
Liya Fan created CALCITE-4363: - Summary: Need a utility to check if a SQL operator is standard Key: CALCITE-4363 URL: https://issues.apache.org/jira/browse/CALCITE-4363 Project: Calcite Issue Type: Wish Components: core Reporter: Liya Fan Assignee: Liya Fan Currently, all standard SQL operators are defined in SqlStdOperatorTable.java as static members, not as enums. This makes it difficult to check if a SQL function is a standard one. In many scenarios, we need such a check. For example, some optimizations can be performed, if we know the function is standard (not user-defined), because user-defined functions may have some unknown side-effects. To support the check, we only need to maintain a set that contains all standard operators. Do you think such a utility should be supported? -- This message was sent by Atlassian Jira (v8.3.4#803005)
