[jira] [Commented] (CALCITE-3379) Support expand STRING column expression in table during sql-to-rel conversion
[
https://issues.apache.org/jira/browse/CALCITE-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948237#comment-16948237
]
Wang Yanlin commented on CALCITE-3379:
--
Currently, this test case in *ServerTest* run success, mind the invalid use of
*trims*.
{noformat}
@Test public void testVirtualColumnWithFunctions() throws Exception {
try (Connection c = connect();
Statement s = c.createStatement()) {
// Test builtin and library functions.
final String create = "create table t1 (\n"
+ " h varchar(3) not null,\n"
+ " i varchar(3),\n"
+ " j int not null as (char_length(h)) virtual,\n"
+ " k varchar(3) null as (trims(i)) virtual)";
boolean b = s.execute(create);
assertThat(b, is(false));
int x = s.executeUpdate("insert into t1 (h, i) values ('abc', 'de ')");
assertThat(x, is(1));
}
}
{noformat}
And exception occurs when trying to query the table.
Agree with [~jinxing6...@126.com], it would be better to throw an exception
when creating the table.
> Support expand STRING column expression in table during sql-to-rel conversion
> -
>
> Key: CALCITE-3379
> URL: https://issues.apache.org/jira/browse/CALCITE-3379
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.21.0
>Reporter: Danny Chen
>Assignee: Danny Chen
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.22.0
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Now there are 2 ways to convert a RelOptTable to LogicalTableScan:
> 1. One way is to open the Config#isConvertTableAccess[1] flag and the
> SqlToRelConverter would invoke the #toRel method which transforms the table
> to a node returned by the user(Usually a table scan).
> 2. Another way is to use the LogicalTableScan rule, this rule would invoke
> RelOptTable#toRel and wrap the returned node with a LogicalTableScan.
>
> The difference between 1 and 2 is that, 2 happens in the planning rule but 1
> happens in sql-to-rel conversion, 1 also supports to expand the table columns
> based on the defined default values expressions, see
> InitializerExpressionFactory#newColumnDefaultValue.
>
> The problem with the InitializerExpressionFactory#newColumnDefaultValue is
> that it uses InitializerContext#convertExpression to convert a SqlNode, but
> if the SqlNode is not validated, we always got a RexCall with
> SqlUnresolvedFunction. We should give the user chance to validate their
> SqlNode or even we can support pure string expressions which can be used to
> persist.
>
> Another problem with #toRel is that after the expressions applied as a
> projection, user has no chance to apply any other relational nodes if they
> want, we can actually support this, the same way as we support the column
> expressions.
>
> [1]https://github.com/apache/calcite/blob/2dc97e6723e1b5bf762540f87b5cd1a848a1/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L5605
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3379) Support expand STRING column expression in table during sql-to-rel conversion
[ https://issues.apache.org/jira/browse/CALCITE-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948301#comment-16948301 ] Danny Chen commented on CALCITE-3379: - Finally i decide not to support a pure string column expression, if user want to persist their expressions, they can firstly transform them to sql strings like "a + 1" or "my_udf(a)", then use the SqlParser to parse it back to SqlNode and transform it to RexNode with the InitializerContext. That means, use should control the expressions se-de by themselves. Personally i think the SqlToRelConverter should not see the parser thing anymore. So i would just close this issue. > Support expand STRING column expression in table during sql-to-rel conversion > - > > Key: CALCITE-3379 > URL: https://issues.apache.org/jira/browse/CALCITE-3379 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.21.0 >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 1.22.0 > > Time Spent: 2h > Remaining Estimate: 0h > > Now there are 2 ways to convert a RelOptTable to LogicalTableScan: > 1. One way is to open the Config#isConvertTableAccess[1] flag and the > SqlToRelConverter would invoke the #toRel method which transforms the table > to a node returned by the user(Usually a table scan). > 2. Another way is to use the LogicalTableScan rule, this rule would invoke > RelOptTable#toRel and wrap the returned node with a LogicalTableScan. > > The difference between 1 and 2 is that, 2 happens in the planning rule but 1 > happens in sql-to-rel conversion, 1 also supports to expand the table columns > based on the defined default values expressions, see > InitializerExpressionFactory#newColumnDefaultValue. > > The problem with the InitializerExpressionFactory#newColumnDefaultValue is > that it uses InitializerContext#convertExpression to convert a SqlNode, but > if the SqlNode is not validated, we always got a RexCall with > SqlUnresolvedFunction. We should give the user chance to validate their > SqlNode or even we can support pure string expressions which can be used to > persist. > > Another problem with #toRel is that after the expressions applied as a > projection, user has no chance to apply any other relational nodes if they > want, we can actually support this, the same way as we support the column > expressions. > > [1]https://github.com/apache/calcite/blob/2dc97e6723e1b5bf762540f87b5cd1a848a1/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L5605 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-3396) Materialization matching succeeds when query and view are both of UNION but have different 'all' property
[
https://issues.apache.org/jira/browse/CALCITE-3396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated CALCITE-3396:
Labels: pull-request-available (was: )
> Materialization matching succeeds when query and view are both of UNION but
> have different 'all' property
> -
>
> Key: CALCITE-3396
> URL: https://issues.apache.org/jira/browse/CALCITE-3396
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: jin xing
>Assignee: jin xing
>Priority: Major
> Labels: pull-request-available
>
> {code:java}
> Materialized-View:
> select * from emps where empid < 300
> union
> select * from emps where empid > 200
> Query:
> select * from emps where empid < 300
> union all
> select * from emps where empid > 200
> {code}
> Above MV and Query have different 'all' property in UNION but they succeed
> matching now.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-3397) AssertionError for interpreter multiset
[
https://issues.apache.org/jira/browse/CALCITE-3397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated CALCITE-3397:
Labels: pull-request-available (was: )
> AssertionError for interpreter multiset
> ---
>
> Key: CALCITE-3397
> URL: https://issues.apache.org/jira/browse/CALCITE-3397
> Project: Calcite
> Issue Type: Bug
>Reporter: Wang Yanlin
>Priority: Major
> Labels: pull-request-available
>
> when interpretering sql
> *select multiset['a', 'b', 'c']*
> got,
> {code:java}
> java.lang.AssertionError: interpreter: no implementation for class
> org.apache.calcite.rel.core.Collect
> at
> org.apache.calcite.interpreter.Interpreter$CompilerImpl.visit(Interpreter.java:460)
> at
> org.apache.calcite.interpreter.Nodes$CoreCompiler.visit(Nodes.java:43)
> at org.apache.calcite.rel.BiRel.childrenAccept(BiRel.java:46)
> at
> org.apache.calcite.interpreter.Interpreter$CompilerImpl.visit(Interpreter.java:447)
> at
> org.apache.calcite.interpreter.Nodes$CoreCompiler.visit(Nodes.java:43)
> at org.apache.calcite.rel.SingleRel.childrenAccept(SingleRel.java:72)
> at
> org.apache.calcite.interpreter.Interpreter$CompilerImpl.visit(Interpreter.java:447)
> at
> org.apache.calcite.interpreter.Nodes$CoreCompiler.visit(Nodes.java:43)
> at
> org.apache.calcite.interpreter.Interpreter$CompilerImpl.visitRoot(Interpreter.java:405)
> at
> org.apache.calcite.interpreter.Interpreter.(Interpreter.java:88)
> at
> org.apache.calcite.test.InterpreterTest.testInterpretMultiset(InterpreterTest.java:127)
> {code}
> Reproduce this with test case in InterpreterTest
> {code:java}
> @Test public void testInterpretMultiset() throws Exception {
> final String sql = "select multiset['a', 'b', 'c']";
> SqlNode parse = planner.parse(sql);
> SqlNode validate = planner.validate(parse);
> RelNode convert = planner.rel(validate).project();
> final Interpreter interpreter = new Interpreter(dataContext, convert);
> assertRows(interpreter, "[[a, b, c]]");
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-3384) Support Kerberos-authentication using SPNEGO over HTTPS
[ https://issues.apache.org/jira/browse/CALCITE-3384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941742#comment-16941742 ] István Tóth edited comment on CALCITE-3384 at 10/10/19 11:01 AM: - The patch mostly just reshuffles existing code. There are some minor differences in behaviour though: * I have removed the explicit check that disables HTTPS SPNEGO from the server side * -I have removed the keystore password checks, so now unprotected keystores are not rejected outright- * If avatica.http.spnego.max_cached and avatica.http.spnego.max_per_route are not set, they default to avatica.pooled.connections.max and avatica.pooled.connections.per.route respectively (which in turn default to the same values that the SPENGO specific parameters used to default to) cc [~elserj] was (Author: stoty): The patch mostly just reshuffles existing code. There are some minor differences in behaviour though: * I have removed the explicit check that disables HTTPS SPNEGO from the server side * I have removed the keystore password checks, so now unprotected keystores are not rejected outright * If avatica.http.spnego.max_cached and avatica.http.spnego.max_per_route are not set, they default to avatica.pooled.connections.max and avatica.pooled.connections.per.route respectively (which in turn default to the same values that the SPENGO specific parameters used to default to) cc [~elserj] > Support Kerberos-authentication using SPNEGO over HTTPS > --- > > Key: CALCITE-3384 > URL: https://issues.apache.org/jira/browse/CALCITE-3384 > Project: Calcite > Issue Type: Improvement >Affects Versions: avatica-1.16.0 >Reporter: István Tóth >Assignee: István Tóth >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Avatica supports both HTTPS connections, and kerberos authentication using > SPNEGO, but not both at same. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-3379) Support expand STRING column expression in table during sql-to-rel conversion
[ https://issues.apache.org/jira/browse/CALCITE-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948269#comment-16948269 ] Danny Chen edited comment on CALCITE-3379 at 10/10/19 7:19 AM: --- > And exception occurs when trying to query the table. [~yanlin-Lynn] What do you mean by an exception throws ? The query in the test case is valid. And why do you think the usage of trims is invalid. was (Author: danny0405): > And exception occurs when trying to query the table. [~yanlin-Lynn] What do you mean by an exception throws ? The query in the test case is valid. > Support expand STRING column expression in table during sql-to-rel conversion > - > > Key: CALCITE-3379 > URL: https://issues.apache.org/jira/browse/CALCITE-3379 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.21.0 >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 1.22.0 > > Time Spent: 2h > Remaining Estimate: 0h > > Now there are 2 ways to convert a RelOptTable to LogicalTableScan: > 1. One way is to open the Config#isConvertTableAccess[1] flag and the > SqlToRelConverter would invoke the #toRel method which transforms the table > to a node returned by the user(Usually a table scan). > 2. Another way is to use the LogicalTableScan rule, this rule would invoke > RelOptTable#toRel and wrap the returned node with a LogicalTableScan. > > The difference between 1 and 2 is that, 2 happens in the planning rule but 1 > happens in sql-to-rel conversion, 1 also supports to expand the table columns > based on the defined default values expressions, see > InitializerExpressionFactory#newColumnDefaultValue. > > The problem with the InitializerExpressionFactory#newColumnDefaultValue is > that it uses InitializerContext#convertExpression to convert a SqlNode, but > if the SqlNode is not validated, we always got a RexCall with > SqlUnresolvedFunction. We should give the user chance to validate their > SqlNode or even we can support pure string expressions which can be used to > persist. > > Another problem with #toRel is that after the expressions applied as a > projection, user has no chance to apply any other relational nodes if they > want, we can actually support this, the same way as we support the column > expressions. > > [1]https://github.com/apache/calcite/blob/2dc97e6723e1b5bf762540f87b5cd1a848a1/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L5605 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3379) Support expand STRING column expression in table during sql-to-rel conversion
[ https://issues.apache.org/jira/browse/CALCITE-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948269#comment-16948269 ] Danny Chen commented on CALCITE-3379: - > And exception occurs when trying to query the table. [~yanlin-Lynn] What do you mean by an exception throws ? The query in the test case is valid. > Support expand STRING column expression in table during sql-to-rel conversion > - > > Key: CALCITE-3379 > URL: https://issues.apache.org/jira/browse/CALCITE-3379 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.21.0 >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 1.22.0 > > Time Spent: 2h > Remaining Estimate: 0h > > Now there are 2 ways to convert a RelOptTable to LogicalTableScan: > 1. One way is to open the Config#isConvertTableAccess[1] flag and the > SqlToRelConverter would invoke the #toRel method which transforms the table > to a node returned by the user(Usually a table scan). > 2. Another way is to use the LogicalTableScan rule, this rule would invoke > RelOptTable#toRel and wrap the returned node with a LogicalTableScan. > > The difference between 1 and 2 is that, 2 happens in the planning rule but 1 > happens in sql-to-rel conversion, 1 also supports to expand the table columns > based on the defined default values expressions, see > InitializerExpressionFactory#newColumnDefaultValue. > > The problem with the InitializerExpressionFactory#newColumnDefaultValue is > that it uses InitializerContext#convertExpression to convert a SqlNode, but > if the SqlNode is not validated, we always got a RexCall with > SqlUnresolvedFunction. We should give the user chance to validate their > SqlNode or even we can support pure string expressions which can be used to > persist. > > Another problem with #toRel is that after the expressions applied as a > projection, user has no chance to apply any other relational nodes if they > want, we can actually support this, the same way as we support the column > expressions. > > [1]https://github.com/apache/calcite/blob/2dc97e6723e1b5bf762540f87b5cd1a848a1/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L5605 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3379) Support expand STRING column expression in table during sql-to-rel conversion
[
https://issues.apache.org/jira/browse/CALCITE-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948284#comment-16948284
]
Wang Yanlin commented on CALCITE-3379:
--
it's *trims* not *trim*.
Trim is valid, but trims is invalid.
Or, just check using a *invalid_function* call like this
{noformat}
final String create = "create table t1 (\n"
+ " h varchar(3) not null,\n"
+ " i varchar(3),\n"
+ " j int not null as (char_length(h)) virtual,\n"
+ " k varchar(3) null as (invalid_function(i)) virtual)";
boolean b = s.execute(create);
assertThat(b, is(false));
int x = s.executeUpdate("insert into t1 (h, i) values ('abc', 'de ')");
assertThat(x, is(1));
{noformat}
> Support expand STRING column expression in table during sql-to-rel conversion
> -
>
> Key: CALCITE-3379
> URL: https://issues.apache.org/jira/browse/CALCITE-3379
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.21.0
>Reporter: Danny Chen
>Assignee: Danny Chen
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.22.0
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Now there are 2 ways to convert a RelOptTable to LogicalTableScan:
> 1. One way is to open the Config#isConvertTableAccess[1] flag and the
> SqlToRelConverter would invoke the #toRel method which transforms the table
> to a node returned by the user(Usually a table scan).
> 2. Another way is to use the LogicalTableScan rule, this rule would invoke
> RelOptTable#toRel and wrap the returned node with a LogicalTableScan.
>
> The difference between 1 and 2 is that, 2 happens in the planning rule but 1
> happens in sql-to-rel conversion, 1 also supports to expand the table columns
> based on the defined default values expressions, see
> InitializerExpressionFactory#newColumnDefaultValue.
>
> The problem with the InitializerExpressionFactory#newColumnDefaultValue is
> that it uses InitializerContext#convertExpression to convert a SqlNode, but
> if the SqlNode is not validated, we always got a RexCall with
> SqlUnresolvedFunction. We should give the user chance to validate their
> SqlNode or even we can support pure string expressions which can be used to
> persist.
>
> Another problem with #toRel is that after the expressions applied as a
> projection, user has no chance to apply any other relational nodes if they
> want, we can actually support this, the same way as we support the column
> expressions.
>
> [1]https://github.com/apache/calcite/blob/2dc97e6723e1b5bf762540f87b5cd1a848a1/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L5605
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (CALCITE-2997) Avoid pushing down join condition in SqlToRelConverter
[
https://issues.apache.org/jira/browse/CALCITE-2997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jin xing closed CALCITE-2997.
-
Resolution: Invalid
> Avoid pushing down join condition in SqlToRelConverter
> --
>
> Key: CALCITE-2997
> URL: https://issues.apache.org/jira/browse/CALCITE-2997
> Project: Calcite
> Issue Type: Bug
>Reporter: jin xing
>Priority: Major
>
> In current code, *SqlToRelConverter:createJoin* is calling
> *RelOptUtil.pushDownJoinConditions* for optimization. And we can find below
> conversion from *SqlNode* to *RelNode*:
> {code:java}
> SqlNode:
> select * from A join B on A.x = B.x * 2
> RelNode (Logical-Plan):
> Join (condition:col0=col1)
> |-Project(x as col0)
> | |-Scan(A)
> |-Project(x * 2 as col1)
> |-Scan(B){code}
> As we can see the logical plan(*RelNode*) posted above is not the pure
> reflection of the original SQL String(*SqlNode*). The optimization is mixed
> into the phase on which AST is converted to Logical-Plan. Actually optimizing
> rule of JoinPushExpressionsRule is doing exactly the same kind of thing.
> Shall we just keep the optimization inside Optimized-Logical-Plan ? I mean
> shall we avoid calling *RelOptUtil.pushDownJoinConditions* in
> *SqlToRelConverter:createJoin*
> I raised this issue because that we are doing something based on the
> Logical-Plan. And it makes us really confused that the Logical-Plan doesn't
> corresponds to SqlNode.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3396) Materialization matching succeeds when query and view are both of UNION but have different 'all' property
jin xing created CALCITE-3396: - Summary: Materialization matching succeeds when query and view are both of UNION but have different 'all' property Key: CALCITE-3396 URL: https://issues.apache.org/jira/browse/CALCITE-3396 Project: Calcite Issue Type: Bug Components: core Reporter: jin xing Assignee: jin xing -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-3396) Materialization matching succeeds when query and view are both of UNION but have different 'all' property
[
https://issues.apache.org/jira/browse/CALCITE-3396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jin xing updated CALCITE-3396:
--
Description:
{code:java}
Materialized-View:
select * from emps where empid < 300
union
select * from emps where empid > 200
Query:
select * from emps where empid < 300
union all
select * from emps where empid > 200
{code}
Above MV and Query have different 'all' property in UNION but they succeed
matching now.
was:
{code:java}
{code}
*Materialized View:*
{code:java}
select * from emps where empid < 300
union
{code}
**
{code:java}
select * from emps where empid > 200
{code}
*Query:*
{code:java}
select * from emps where empid < 300
union all
select * from emps where empid > 200
{code}
Above MV and Query have different 'all' property in UNION but they succeed
matching now.
> Materialization matching succeeds when query and view are both of UNION but
> have different 'all' property
> -
>
> Key: CALCITE-3396
> URL: https://issues.apache.org/jira/browse/CALCITE-3396
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: jin xing
>Assignee: jin xing
>Priority: Major
>
> {code:java}
> Materialized-View:
> select * from emps where empid < 300
> union
> select * from emps where empid > 200
> Query:
> select * from emps where empid < 300
> union all
> select * from emps where empid > 200
> {code}
> Above MV and Query have different 'all' property in UNION but they succeed
> matching now.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3398) Release avatica 1.16.0
Francis Chuang created CALCITE-3398: --- Summary: Release avatica 1.16.0 Key: CALCITE-3398 URL: https://issues.apache.org/jira/browse/CALCITE-3398 Project: Calcite Issue Type: Improvement Components: avatica Reporter: Francis Chuang Assignee: Francis Chuang Fix For: avatica-1.16.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3379) Support expand STRING column expression in table during sql-to-rel conversion
[ https://issues.apache.org/jira/browse/CALCITE-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948295#comment-16948295 ] Danny Chen commented on CALCITE-3379: - Okey, got your idea, personally, i'm inclined to validate the nodes when the query really happens, just like many other sql engines do to VIEWs [1]. The computed column actually is a VIEW. [1] https://docs.microsoft.com/en-us/sql/t-sql/statements/create-view-transact-sql?view=sql-server-2017#remarks > Support expand STRING column expression in table during sql-to-rel conversion > - > > Key: CALCITE-3379 > URL: https://issues.apache.org/jira/browse/CALCITE-3379 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.21.0 >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 1.22.0 > > Time Spent: 2h > Remaining Estimate: 0h > > Now there are 2 ways to convert a RelOptTable to LogicalTableScan: > 1. One way is to open the Config#isConvertTableAccess[1] flag and the > SqlToRelConverter would invoke the #toRel method which transforms the table > to a node returned by the user(Usually a table scan). > 2. Another way is to use the LogicalTableScan rule, this rule would invoke > RelOptTable#toRel and wrap the returned node with a LogicalTableScan. > > The difference between 1 and 2 is that, 2 happens in the planning rule but 1 > happens in sql-to-rel conversion, 1 also supports to expand the table columns > based on the defined default values expressions, see > InitializerExpressionFactory#newColumnDefaultValue. > > The problem with the InitializerExpressionFactory#newColumnDefaultValue is > that it uses InitializerContext#convertExpression to convert a SqlNode, but > if the SqlNode is not validated, we always got a RexCall with > SqlUnresolvedFunction. We should give the user chance to validate their > SqlNode or even we can support pure string expressions which can be used to > persist. > > Another problem with #toRel is that after the expressions applied as a > projection, user has no chance to apply any other relational nodes if they > want, we can actually support this, the same way as we support the column > expressions. > > [1]https://github.com/apache/calcite/blob/2dc97e6723e1b5bf762540f87b5cd1a848a1/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L5605 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-3396) Materialization matching succeeds when query and view are both of UNION but have different 'all' property
[ https://issues.apache.org/jira/browse/CALCITE-3396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jin xing updated CALCITE-3396: -- Description: *Materialized View:* select * from emps where empid < 300 union select * from emps where empid > 200 *Query:* select * from emps where empid < 300 union all select * from emps where empid > 200 Above MV and Query have different 'all' property in UNION but they succeed matching now. > Materialization matching succeeds when query and view are both of UNION but > have different 'all' property > - > > Key: CALCITE-3396 > URL: https://issues.apache.org/jira/browse/CALCITE-3396 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: jin xing >Assignee: jin xing >Priority: Major > > *Materialized View:* > select * from emps where empid < 300 > union > select * from emps where empid > 200 > > *Query:* > select * from emps where empid < 300 > union all > select * from emps where empid > 200 > > Above MV and Query have different 'all' property in UNION but they succeed > matching now. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-3396) Materialization matching succeeds when query and view are both of UNION but have different 'all' property
[
https://issues.apache.org/jira/browse/CALCITE-3396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jin xing updated CALCITE-3396:
--
Description:
{code:java}
{code}
*Materialized View:*
{code:java}
select * from emps where empid < 300
union
{code}
**
{code:java}
select * from emps where empid > 200
{code}
*Query:*
{code:java}
select * from emps where empid < 300
union all
select * from emps where empid > 200
{code}
Above MV and Query have different 'all' property in UNION but they succeed
matching now.
was:
*Materialized View:*
select * from emps where empid < 300
union
select * from emps where empid > 200
*Query:*
select * from emps where empid < 300
union all
select * from emps where empid > 200
Above MV and Query have different 'all' property in UNION but they succeed
matching now.
> Materialization matching succeeds when query and view are both of UNION but
> have different 'all' property
> -
>
> Key: CALCITE-3396
> URL: https://issues.apache.org/jira/browse/CALCITE-3396
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: jin xing
>Assignee: jin xing
>Priority: Major
>
>
> {code:java}
> {code}
> *Materialized View:*
> {code:java}
> select * from emps where empid < 300
> union
> {code}
>
> **
> {code:java}
> select * from emps where empid > 200
> {code}
> *Query:*
> {code:java}
> select * from emps where empid < 300
> union all
> select * from emps where empid > 200
> {code}
>
> Above MV and Query have different 'all' property in UNION but they succeed
> matching now.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-3397) AssertionError for interpreter multiset
[
https://issues.apache.org/jira/browse/CALCITE-3397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wang Yanlin updated CALCITE-3397:
-
Description:
when interpretering sql
*select multiset['a', 'b', 'c']*
got,
{code:java}
java.lang.AssertionError: interpreter: no implementation for class
org.apache.calcite.rel.core.Collect
at
org.apache.calcite.interpreter.Interpreter$CompilerImpl.visit(Interpreter.java:460)
at
org.apache.calcite.interpreter.Nodes$CoreCompiler.visit(Nodes.java:43)
at org.apache.calcite.rel.BiRel.childrenAccept(BiRel.java:46)
at
org.apache.calcite.interpreter.Interpreter$CompilerImpl.visit(Interpreter.java:447)
at
org.apache.calcite.interpreter.Nodes$CoreCompiler.visit(Nodes.java:43)
at org.apache.calcite.rel.SingleRel.childrenAccept(SingleRel.java:72)
at
org.apache.calcite.interpreter.Interpreter$CompilerImpl.visit(Interpreter.java:447)
at
org.apache.calcite.interpreter.Nodes$CoreCompiler.visit(Nodes.java:43)
at
org.apache.calcite.interpreter.Interpreter$CompilerImpl.visitRoot(Interpreter.java:405)
at
org.apache.calcite.interpreter.Interpreter.(Interpreter.java:88)
at
org.apache.calcite.test.InterpreterTest.testInterpretMultiset(InterpreterTest.java:127)
{code}
Reproduce this with test case in InterpreterTest
{code:java}
@Test public void testInterpretMultiset() throws Exception {
final String sql = "select multiset['a', 'b', 'c']";
SqlNode parse = planner.parse(sql);
SqlNode validate = planner.validate(parse);
RelNode convert = planner.rel(validate).project();
final Interpreter interpreter = new Interpreter(dataContext, convert);
assertRows(interpreter, "[[a, b, c]]");
}
{code}
was:
when interpretering sql
got,
{code:java}
java.lang.AssertionError: interpreter: no implementation for class
org.apache.calcite.rel.core.Collect
at
org.apache.calcite.interpreter.Interpreter$CompilerImpl.visit(Interpreter.java:460)
at
org.apache.calcite.interpreter.Nodes$CoreCompiler.visit(Nodes.java:43)
at org.apache.calcite.rel.BiRel.childrenAccept(BiRel.java:46)
at
org.apache.calcite.interpreter.Interpreter$CompilerImpl.visit(Interpreter.java:447)
at
org.apache.calcite.interpreter.Nodes$CoreCompiler.visit(Nodes.java:43)
at org.apache.calcite.rel.SingleRel.childrenAccept(SingleRel.java:72)
at
org.apache.calcite.interpreter.Interpreter$CompilerImpl.visit(Interpreter.java:447)
at
org.apache.calcite.interpreter.Nodes$CoreCompiler.visit(Nodes.java:43)
at
org.apache.calcite.interpreter.Interpreter$CompilerImpl.visitRoot(Interpreter.java:405)
at
org.apache.calcite.interpreter.Interpreter.(Interpreter.java:88)
at
org.apache.calcite.test.InterpreterTest.testInterpretMultiset(InterpreterTest.java:127)
{code}
Reproduce this with test case
{code:java}
@Test public void testInterpretMultiset() throws Exception {
final String sql = "select multiset['a', 'b', 'c']";
SqlNode parse = planner.parse(sql);
SqlNode validate = planner.validate(parse);
RelNode convert = planner.rel(validate).project();
final Interpreter interpreter = new Interpreter(dataContext, convert);
assertRows(interpreter, "[[a, b, c]]");
}
{code}
> AssertionError for interpreter multiset
> ---
>
> Key: CALCITE-3397
> URL: https://issues.apache.org/jira/browse/CALCITE-3397
> Project: Calcite
> Issue Type: Bug
>Reporter: Wang Yanlin
>Priority: Major
>
> when interpretering sql
> *select multiset['a', 'b', 'c']*
> got,
> {code:java}
> java.lang.AssertionError: interpreter: no implementation for class
> org.apache.calcite.rel.core.Collect
> at
> org.apache.calcite.interpreter.Interpreter$CompilerImpl.visit(Interpreter.java:460)
> at
> org.apache.calcite.interpreter.Nodes$CoreCompiler.visit(Nodes.java:43)
> at org.apache.calcite.rel.BiRel.childrenAccept(BiRel.java:46)
> at
> org.apache.calcite.interpreter.Interpreter$CompilerImpl.visit(Interpreter.java:447)
> at
> org.apache.calcite.interpreter.Nodes$CoreCompiler.visit(Nodes.java:43)
> at org.apache.calcite.rel.SingleRel.childrenAccept(SingleRel.java:72)
> at
> org.apache.calcite.interpreter.Interpreter$CompilerImpl.visit(Interpreter.java:447)
> at
> org.apache.calcite.interpreter.Nodes$CoreCompiler.visit(Nodes.java:43)
> at
> org.apache.calcite.interpreter.Interpreter$CompilerImpl.visitRoot(Interpreter.java:405)
> at
> org.apache.calcite.interpreter.Interpreter.(Interpreter.java:88)
> at
> org.apache.calcite.test.InterpreterTest.testInterpretMultiset(InterpreterTest.java:127)
> {code}
> Reproduce this with test case in InterpreterTest
> {code:java}
> @Test public void testInterpretMultiset()
[jira] [Created] (CALCITE-3397) AssertionError for interpreter multiset
Wang Yanlin created CALCITE-3397:
Summary: AssertionError for interpreter multiset
Key: CALCITE-3397
URL: https://issues.apache.org/jira/browse/CALCITE-3397
Project: Calcite
Issue Type: Bug
Reporter: Wang Yanlin
when interpretering sql
got,
{code:java}
java.lang.AssertionError: interpreter: no implementation for class
org.apache.calcite.rel.core.Collect
at
org.apache.calcite.interpreter.Interpreter$CompilerImpl.visit(Interpreter.java:460)
at
org.apache.calcite.interpreter.Nodes$CoreCompiler.visit(Nodes.java:43)
at org.apache.calcite.rel.BiRel.childrenAccept(BiRel.java:46)
at
org.apache.calcite.interpreter.Interpreter$CompilerImpl.visit(Interpreter.java:447)
at
org.apache.calcite.interpreter.Nodes$CoreCompiler.visit(Nodes.java:43)
at org.apache.calcite.rel.SingleRel.childrenAccept(SingleRel.java:72)
at
org.apache.calcite.interpreter.Interpreter$CompilerImpl.visit(Interpreter.java:447)
at
org.apache.calcite.interpreter.Nodes$CoreCompiler.visit(Nodes.java:43)
at
org.apache.calcite.interpreter.Interpreter$CompilerImpl.visitRoot(Interpreter.java:405)
at
org.apache.calcite.interpreter.Interpreter.(Interpreter.java:88)
at
org.apache.calcite.test.InterpreterTest.testInterpretMultiset(InterpreterTest.java:127)
{code}
Reproduce this with test case
{code:java}
@Test public void testInterpretMultiset() throws Exception {
final String sql = "select multiset['a', 'b', 'c']";
SqlNode parse = planner.parse(sql);
SqlNode validate = planner.validate(parse);
RelNode convert = planner.rel(validate).project();
final Interpreter interpreter = new Interpreter(dataContext, convert);
assertRows(interpreter, "[[a, b, c]]");
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-3384) Support Kerberos-authentication using SPNEGO over HTTPS
[ https://issues.apache.org/jira/browse/CALCITE-3384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser resolved CALCITE-3384. - Fix Version/s: avatica-1.16.0 Resolution: Fixed Thanks for the contribution, Istvan! Great to see someone poking at this. > Support Kerberos-authentication using SPNEGO over HTTPS > --- > > Key: CALCITE-3384 > URL: https://issues.apache.org/jira/browse/CALCITE-3384 > Project: Calcite > Issue Type: Improvement >Affects Versions: avatica-1.16.0 >Reporter: István Tóth >Assignee: István Tóth >Priority: Major > Labels: pull-request-available > Fix For: avatica-1.16.0 > > Time Spent: 50m > Remaining Estimate: 0h > > Avatica supports both HTTPS connections, and kerberos authentication using > SPNEGO, but not both at same. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-3399) RelFieldTrimmer trim fields for UNION, INTERSECT, INTERSECT ALL, EXCEPT, EXCEPT ALL
jin xing created CALCITE-3399: - Summary: RelFieldTrimmer trim fields for UNION, INTERSECT, INTERSECT ALL, EXCEPT, EXCEPT ALL Key: CALCITE-3399 URL: https://issues.apache.org/jira/browse/CALCITE-3399 Project: Calcite Issue Type: Bug Components: core Reporter: jin xing Assignee: jin xing RelFieldTrimmer#trimFields provides functionality to trim fields for UNION, UNION ALL, INTERSECT, INTERSECT ALL, EXCEPT, EXCEPT ALL; But UNION ALL, INTERSECT, INTERSECT ALL, EXCEPT, EXCEPT ALL works by checking duplication. Column pruning on inputs of SetOp might lead to different semantics. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-3399) RelFieldTrimmer trim fields for UNION, INTERSECT, INTERSECT ALL, EXCEPT, EXCEPT ALL
[
https://issues.apache.org/jira/browse/CALCITE-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated CALCITE-3399:
Labels: pull-request-available (was: )
> RelFieldTrimmer trim fields for UNION, INTERSECT, INTERSECT ALL, EXCEPT,
> EXCEPT ALL
> ---
>
> Key: CALCITE-3399
> URL: https://issues.apache.org/jira/browse/CALCITE-3399
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: jin xing
>Assignee: jin xing
>Priority: Major
> Labels: pull-request-available
>
> *RelFieldTrimmer#trimFields* provides functionality to trim fields for
> *UNION, UNION ALL, INTERSECT, INTERSECT ALL, EXCEPT, EXCEPT ALL;*
> But *UNION ALL, INTERSECT, INTERSECT ALL, EXCEPT, EXCEPT ALL* works by
> checking duplication. Column pruning on inputs of *SetOp* might lead to
> different semantics.
>
> Take below example for illustration
> {code:java}
> emp0:
> name, deptno
> "A", 0
> "B", 1
> emp1:
> name, deptno
> "C", 0
> "D", 2
> select deptno from
> (select name, deptno from emp0
> intersect
> select name, deptno from emp1)
> {code}
> Run above Sql on emp0 and emp1, result is not empty if trim fields on inputs
> of INTERSECT, but result is empty if DO NOT trim fields on inputs of INTERSECT
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-3399) RelFieldTrimmer trim fields for UNION, INTERSECT, INTERSECT ALL, EXCEPT, EXCEPT ALL
[
https://issues.apache.org/jira/browse/CALCITE-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jin xing updated CALCITE-3399:
--
Description:
*RelFieldTrimmer#trimFields* provides functionality to trim fields for *UNION,
UNION ALL, INTERSECT, INTERSECT ALL, EXCEPT, EXCEPT ALL;*
But *UNION ALL, INTERSECT, INTERSECT ALL, EXCEPT, EXCEPT ALL* works by checking
duplication. Column pruning on inputs of *SetOp* might lead to different
semantics.
Take below example for illustration
{code:java}
emp0:
name, deptno
"A", 0
"B", 1
emp1:
name, deptno
"C", 0
"D", 2
select deptno from
(select name, deptno from emp0
intersect
select name, deptno from emp1)
{code}
Run above Sql on emp0 and emp1, result is not empty if trim fields on inputs of
INTERSECT, but result is empty if DO NOT trim fields on inputs of INTERSECT
was:
RelFieldTrimmer#trimFields provides functionality to trim fields for UNION,
UNION ALL, INTERSECT, INTERSECT ALL, EXCEPT, EXCEPT ALL;
But UNION ALL, INTERSECT, INTERSECT ALL, EXCEPT, EXCEPT ALL works by checking
duplication. Column pruning on inputs of SetOp might lead to different
semantics.
> RelFieldTrimmer trim fields for UNION, INTERSECT, INTERSECT ALL, EXCEPT,
> EXCEPT ALL
> ---
>
> Key: CALCITE-3399
> URL: https://issues.apache.org/jira/browse/CALCITE-3399
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: jin xing
>Assignee: jin xing
>Priority: Major
>
> *RelFieldTrimmer#trimFields* provides functionality to trim fields for
> *UNION, UNION ALL, INTERSECT, INTERSECT ALL, EXCEPT, EXCEPT ALL;*
> But *UNION ALL, INTERSECT, INTERSECT ALL, EXCEPT, EXCEPT ALL* works by
> checking duplication. Column pruning on inputs of *SetOp* might lead to
> different semantics.
>
> Take below example for illustration
> {code:java}
> emp0:
> name, deptno
> "A", 0
> "B", 1
> emp1:
> name, deptno
> "C", 0
> "D", 2
> select deptno from
> (select name, deptno from emp0
> intersect
> select name, deptno from emp1)
> {code}
> Run above Sql on emp0 and emp1, result is not empty if trim fields on inputs
> of INTERSECT, but result is empty if DO NOT trim fields on inputs of INTERSECT
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-3383) Allow plural time units in interval literals
[
https://issues.apache.org/jira/browse/CALCITE-3383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Hyde resolved CALCITE-3383.
--
Fix Version/s: 1.22.0
Resolution: Fixed
Fixed in
[4be2d08e|https://github.com/apache/calcite/commit/4be2d08e2116234a7df982014f8ae5bc79a7e0bf].
> Allow plural time units in interval literals
>
>
> Key: CALCITE-3383
> URL: https://issues.apache.org/jira/browse/CALCITE-3383
> Project: Calcite
> Issue Type: Bug
>Reporter: Julian Hyde
>Assignee: Julian Hyde
>Priority: Major
> Fix For: 1.22.0
>
>
> Allow plural time units in interval literals. For example, currently, Calcite
> allows {{INTERVAL '2' DAY}} but gives an error for {{INTERVAL '2' DAYS}}.
> (Note: DAY vs DAYS.)
> Standard SQL only allows the first; PostgreSQL allows both.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-3368) Some problems simplifying ‘expression IS NULL’
[
https://issues.apache.org/jira/browse/CALCITE-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leonard Xu updated CALCITE-3368:
Description:
'is null' expression in SQL may be optimized incorrectly in the underlying
implementation.
When I write a Fink SQL to test overflow just like
{code:java}
select
case when (f0 + f1) is null then 'null' else 'not null' end
from testTable
{code}
, I found expression '(f0 + f1) is null ' has been optimized by Calcite, and
the optimization may be incorrect.
The underlying implementation is that Calcite's simplification logic of isNull
expression in SQL will convert from
*"f(operand0, operand1) IS NULL"* to
*"operand0 IS NULL OR operand1 IS NULL"* if the Policy of RexNode‘s SqlKind
is ANY。
This simplification leads to the expression will not calculate the real
value of *f(operand0, operand1)* (eg.. '(f0 + f1)' in my case ），but '(f0 +
f1)' maybe overflows after operation.
{code:java}
//org.apache.calcite.rex.RexSimplify.java
private RexNode simplifyIsNull(RexNode a) {
// Simplify the argument first,
// call ourselves recursively to see whether we can make more progress.
// For example, given
// "(CASE WHEN FALSE THEN 1 ELSE 2) IS NULL" we first simplify the
// argument to "2", and only then we can simplify "2 IS NULL" to "FALSE".
a = simplify(a, UNKNOWN);
if (!a.getType().isNullable() && isSafeExpression(a)) {
return rexBuilder.makeLiteral(false);
}
if (RexUtil.isNull(a)) {
return rexBuilder.makeLiteral(true);
}
if (a.getKind() == SqlKind.CAST) {
return null;
}
switch (Strong.policy(a.getKind())) {
case NOT_NULL:
return rexBuilder.makeLiteral(false);
case ANY:
// "f" is a strong operator, so "f(operand0, operand1) IS NULL" simplifies
// to "operand0 IS NULL OR operand1 IS NULL"
final List operands = new ArrayList<>();
for (RexNode operand : ((RexCall) a).getOperands()) {
final RexNode simplified = simplifyIsNull(operand);
if (simplified == null) {
operands.add(
rexBuilder.makeCall(SqlStdOperatorTable.IS_NULL, operand));
} else {
operands.add(simplified);
}
}
return RexUtil.composeDisjunction(rexBuilder, operands, false);
case AS_IS:
default:
return null;
}
}{code}
And most of calculating SqlKinds are assigned *Policy.ANY* at present.
{code:java}
//org.apache.calcite.plan.Strong.java
public static Policy policy(SqlKind kind) {
return MAP.getOrDefault(kind, Policy.AS_IS);
}
map.put(SqlKind.PLUS, Policy.ANY);
map.put(SqlKind.PLUS_PREFIX, Policy.ANY);
map.put(SqlKind.MINUS, Policy.ANY);
map.put(SqlKind.MINUS_PREFIX, Policy.ANY);
map.put(SqlKind.TIMES, Policy.ANY);
map.put(SqlKind.DIVIDE, Policy.ANY);
* that operator evaluates to null. */
public enum Policy {
/** This kind of expression is never null. No need to look at its arguments,
* if it has any. */
NOT_NULL,
/** This kind of expression has its own particular rules about whether it
* is null. */
CUSTOM,
/** This kind of expression is null if and only if at least one of its
* arguments is null. */
ANY,
/** This kind of expression may be null. There is no way to rewrite. */
AS_IS,
}{code}
It may be an obvious nonequivalent simplification in SQL. And this issue come
from Flink （FLINK-14030）.
[~danny0405]， Could you have a look at this?
was:
'is null' expression in SQL may be optimized incorrectly in the underlying
implementation.
When I write a Fink SQL to test overflow just like
{code:java}
select
case when (f0 + f1) is null then 'null' else 'not null' end
from testTable
{code}
, I found expression '(f0 + f1) is null ' has been optimized by Calcite, and
the optimization may be incorrect.
The underlying implementation is that Calcite's simplification logic of isNull
expression in SQL will convert from
*"f(operand0, operand1) IS NULL"* to
*"operand0 IS NULL OR operand1 IS NULL"* if the Policy of RexNode‘s SqlKind
is ANY。
This simplification leads to the expression will not calculate the real
value of *f(operand0, operand1)* (eg.. '(f0 + f1)' in my case ），but '(f0 +
f1)' maybe overflows after operation.
{code:java}
//org.apache.calcite.rex.RexSimplify.java
private RexNode simplifyIsNull(RexNode a) {
// Simplify the argument first,
// call ourselves recursively to see whether we can make more progress.
// For example, given
// "(CASE WHEN FALSE THEN 1 ELSE 2) IS NULL" we first simplify the
// argument to "2", and only then we can simplify "2 IS NULL" to "FALSE".
a = simplify(a, UNKNOWN);
if (!a.getType().isNullable() && isSafeExpression(a)) {
return rexBuilder.makeLiteral(false);
}
if (RexUtil.isNull(a)) {
return rexBuilder.makeLiteral(true);
}
if (a.getKind() == SqlKind.CAST) {
return null;
}
switch (Strong.policy(a.getKind())) {
case NOT_NULL:
return rexBuilder.makeLiteral(false);
case ANY:
// "f" is a strong operator, so "f(operand0, operand1) IS NULL" simplifies
// to "operand0 IS
[jira] [Commented] (CALCITE-3379) Support expand STRING column expression in table during sql-to-rel conversion
[ https://issues.apache.org/jira/browse/CALCITE-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949061#comment-16949061 ] Wang Yanlin commented on CALCITE-3379: -- Agree that we need to validate both DDL and expression of table columns. > Support expand STRING column expression in table during sql-to-rel conversion > - > > Key: CALCITE-3379 > URL: https://issues.apache.org/jira/browse/CALCITE-3379 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.21.0 >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 1.22.0 > > Time Spent: 2h > Remaining Estimate: 0h > > Now there are 2 ways to convert a RelOptTable to LogicalTableScan: > 1. One way is to open the Config#isConvertTableAccess[1] flag and the > SqlToRelConverter would invoke the #toRel method which transforms the table > to a node returned by the user(Usually a table scan). > 2. Another way is to use the LogicalTableScan rule, this rule would invoke > RelOptTable#toRel and wrap the returned node with a LogicalTableScan. > > The difference between 1 and 2 is that, 2 happens in the planning rule but 1 > happens in sql-to-rel conversion, 1 also supports to expand the table columns > based on the defined default values expressions, see > InitializerExpressionFactory#newColumnDefaultValue. > > The problem with the InitializerExpressionFactory#newColumnDefaultValue is > that it uses InitializerContext#convertExpression to convert a SqlNode, but > if the SqlNode is not validated, we always got a RexCall with > SqlUnresolvedFunction. We should give the user chance to validate their > SqlNode or even we can support pure string expressions which can be used to > persist. > > Another problem with #toRel is that after the expressions applied as a > projection, user has no chance to apply any other relational nodes if they > want, we can actually support this, the same way as we support the column > expressions. > > [1]https://github.com/apache/calcite/blob/2dc97e6723e1b5bf762540f87b5cd1a848a1/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L5605 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3376) VolcanoPlanner CannotPlanException: best rel is null even though there is an option with non-infinite cost
[
https://issues.apache.org/jira/browse/CALCITE-3376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949062#comment-16949062
]
Wang Yanlin commented on CALCITE-3376:
--
This seems to be fixed after https://issues.apache.org/jira/browse/CALCITE-3330
merged.
> VolcanoPlanner CannotPlanException: best rel is null even though there is an
> option with non-infinite cost
> --
>
> Key: CALCITE-3376
> URL: https://issues.apache.org/jira/browse/CALCITE-3376
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.21.0
>Reporter: Ruben Q L
>Priority: Major
> Attachments: Diagram.png, Graphviz.png, logsTRACE.txt, stackTrace.txt
>
>
> The problem can be reproduced by adding this test to PlannerTest.java:
> {code:java}
> @Test public void testCannotPlanException() throws Exception {
> RelBuilder builder = RelBuilder.create(RelBuilderTest.config().build());
> RuleSet ruleSet =
> RuleSets.ofList(
> //EnumerableRules.ENUMERABLE_JOIN_RULE, // with this rule it
> works!
> JoinToCorrelateRule.INSTANCE,
> EnumerableRules.ENUMERABLE_CORRELATE_RULE,
> EnumerableRules.ENUMERABLE_PROJECT_RULE,
> EnumerableRules.ENUMERABLE_FILTER_RULE,
> EnumerableRules.ENUMERABLE_SORT_RULE,
> EnumerableRules.ENUMERABLE_UNION_RULE,
> EnumerableRules.ENUMERABLE_TABLE_SCAN_RULE);
> builder
> .scan("EMP")
> .scan("EMP")
> .union(true)
> .scan("EMP")
> .scan("EMP")
> .union(true)
> .join(
> JoinRelType.INNER,
> builder.equals(
> builder.field(2, 0, "DEPTNO"),
> builder.field(2, 1, "EMPNO")));
> RelNode relNode = builder.build();
> RelOptPlanner planner = relNode.getCluster().getPlanner();
> Program program = Programs.of(ruleSet);
> RelTraitSet toTraits = relNode.getTraitSet()
> .replace(EnumerableConvention.INSTANCE);
> RelNode output = program.run(planner, relNode, toTraits,
> ImmutableList.of(), ImmutableList.of());
> String outputStr = toString(output);
> }
> {code}
> Running this test causes the following exception (full stack trace attached):
> {code:java}
> org.apache.calcite.plan.RelOptPlanner$CannotPlanException: There are not
> enough rules to produce a node with desired properties:
> convention=ENUMERABLE, sort=[]. All the inputs have relevant nodes, however
> the cost is still infinite.
> Root: rel#13:Subset#2.ENUMERABLE.[]
> {code}
> The last part of the message (_All the inputs have relevant nodes, however
> the cost is still infinite_) seems relevant, because we can see that
> {{rel#13}}'s best is {{null}}, and it should be {{rel#21}} (which has a
> non-infinite cost):
> {code:java}
> rel#13:Subset#2.ENUMERABLE.[], best=null, importance=1.0
> rel#14:AbstractConverter.ENUMERABLE.[](input=RelSubset#12,
> convention=ENUMERABLE, sort=[]), rowcount=117.6, cumulative cost={inf}
> rel#21:EnumerableCorrelate.ENUMERABLE.[](left=RelSubset#19,
> right=RelSubset#20, correlation=$cor0, joinType=inner, requiredColumns={7}),
> rowcount=1.0, cumulative cost={1770.60001 rows, 2466.0 cpu, 0.0 io}
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Issue Comment Deleted] (CALCITE-3376) VolcanoPlanner CannotPlanException: best rel is null even though there is an option with non-infinite cost
[
https://issues.apache.org/jira/browse/CALCITE-3376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wang Yanlin updated CALCITE-3376:
-
Comment: was deleted
(was: This seems to be fixed after
https://issues.apache.org/jira/browse/CALCITE-3330 merged.)
> VolcanoPlanner CannotPlanException: best rel is null even though there is an
> option with non-infinite cost
> --
>
> Key: CALCITE-3376
> URL: https://issues.apache.org/jira/browse/CALCITE-3376
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.21.0
>Reporter: Ruben Q L
>Priority: Major
> Attachments: Diagram.png, Graphviz.png, logsTRACE.txt, stackTrace.txt
>
>
> The problem can be reproduced by adding this test to PlannerTest.java:
> {code:java}
> @Test public void testCannotPlanException() throws Exception {
> RelBuilder builder = RelBuilder.create(RelBuilderTest.config().build());
> RuleSet ruleSet =
> RuleSets.ofList(
> //EnumerableRules.ENUMERABLE_JOIN_RULE, // with this rule it
> works!
> JoinToCorrelateRule.INSTANCE,
> EnumerableRules.ENUMERABLE_CORRELATE_RULE,
> EnumerableRules.ENUMERABLE_PROJECT_RULE,
> EnumerableRules.ENUMERABLE_FILTER_RULE,
> EnumerableRules.ENUMERABLE_SORT_RULE,
> EnumerableRules.ENUMERABLE_UNION_RULE,
> EnumerableRules.ENUMERABLE_TABLE_SCAN_RULE);
> builder
> .scan("EMP")
> .scan("EMP")
> .union(true)
> .scan("EMP")
> .scan("EMP")
> .union(true)
> .join(
> JoinRelType.INNER,
> builder.equals(
> builder.field(2, 0, "DEPTNO"),
> builder.field(2, 1, "EMPNO")));
> RelNode relNode = builder.build();
> RelOptPlanner planner = relNode.getCluster().getPlanner();
> Program program = Programs.of(ruleSet);
> RelTraitSet toTraits = relNode.getTraitSet()
> .replace(EnumerableConvention.INSTANCE);
> RelNode output = program.run(planner, relNode, toTraits,
> ImmutableList.of(), ImmutableList.of());
> String outputStr = toString(output);
> }
> {code}
> Running this test causes the following exception (full stack trace attached):
> {code:java}
> org.apache.calcite.plan.RelOptPlanner$CannotPlanException: There are not
> enough rules to produce a node with desired properties:
> convention=ENUMERABLE, sort=[]. All the inputs have relevant nodes, however
> the cost is still infinite.
> Root: rel#13:Subset#2.ENUMERABLE.[]
> {code}
> The last part of the message (_All the inputs have relevant nodes, however
> the cost is still infinite_) seems relevant, because we can see that
> {{rel#13}}'s best is {{null}}, and it should be {{rel#21}} (which has a
> non-infinite cost):
> {code:java}
> rel#13:Subset#2.ENUMERABLE.[], best=null, importance=1.0
> rel#14:AbstractConverter.ENUMERABLE.[](input=RelSubset#12,
> convention=ENUMERABLE, sort=[]), rowcount=117.6, cumulative cost={inf}
> rel#21:EnumerableCorrelate.ENUMERABLE.[](left=RelSubset#19,
> right=RelSubset#20, correlation=$cor0, joinType=inner, requiredColumns={7}),
> rowcount=1.0, cumulative cost={1770.60001 rows, 2466.0 cpu, 0.0 io}
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-3379) Support expand STRING column expression in table during sql-to-rel conversion
[ https://issues.apache.org/jira/browse/CALCITE-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948295#comment-16948295 ] Danny Chen edited comment on CALCITE-3379 at 10/11/19 1:43 AM: --- Okey, got your idea, personally, i'm inclined to validate the nodes when the query really happens, just like many other sql engines do to VIEWs [1]. The computed column actually is a VIEW. I did check the MS-SQL 2017 and it validates for create table, so maybe we also need a validation phrase for DDL nodes, we may need 2 kinds of validation: # Syntax check for the DDLs # Expression validation for the table columns [1] https://docs.microsoft.com/en-us/sql/t-sql/statements/create-view-transact-sql?view=sql-server-2017#remarks was (Author: danny0405): Okey, got your idea, personally, i'm inclined to validate the nodes when the query really happens, just like many other sql engines do to VIEWs [1]. The computed column actually is a VIEW. [1] https://docs.microsoft.com/en-us/sql/t-sql/statements/create-view-transact-sql?view=sql-server-2017#remarks > Support expand STRING column expression in table during sql-to-rel conversion > - > > Key: CALCITE-3379 > URL: https://issues.apache.org/jira/browse/CALCITE-3379 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.21.0 >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 1.22.0 > > Time Spent: 2h > Remaining Estimate: 0h > > Now there are 2 ways to convert a RelOptTable to LogicalTableScan: > 1. One way is to open the Config#isConvertTableAccess[1] flag and the > SqlToRelConverter would invoke the #toRel method which transforms the table > to a node returned by the user(Usually a table scan). > 2. Another way is to use the LogicalTableScan rule, this rule would invoke > RelOptTable#toRel and wrap the returned node with a LogicalTableScan. > > The difference between 1 and 2 is that, 2 happens in the planning rule but 1 > happens in sql-to-rel conversion, 1 also supports to expand the table columns > based on the defined default values expressions, see > InitializerExpressionFactory#newColumnDefaultValue. > > The problem with the InitializerExpressionFactory#newColumnDefaultValue is > that it uses InitializerContext#convertExpression to convert a SqlNode, but > if the SqlNode is not validated, we always got a RexCall with > SqlUnresolvedFunction. We should give the user chance to validate their > SqlNode or even we can support pure string expressions which can be used to > persist. > > Another problem with #toRel is that after the expressions applied as a > projection, user has no chance to apply any other relational nodes if they > want, we can actually support this, the same way as we support the column > expressions. > > [1]https://github.com/apache/calcite/blob/2dc97e6723e1b5bf762540f87b5cd1a848a1/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L5605 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3368) Some problems simplifying ‘expression IS NULL’
[
https://issues.apache.org/jira/browse/CALCITE-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949054#comment-16949054
]
Leonard Xu commented on CALCITE-3368:
-
[~kgyrtkirk] [~danny0405]
Could you have a look on the
[1491|[https://github.com/apache/calcite/pull/1491]],
Regarding to the - / + / * operators are safe in most cases, do we need to add
a switch for the simplification for better performance?
And the test cases in my pr is enough ?
Any guidance and suggestion are welcome, thanks very much.
> Some problems simplifying ‘expression IS NULL’
> --
>
> Key: CALCITE-3368
> URL: https://issues.apache.org/jira/browse/CALCITE-3368
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.21.0
>Reporter: Leonard Xu
>Assignee: Leonard Xu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> 'is null' expression in SQL may be optimized incorrectly in the underlying
> implementation.
>
> When I write a Fink SQL to test overflow just like
> {code:java}
> select
>case when (f0 + f1) is null then 'null' else 'not null' end
> from testTable
> {code}
> , I found expression '(f0 + f1) is null ' has been optimized by Calcite, and
> the optimization may be incorrect.
>
> The underlying implementation is that Calcite's simplification logic of
> isNull expression in SQL will convert from
> *"f(operand0, operand1) IS NULL"* to
> *"operand0 IS NULL OR operand1 IS NULL"* if the Policy of RexNode‘s SqlKind
> is ANY。
> This simplification leads to the expression will not calculate the real
> value of *f(operand0, operand1)* (eg.. '(f0 + f1)' in my case ），but '(f0 +
> f1)' maybe overflows after operation.
> {code:java}
> //org.apache.calcite.rex.RexSimplify.java
> private RexNode simplifyIsNull(RexNode a) {
> // Simplify the argument first,
> // call ourselves recursively to see whether we can make more progress.
> // For example, given
> // "(CASE WHEN FALSE THEN 1 ELSE 2) IS NULL" we first simplify the
> // argument to "2", and only then we can simplify "2 IS NULL" to "FALSE".
> a = simplify(a, UNKNOWN);
> if (!a.getType().isNullable() && isSafeExpression(a)) {
> return rexBuilder.makeLiteral(false);
> }
> if (RexUtil.isNull(a)) {
> return rexBuilder.makeLiteral(true);
> }
> if (a.getKind() == SqlKind.CAST) {
> return null;
> }
> switch (Strong.policy(a.getKind())) {
> case NOT_NULL:
> return rexBuilder.makeLiteral(false);
> case ANY:
> // "f" is a strong operator, so "f(operand0, operand1) IS NULL" simplifies
> // to "operand0 IS NULL OR operand1 IS NULL"
> final List operands = new ArrayList<>();
> for (RexNode operand : ((RexCall) a).getOperands()) {
> final RexNode simplified = simplifyIsNull(operand);
> if (simplified == null) {
> operands.add(
> rexBuilder.makeCall(SqlStdOperatorTable.IS_NULL, operand));
> } else {
> operands.add(simplified);
> }
> }
> return RexUtil.composeDisjunction(rexBuilder, operands, false);
> case AS_IS:
> default:
> return null;
> }
> }{code}
> And most of calculating SqlKinds are assigned *Policy.ANY* at present.
> {code:java}
> //org.apache.calcite.plan.Strong.java
> public static Policy policy(SqlKind kind) {
> return MAP.getOrDefault(kind, Policy.AS_IS);
> }
>
> map.put(SqlKind.PLUS, Policy.ANY);
> map.put(SqlKind.PLUS_PREFIX, Policy.ANY);
> map.put(SqlKind.MINUS, Policy.ANY);
> map.put(SqlKind.MINUS_PREFIX, Policy.ANY);
> map.put(SqlKind.TIMES, Policy.ANY);
> map.put(SqlKind.DIVIDE, Policy.ANY);
> * that operator evaluates to null. */
> public enum Policy {
> /** This kind of expression is never null. No need to look at its arguments,
>* if it has any. */
> NOT_NULL,
> /** This kind of expression has its own particular rules about whether it
>* is null. */
> CUSTOM,
> /** This kind of expression is null if and only if at least one of its
>* arguments is null. */
> ANY,
> /** This kind of expression may be null. There is no way to rewrite. */
> AS_IS,
> }{code}
>
> It may be an obvious nonequivalent simplification in SQL. And this issue come
> from Flink （FLINK-14030）.
> [~danny0405]， Could you have a look at this?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-963) Hoist literals
[
https://issues.apache.org/jira/browse/CALCITE-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949073#comment-16949073
]
Danny Chen commented on CALCITE-963:
[~ScottReynolds] Sorry to bother, can you explain more about
EnumerableRelImplementaor#stash ? I only see one direct invoke[1]
> This method takes an object stores it in DataContext and returns an
> Expression that calls DataContext#get to pull out the object.
I didn't see a logic like what you described. It seems that
EnumerableRelImplementaor#stash did some expression cache, but for constant, it
always returns a constant expression directly.
> Hoist literals
> --
>
> Key: CALCITE-963
> URL: https://issues.apache.org/jira/browse/CALCITE-963
> Project: Calcite
> Issue Type: Bug
>Reporter: Julian Hyde
>Priority: Major
> Labels: pull-request-available
> Attachments: HoistedVariables.png
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Convert literals into (internal) bind variables so that statements that
> differ only in literal values can be executed using the same plan.
> In [mail
> thread|http://mail-archives.apache.org/mod_mbox/calcite-dev/201511.mbox/%3c56437bf8.70...@gmail.com%3E]
> Homer wrote:
> {quote}Imagine that it is common to run a large number of very similar
> machine generated queries that just change the literals in the sql query.
> For example (the real queries would be much more complex):
> {code}Select * from emp where empno = 1;
> Select * from emp where empno = 2;
> etc.{code}
> The plan that is likely being generated for these kind of queries is going to
> be very much the same each time, so to save some time, I would like to
> recognize that the literals are all that have changed in a query and use the
> previously optimized execution plan and just replace the literals.{quote}
> I think this could be done as a transform on the initial RelNode tree. It
> would find literals (RexLiteral), replace them with bind variables
> (RexDynamicParam) and write the value into a pool. The next statement would
> go through the same process and the RelNode tree would be identical, but with
> possibly different values for the bind variables.
> The bind variables are of course internal; not visible from JDBC. When the
> statement is executed, the bind variables are implicitly bound.
> Statements would be held in a Guava cache.
> This would be enabled by a config parameter. Unfortunately I don't think we
> could do this by default -- we'd lose optimization power because we would no
> longer be able to do constant reduction.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-3397) AssertionError for interpreter multiset
[
https://issues.apache.org/jira/browse/CALCITE-3397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen updated CALCITE-3397:
Component/s: core
> AssertionError for interpreter multiset
> ---
>
> Key: CALCITE-3397
> URL: https://issues.apache.org/jira/browse/CALCITE-3397
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Wang Yanlin
>Assignee: Danny Chen
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.22.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> when interpretering sql
> *select multiset['a', 'b', 'c']*
> got,
> {code:java}
> java.lang.AssertionError: interpreter: no implementation for class
> org.apache.calcite.rel.core.Collect
> at
> org.apache.calcite.interpreter.Interpreter$CompilerImpl.visit(Interpreter.java:460)
> at
> org.apache.calcite.interpreter.Nodes$CoreCompiler.visit(Nodes.java:43)
> at org.apache.calcite.rel.BiRel.childrenAccept(BiRel.java:46)
> at
> org.apache.calcite.interpreter.Interpreter$CompilerImpl.visit(Interpreter.java:447)
> at
> org.apache.calcite.interpreter.Nodes$CoreCompiler.visit(Nodes.java:43)
> at org.apache.calcite.rel.SingleRel.childrenAccept(SingleRel.java:72)
> at
> org.apache.calcite.interpreter.Interpreter$CompilerImpl.visit(Interpreter.java:447)
> at
> org.apache.calcite.interpreter.Nodes$CoreCompiler.visit(Nodes.java:43)
> at
> org.apache.calcite.interpreter.Interpreter$CompilerImpl.visitRoot(Interpreter.java:405)
> at
> org.apache.calcite.interpreter.Interpreter.(Interpreter.java:88)
> at
> org.apache.calcite.test.InterpreterTest.testInterpretMultiset(InterpreterTest.java:127)
> {code}
> Reproduce this with test case in InterpreterTest
> {code:java}
> @Test public void testInterpretMultiset() throws Exception {
> final String sql = "select multiset['a', 'b', 'c']";
> SqlNode parse = planner.parse(sql);
> SqlNode validate = planner.validate(parse);
> RelNode convert = planner.rel(validate).project();
> final Interpreter interpreter = new Interpreter(dataContext, convert);
> assertRows(interpreter, "[[a, b, c]]");
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-3397) AssertionError for interpreter multiset
[
https://issues.apache.org/jira/browse/CALCITE-3397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen resolved CALCITE-3397.
-
Fix Version/s: 1.22.0
Assignee: Danny Chen
Resolution: Fixed
Fixed in
[f7ff1c9|https://github.com/apache/calcite/commit/f7ff1c906212b34a41599c5009527639ad2e8718],
thanks for your PR, [~yanlin-Lynn] !
> AssertionError for interpreter multiset
> ---
>
> Key: CALCITE-3397
> URL: https://issues.apache.org/jira/browse/CALCITE-3397
> Project: Calcite
> Issue Type: Bug
>Reporter: Wang Yanlin
>Assignee: Danny Chen
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.22.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> when interpretering sql
> *select multiset['a', 'b', 'c']*
> got,
> {code:java}
> java.lang.AssertionError: interpreter: no implementation for class
> org.apache.calcite.rel.core.Collect
> at
> org.apache.calcite.interpreter.Interpreter$CompilerImpl.visit(Interpreter.java:460)
> at
> org.apache.calcite.interpreter.Nodes$CoreCompiler.visit(Nodes.java:43)
> at org.apache.calcite.rel.BiRel.childrenAccept(BiRel.java:46)
> at
> org.apache.calcite.interpreter.Interpreter$CompilerImpl.visit(Interpreter.java:447)
> at
> org.apache.calcite.interpreter.Nodes$CoreCompiler.visit(Nodes.java:43)
> at org.apache.calcite.rel.SingleRel.childrenAccept(SingleRel.java:72)
> at
> org.apache.calcite.interpreter.Interpreter$CompilerImpl.visit(Interpreter.java:447)
> at
> org.apache.calcite.interpreter.Nodes$CoreCompiler.visit(Nodes.java:43)
> at
> org.apache.calcite.interpreter.Interpreter$CompilerImpl.visitRoot(Interpreter.java:405)
> at
> org.apache.calcite.interpreter.Interpreter.(Interpreter.java:88)
> at
> org.apache.calcite.test.InterpreterTest.testInterpretMultiset(InterpreterTest.java:127)
> {code}
> Reproduce this with test case in InterpreterTest
> {code:java}
> @Test public void testInterpretMultiset() throws Exception {
> final String sql = "select multiset['a', 'b', 'c']";
> SqlNode parse = planner.parse(sql);
> SqlNode validate = planner.validate(parse);
> RelNode convert = planner.rel(validate).project();
> final Interpreter interpreter = new Interpreter(dataContext, convert);
> assertRows(interpreter, "[[a, b, c]]");
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
