[jira] [Commented] (CALCITE-3376) VolcanoPlanner CannotPlanException: best rel is null even though there is an option with non-infinite cost
[
https://issues.apache.org/jira/browse/CALCITE-3376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16939547#comment-16939547
]
Danny Chen commented on CALCITE-3376:
-
[~rubenql] Did you try CALCITE-2018 ? It fix some cases of metadata cache, can
you revert change 2166 to see it that causes the problem, I might take some
time to see 2018 tomorrow.
> VolcanoPlanner CannotPlanException: best rel is null even though there is an
> option with non-infinite cost
> --
>
> Key: CALCITE-3376
> URL: https://issues.apache.org/jira/browse/CALCITE-3376
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.21.0
>Reporter: Ruben Q L
>Priority: Major
> Attachments: Diagram.png, Graphviz.png, logsTRACE.txt, stackTrace.txt
>
>
> The problem can be reproduced by adding this test to PlannerTest.java:
> {code:java}
> @Test public void testCannotPlanException() throws Exception {
> RelBuilder builder = RelBuilder.create(RelBuilderTest.config().build());
> RuleSet ruleSet =
> RuleSets.ofList(
> //EnumerableRules.ENUMERABLE_JOIN_RULE, // with this rule it
> works!
> JoinToCorrelateRule.INSTANCE,
> EnumerableRules.ENUMERABLE_CORRELATE_RULE,
> EnumerableRules.ENUMERABLE_PROJECT_RULE,
> EnumerableRules.ENUMERABLE_FILTER_RULE,
> EnumerableRules.ENUMERABLE_SORT_RULE,
> EnumerableRules.ENUMERABLE_UNION_RULE,
> EnumerableRules.ENUMERABLE_TABLE_SCAN_RULE);
> builder
> .scan("EMP")
> .scan("EMP")
> .union(true)
> .scan("EMP")
> .scan("EMP")
> .union(true)
> .join(
> JoinRelType.INNER,
> builder.equals(
> builder.field(2, 0, "DEPTNO"),
> builder.field(2, 1, "EMPNO")));
> RelNode relNode = builder.build();
> RelOptPlanner planner = relNode.getCluster().getPlanner();
> Program program = Programs.of(ruleSet);
> RelTraitSet toTraits = relNode.getTraitSet()
> .replace(EnumerableConvention.INSTANCE);
> RelNode output = program.run(planner, relNode, toTraits,
> ImmutableList.of(), ImmutableList.of());
> String outputStr = toString(output);
> }
> {code}
> Running this test causes the following exception (full stack trace attached):
> {code:java}
> org.apache.calcite.plan.RelOptPlanner$CannotPlanException: There are not
> enough rules to produce a node with desired properties:
> convention=ENUMERABLE, sort=[]. All the inputs have relevant nodes, however
> the cost is still infinite.
> Root: rel#13:Subset#2.ENUMERABLE.[]
> {code}
> The last part of the message (_All the inputs have relevant nodes, however
> the cost is still infinite_) seems relevant, because we can see that
> {{rel#13}}'s best is {{null}}, and it should be {{rel#21}} (which has a
> non-infinite cost):
> {code:java}
> rel#13:Subset#2.ENUMERABLE.[], best=null, importance=1.0
> rel#14:AbstractConverter.ENUMERABLE.[](input=RelSubset#12,
> convention=ENUMERABLE, sort=[]), rowcount=117.6, cumulative cost={inf}
> rel#21:EnumerableCorrelate.ENUMERABLE.[](left=RelSubset#19,
> right=RelSubset#20, correlation=$cor0, joinType=inner, requiredColumns={7}),
> rowcount=1.0, cumulative cost={1770.60001 rows, 2466.0 cpu, 0.0 io}
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3368) 'is null' expression in SQL may be optimized incorrectly in the underlying implementation
[
https://issues.apache.org/jira/browse/CALCITE-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16940255#comment-16940255
]
Danny Chen commented on CALCITE-3368:
-
I tried this test case in SqlToRelConverterTest:
{code:java}
@Test public void testExpressionIsNullSimplify() {
final String sql = "SELECT (empno + sal) is null\n"
+ "FROM emp";
sql(sql).ok();
}
{code}
Now Calcite returns a the plan after sql-to-rel conversion:
{code:sql}
LogicalProject(EXPR$0=[false])
LogicalTableScan(table=[[CATALOG, SALES, EMP]])
{code}
I digged a little and found that the SafeRexVisitor thinks that PLUS is a safe
expression, so "(a + b) is null" can be simplified to "false" if its return
type is not nullable[1].
[~kgyrtkirk], why do you think that "a + b" is a safe expression ? If "a + b"
overflows, then most of the engines would throws exception or just returns
null. But if we simplify "(a + b) is null" to false, we would never have such
behaviors again(Because this simplify happens in sql-to-rel conversion, we even
have no way to configure it).
[1]
[https://github.com/apache/calcite/blob/2dc97e6723e1b5bf762540f87b5cd1a848a1/core/src/main/java/org/apache/calcite/rex/RexSimplify.java#L792]
> 'is null' expression in SQL may be optimized incorrectly in the underlying
> implementation
> -
>
> Key: CALCITE-3368
> URL: https://issues.apache.org/jira/browse/CALCITE-3368
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.21.0
>Reporter: Leonard Xu
>Assignee: Danny Chen
>Priority: Major
>
> 'is null' expression in SQL may be optimized incorrectly in the underlying
> implementation.
>
> When I write a Fink SQL to test overflow just like
> {code:java}
> select
>case when (f0 + f1) is null then 'null' else 'not null' end
> from testTable
> {code}
> , I found expression '(f0 + f1) is null ' has been optimized by Calcite, and
> the optimization may be incorrect.
>
> The underlying implementation is that Calcite's simplification logic of
> isNull expression in SQL will convert from
> *"f(operand0, operand1) IS NULL"* to
> *"operand0 IS NULL OR operand1 IS NULL"* if the Policy of RexNode‘s SqlKind
> is ANY。
> This simplification leads to the expression will not calculate the real
> value of *f(operand0, operand1)* (eg.. '(f0 + f1)' in my case ），but '(f0 +
> f1)' maybe overflows after operation.
> {code:java}
> //org.apache.calcite.rex.RexSimplify.java
> private RexNode simplifyIsNull(RexNode a) {
> // Simplify the argument first,
> // call ourselves recursively to see whether we can make more progress.
> // For example, given
> // "(CASE WHEN FALSE THEN 1 ELSE 2) IS NULL" we first simplify the
> // argument to "2", and only then we can simplify "2 IS NULL" to "FALSE".
> a = simplify(a, UNKNOWN);
> if (!a.getType().isNullable() && isSafeExpression(a)) {
> return rexBuilder.makeLiteral(false);
> }
> if (RexUtil.isNull(a)) {
> return rexBuilder.makeLiteral(true);
> }
> if (a.getKind() == SqlKind.CAST) {
> return null;
> }
> switch (Strong.policy(a.getKind())) {
> case NOT_NULL:
> return rexBuilder.makeLiteral(false);
> case ANY:
> // "f" is a strong operator, so "f(operand0, operand1) IS NULL" simplifies
> // to "operand0 IS NULL OR operand1 IS NULL"
> final List operands = new ArrayList<>();
> for (RexNode operand : ((RexCall) a).getOperands()) {
> final RexNode simplified = simplifyIsNull(operand);
> if (simplified == null) {
> operands.add(
> rexBuilder.makeCall(SqlStdOperatorTable.IS_NULL, operand));
> } else {
> operands.add(simplified);
> }
> }
> return RexUtil.composeDisjunction(rexBuilder, operands, false);
> case AS_IS:
> default:
> return null;
> }
> }{code}
> And most of calculating SqlKinds are assigned *Policy.ANY* at present.
> {code:java}
> //org.apache.calcite.plan.Strong.java
> public static Policy policy(SqlKind kind) {
> return MAP.getOrDefault(kind, Policy.AS_IS);
> }
>
> map.put(SqlKind.PLUS, Policy.ANY);
> map.put(SqlKind.PLUS_PREFIX, Policy.ANY);
> map.put(SqlKind.MINUS, Policy.ANY);
> map.put(SqlKind.MINUS_PREFIX, Policy.ANY);
> map.put(SqlKind.TIMES, Policy.ANY);
> map.put(SqlKind.DIVIDE, Policy.ANY);
> * that operator evaluates to null. */
> public enum Policy {
> /** This kind of expression is never null. No need to look at its arguments,
>* if it has any. */
> NOT_NULL,
> /** This kind of expression has its own particular rules about whether it
>* is null. */
> CUSTOM,
> /** This kind of expression is null if and only if at least one of its
>* arguments is null. */
> ANY,
> /** This kind of expression may be null. There is no way to rewrite. */
> AS_IS,
> }{code}
>
> It may
[jira] [Updated] (CALCITE-3368) Some problems about expression is null simplification
[
https://issues.apache.org/jira/browse/CALCITE-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen updated CALCITE-3368:
Summary: Some problems about expression is null simplification (was: 'is
null' expression in SQL may be optimized incorrectly in the underlying
implementation)
> Some problems about expression is null simplification
> -
>
> Key: CALCITE-3368
> URL: https://issues.apache.org/jira/browse/CALCITE-3368
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.21.0
>Reporter: Leonard Xu
>Assignee: Danny Chen
>Priority: Major
>
> 'is null' expression in SQL may be optimized incorrectly in the underlying
> implementation.
>
> When I write a Fink SQL to test overflow just like
> {code:java}
> select
>case when (f0 + f1) is null then 'null' else 'not null' end
> from testTable
> {code}
> , I found expression '(f0 + f1) is null ' has been optimized by Calcite, and
> the optimization may be incorrect.
>
> The underlying implementation is that Calcite's simplification logic of
> isNull expression in SQL will convert from
> *"f(operand0, operand1) IS NULL"* to
> *"operand0 IS NULL OR operand1 IS NULL"* if the Policy of RexNode‘s SqlKind
> is ANY。
> This simplification leads to the expression will not calculate the real
> value of *f(operand0, operand1)* (eg.. '(f0 + f1)' in my case ），but '(f0 +
> f1)' maybe overflows after operation.
> {code:java}
> //org.apache.calcite.rex.RexSimplify.java
> private RexNode simplifyIsNull(RexNode a) {
> // Simplify the argument first,
> // call ourselves recursively to see whether we can make more progress.
> // For example, given
> // "(CASE WHEN FALSE THEN 1 ELSE 2) IS NULL" we first simplify the
> // argument to "2", and only then we can simplify "2 IS NULL" to "FALSE".
> a = simplify(a, UNKNOWN);
> if (!a.getType().isNullable() && isSafeExpression(a)) {
> return rexBuilder.makeLiteral(false);
> }
> if (RexUtil.isNull(a)) {
> return rexBuilder.makeLiteral(true);
> }
> if (a.getKind() == SqlKind.CAST) {
> return null;
> }
> switch (Strong.policy(a.getKind())) {
> case NOT_NULL:
> return rexBuilder.makeLiteral(false);
> case ANY:
> // "f" is a strong operator, so "f(operand0, operand1) IS NULL" simplifies
> // to "operand0 IS NULL OR operand1 IS NULL"
> final List operands = new ArrayList<>();
> for (RexNode operand : ((RexCall) a).getOperands()) {
> final RexNode simplified = simplifyIsNull(operand);
> if (simplified == null) {
> operands.add(
> rexBuilder.makeCall(SqlStdOperatorTable.IS_NULL, operand));
> } else {
> operands.add(simplified);
> }
> }
> return RexUtil.composeDisjunction(rexBuilder, operands, false);
> case AS_IS:
> default:
> return null;
> }
> }{code}
> And most of calculating SqlKinds are assigned *Policy.ANY* at present.
> {code:java}
> //org.apache.calcite.plan.Strong.java
> public static Policy policy(SqlKind kind) {
> return MAP.getOrDefault(kind, Policy.AS_IS);
> }
>
> map.put(SqlKind.PLUS, Policy.ANY);
> map.put(SqlKind.PLUS_PREFIX, Policy.ANY);
> map.put(SqlKind.MINUS, Policy.ANY);
> map.put(SqlKind.MINUS_PREFIX, Policy.ANY);
> map.put(SqlKind.TIMES, Policy.ANY);
> map.put(SqlKind.DIVIDE, Policy.ANY);
> * that operator evaluates to null. */
> public enum Policy {
> /** This kind of expression is never null. No need to look at its arguments,
>* if it has any. */
> NOT_NULL,
> /** This kind of expression has its own particular rules about whether it
>* is null. */
> CUSTOM,
> /** This kind of expression is null if and only if at least one of its
>* arguments is null. */
> ANY,
> /** This kind of expression may be null. There is no way to rewrite. */
> AS_IS,
> }{code}
>
> It may be an obvious nonequivalent simplification in SQL. And this issue come
> from Flink （FLINK-14030）.
> [~danny0405]， Could you have a look at this?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-3368) Some problems about expression is null simplification
[
https://issues.apache.org/jira/browse/CALCITE-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16940255#comment-16940255
]
Danny Chen edited comment on CALCITE-3368 at 9/29/19 6:14 AM:
--
I tried this test case in SqlToRelConverterTest:
{code:java}
@Test public void testExpressionIsNullSimplify() {
final String sql = "SELECT (empno + sal) is null\n"
+ "FROM emp";
sql(sql).ok();
}
{code}
Now Calcite returns a the plan after sql-to-rel conversion:
{code:sql}
LogicalProject(EXPR$0=[false])
LogicalTableScan(table=[[CATALOG, SALES, EMP]])
{code}
I dug a little and found that the SafeRexVisitor thinks that PLUS is a safe
expression, so "(a + b) is null" can be simplified to "false" if its return
type is not nullable[1].
[~kgyrtkirk], why do you think that "a + b" is a safe expression ? If "a + b"
overflows, then most of the engines would throws exception or just returns
null. But if we simplify "(a + b) is null" to false, we would never have such
behaviors again(Because this simplify happens in sql-to-rel conversion, we even
have no way to configure it).
[1]
[https://github.com/apache/calcite/blob/2dc97e6723e1b5bf762540f87b5cd1a848a1/core/src/main/java/org/apache/calcite/rex/RexSimplify.java#L792]
was (Author: danny0405):
I tried this test case in SqlToRelConverterTest:
{code:java}
@Test public void testExpressionIsNullSimplify() {
final String sql = "SELECT (empno + sal) is null\n"
+ "FROM emp";
sql(sql).ok();
}
{code}
Now Calcite returns a the plan after sql-to-rel conversion:
{code:sql}
LogicalProject(EXPR$0=[false])
LogicalTableScan(table=[[CATALOG, SALES, EMP]])
{code}
I digged a little and found that the SafeRexVisitor thinks that PLUS is a safe
expression, so "(a + b) is null" can be simplified to "false" if its return
type is not nullable[1].
[~kgyrtkirk], why do you think that "a + b" is a safe expression ? If "a + b"
overflows, then most of the engines would throws exception or just returns
null. But if we simplify "(a + b) is null" to false, we would never have such
behaviors again(Because this simplify happens in sql-to-rel conversion, we even
have no way to configure it).
[1]
[https://github.com/apache/calcite/blob/2dc97e6723e1b5bf762540f87b5cd1a848a1/core/src/main/java/org/apache/calcite/rex/RexSimplify.java#L792]
> Some problems about expression is null simplification
> -
>
> Key: CALCITE-3368
> URL: https://issues.apache.org/jira/browse/CALCITE-3368
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.21.0
>Reporter: Leonard Xu
>Assignee: Danny Chen
>Priority: Major
>
> 'is null' expression in SQL may be optimized incorrectly in the underlying
> implementation.
>
> When I write a Fink SQL to test overflow just like
> {code:java}
> select
>case when (f0 + f1) is null then 'null' else 'not null' end
> from testTable
> {code}
> , I found expression '(f0 + f1) is null ' has been optimized by Calcite, and
> the optimization may be incorrect.
>
> The underlying implementation is that Calcite's simplification logic of
> isNull expression in SQL will convert from
> *"f(operand0, operand1) IS NULL"* to
> *"operand0 IS NULL OR operand1 IS NULL"* if the Policy of RexNode‘s SqlKind
> is ANY。
> This simplification leads to the expression will not calculate the real
> value of *f(operand0, operand1)* (eg.. '(f0 + f1)' in my case ），but '(f0 +
> f1)' maybe overflows after operation.
> {code:java}
> //org.apache.calcite.rex.RexSimplify.java
> private RexNode simplifyIsNull(RexNode a) {
> // Simplify the argument first,
> // call ourselves recursively to see whether we can make more progress.
> // For example, given
> // "(CASE WHEN FALSE THEN 1 ELSE 2) IS NULL" we first simplify the
> // argument to "2", and only then we can simplify "2 IS NULL" to "FALSE".
> a = simplify(a, UNKNOWN);
> if (!a.getType().isNullable() && isSafeExpression(a)) {
> return rexBuilder.makeLiteral(false);
> }
> if (RexUtil.isNull(a)) {
> return rexBuilder.makeLiteral(true);
> }
> if (a.getKind() == SqlKind.CAST) {
> return null;
> }
> switch (Strong.policy(a.getKind())) {
> case NOT_NULL:
> return rexBuilder.makeLiteral(false);
> case ANY:
> // "f" is a strong operator, so "f(operand0, operand1) IS NULL" simplifies
> // to "operand0 IS NULL OR operand1 IS NULL"
> final List operands = new ArrayList<>();
> for (RexNode operand : ((RexCall) a).getOperands()) {
> final RexNode simplified = simplifyIsNull(operand);
> if (simplified == null) {
> operands.add(
> rexBuilder.makeCall(SqlStdOperatorTable.IS_NULL, operand));
> } else {
> operands.add(simplified);
> }
> }
> return RexUtil.composeDisjunction(rexBuilder, operan
[jira] [Comment Edited] (CALCITE-3368) Some problems about expression is null simplification
[
https://issues.apache.org/jira/browse/CALCITE-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16940255#comment-16940255
]
Danny Chen edited comment on CALCITE-3368 at 9/29/19 6:32 AM:
--
I tried this test case in SqlToRelConverterTest:
{code:java}
@Test public void testExpressionIsNullSimplify() {
final String sql = "SELECT (empno + sal) is null\n"
+ "FROM emp";
sql(sql).ok();
}
{code}
Now Calcite returns a the plan after sql-to-rel conversion:
{code:sql}
LogicalProject(EXPR$0=[false])
LogicalTableScan(table=[[CATALOG, SALES, EMP]])
{code}
I dug a little and found that the SafeRexVisitor thinks that PLUS is a safe
expression, so "(a + b) is null" can be simplified to "false" if its return
type is not nullable[1].
[~kgyrtkirk], why do you think that "a + b" is a safe expression ? If "a + b"
overflows, then most of the engines would either throw exception or just return
null. But if we simplify "(a + b) is null" to false, there is no change to have
such behaviors again(Because this simplify happens in sql-to-rel conversion, we
even have no way to configure it).
[1]
[https://github.com/apache/calcite/blob/2dc97e6723e1b5bf762540f87b5cd1a848a1/core/src/main/java/org/apache/calcite/rex/RexSimplify.java#L792]
was (Author: danny0405):
I tried this test case in SqlToRelConverterTest:
{code:java}
@Test public void testExpressionIsNullSimplify() {
final String sql = "SELECT (empno + sal) is null\n"
+ "FROM emp";
sql(sql).ok();
}
{code}
Now Calcite returns a the plan after sql-to-rel conversion:
{code:sql}
LogicalProject(EXPR$0=[false])
LogicalTableScan(table=[[CATALOG, SALES, EMP]])
{code}
I dug a little and found that the SafeRexVisitor thinks that PLUS is a safe
expression, so "(a + b) is null" can be simplified to "false" if its return
type is not nullable[1].
[~kgyrtkirk], why do you think that "a + b" is a safe expression ? If "a + b"
overflows, then most of the engines would throws exception or just returns
null. But if we simplify "(a + b) is null" to false, we would never have such
behaviors again(Because this simplify happens in sql-to-rel conversion, we even
have no way to configure it).
[1]
[https://github.com/apache/calcite/blob/2dc97e6723e1b5bf762540f87b5cd1a848a1/core/src/main/java/org/apache/calcite/rex/RexSimplify.java#L792]
> Some problems about expression is null simplification
> -
>
> Key: CALCITE-3368
> URL: https://issues.apache.org/jira/browse/CALCITE-3368
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.21.0
>Reporter: Leonard Xu
>Assignee: Danny Chen
>Priority: Major
>
> 'is null' expression in SQL may be optimized incorrectly in the underlying
> implementation.
>
> When I write a Fink SQL to test overflow just like
> {code:java}
> select
>case when (f0 + f1) is null then 'null' else 'not null' end
> from testTable
> {code}
> , I found expression '(f0 + f1) is null ' has been optimized by Calcite, and
> the optimization may be incorrect.
>
> The underlying implementation is that Calcite's simplification logic of
> isNull expression in SQL will convert from
> *"f(operand0, operand1) IS NULL"* to
> *"operand0 IS NULL OR operand1 IS NULL"* if the Policy of RexNode‘s SqlKind
> is ANY。
> This simplification leads to the expression will not calculate the real
> value of *f(operand0, operand1)* (eg.. '(f0 + f1)' in my case ），but '(f0 +
> f1)' maybe overflows after operation.
> {code:java}
> //org.apache.calcite.rex.RexSimplify.java
> private RexNode simplifyIsNull(RexNode a) {
> // Simplify the argument first,
> // call ourselves recursively to see whether we can make more progress.
> // For example, given
> // "(CASE WHEN FALSE THEN 1 ELSE 2) IS NULL" we first simplify the
> // argument to "2", and only then we can simplify "2 IS NULL" to "FALSE".
> a = simplify(a, UNKNOWN);
> if (!a.getType().isNullable() && isSafeExpression(a)) {
> return rexBuilder.makeLiteral(false);
> }
> if (RexUtil.isNull(a)) {
> return rexBuilder.makeLiteral(true);
> }
> if (a.getKind() == SqlKind.CAST) {
> return null;
> }
> switch (Strong.policy(a.getKind())) {
> case NOT_NULL:
> return rexBuilder.makeLiteral(false);
> case ANY:
> // "f" is a strong operator, so "f(operand0, operand1) IS NULL" simplifies
> // to "operand0 IS NULL OR operand1 IS NULL"
> final List operands = new ArrayList<>();
> for (RexNode operand : ((RexCall) a).getOperands()) {
> final RexNode simplified = simplifyIsNull(operand);
> if (simplified == null) {
> operands.add(
> rexBuilder.makeCall(SqlStdOperatorTable.IS_NULL, operand));
> } else {
> operands.add(simplified);
> }
> }
> return RexUtil.composeDisjunction(rexBuilde
[jira] [Created] (CALCITE-3379) Support expand atom expression in table to relational node conversion
Danny Chen created CALCITE-3379: --- Summary: Support expand atom expression in table to relational node conversion Key: CALCITE-3379 URL: https://issues.apache.org/jira/browse/CALCITE-3379 Project: Calcite Issue Type: Improvement Components: core Affects Versions: 1.21.0 Reporter: Danny Chen Assignee: Danny Chen Fix For: 1.22.0 Now there are 2 ways to convert a RelOptTable to LogicalTableScan: 1. One way is tp open the Config#sConvertTableAccess[1] flag and the SqlToRelConverter would invoke the #toRel method which transforms the table to a node returned by the user(Usually a table scan). 2. Another way it to use the LogicalTableScan rule, this rule would invoke RelOptTable#toRel and wrap the returned node with a LogicalTableScan. The different between 1 and 2 is that, 2 happens in the planning rule but 1 happens in sql-to-rel conversion, 1 also supports to expand the table columns based on the defined default values expressions, see InitializerExpressionFactory#newColumnDefaultValue. The problem with the InitializerExpressionFactory#newColumnDefaultValue is that it use InitializerContext#convertExpression to convert a SqlNode, if the SqlNode is not validated, we always got a RexCall with SqlUnresolvedFunction. We should give the user change to validate their SqlNode or even we can support pure string expressions because they can be used to persist. Another problem with #toRel is that after the expressions applied as a projection, use has no change to apply any other rel nodes if they want, we can actually support this, the same way as we support the column expressions. [1]https://github.com/apache/calcite/blob/2dc97e6723e1b5bf762540f87b5cd1a848a1/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L5605 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3388) StackOverflowError for creating structured RelDataType from class type
[
https://issues.apache.org/jira/browse/CALCITE-3388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16946397#comment-16946397
]
Danny Chen commented on CALCITE-3388:
-
How could a type reference itself ?
{code:java}
private static class RecursionStruct {
public Integer intField;
public RecursionStruct next;
}
{code}
The example you gave is confusing.
> StackOverflowError for creating structured RelDataType from class type
> --
>
> Key: CALCITE-3388
> URL: https://issues.apache.org/jira/browse/CALCITE-3388
> Project: Calcite
> Issue Type: Bug
>Reporter: Wang Yanlin
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> When creating a structured RelDataType from a java type with recursion
> reference,
> StackOverflowError occurs, full stack trace
>
> {noformat}
> java.lang.StackOverflowError
> at java.lang.Class.privateGetPublicFields(Class.java:2600)
> at java.lang.Class.getFields(Class.java:1557)
> at
> org.apache.calcite.rel.type.RelDataTypeFactoryImpl.fieldsOf(RelDataTypeFactoryImpl.java:440)
> at
> org.apache.calcite.rel.type.RelDataTypeFactoryImpl.access$700(RelDataTypeFactoryImpl.java:50)
> at
> org.apache.calcite.rel.type.RelDataTypeFactoryImpl$JavaType.(RelDataTypeFactoryImpl.java:568)
> at
> org.apache.calcite.rel.type.RelDataTypeFactoryImpl$JavaType.(RelDataTypeFactoryImpl.java:560)
> at
> org.apache.calcite.rel.type.RelDataTypeFactoryImpl$JavaType.(RelDataTypeFactoryImpl.java:554)
> at
> org.apache.calcite.rel.type.RelDataTypeFactoryImpl.createJavaType(RelDataTypeFactoryImpl.java:120)
> at
> org.apache.calcite.jdbc.JavaTypeFactoryImpl.createType(JavaTypeFactoryImpl.java:143)
> at
> org.apache.calcite.jdbc.JavaTypeFactoryImpl.createStructType(JavaTypeFactoryImpl.java:82)
> at
> org.apache.calcite.jdbc.JavaTypeFactoryImpl.createType(JavaTypeFactoryImpl.java:158)
> at
> org.apache.calcite.jdbc.JavaTypeFactoryImpl.createStructType(JavaTypeFactoryImpl.java:82)
> at
> org.apache.calcite.jdbc.JavaTypeFactoryImpl.createType(JavaTypeFactoryImpl.java:158)
> at
> org.apache.calcite.jdbc.JavaTypeFactoryImpl.createStructType(JavaTypeFactoryImpl.java:82)
> at
> org.apache.calcite.jdbc.JavaTypeFactoryImpl.createType(JavaTypeFactoryImpl.java:158)
> at
> org.apache.calcite.jdbc.JavaTypeFactoryImpl.createStructType(JavaTypeFactoryImpl.java:82)
> {noformat}
> Add the test case in JavaTypeFactoryTest to reproduce the exception
> {noformat}
> /***/
> private static class RecursionStruct {
> public Integer intField;
> public RecursionStruct next;
> }
> /***/
> private static class RecursionStruct1 {
> public Integer intField;
> public RecursionStruct2 struct2;
> }
> /***/
> private static class RecursionStruct2 {
> public Integer intField;
> public RecursionStruct1 struct1;
> }
> @Test public void testRecursion() {
> TYPE_FACTORY.createStructType(RecursionStruct.class);
> TYPE_FACTORY.createStructType(RecursionStruct1.class);
> }
> {noformat}
>
> Better to check for recursion and throw an exception explicitly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3358) Make Function DDLs implement SqlExecutableStatement
[ https://issues.apache.org/jira/browse/CALCITE-3358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16946417#comment-16946417 ] Danny Chen commented on CALCITE-3358: - Not able to, seems you do not have the JIRA contribution permission. > Make Function DDLs implement SqlExecutableStatement > --- > > Key: CALCITE-3358 > URL: https://issues.apache.org/jira/browse/CALCITE-3358 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Zhenqiu Huang >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3368) Some problems simplifying ‘expression IS NULL’
[
https://issues.apache.org/jira/browse/CALCITE-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16946421#comment-16946421
]
Danny Chen commented on CALCITE-3368:
-
[~kgyrtkirk] Can you please take a look on this issue ?
> Some problems simplifying ‘expression IS NULL’
> --
>
> Key: CALCITE-3368
> URL: https://issues.apache.org/jira/browse/CALCITE-3368
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.21.0
>Reporter: Leonard Xu
>Assignee: Danny Chen
>Priority: Major
>
> 'is null' expression in SQL may be optimized incorrectly in the underlying
> implementation.
>
> When I write a Fink SQL to test overflow just like
> {code:java}
> select
>case when (f0 + f1) is null then 'null' else 'not null' end
> from testTable
> {code}
> , I found expression '(f0 + f1) is null ' has been optimized by Calcite, and
> the optimization may be incorrect.
>
> The underlying implementation is that Calcite's simplification logic of
> isNull expression in SQL will convert from
> *"f(operand0, operand1) IS NULL"* to
> *"operand0 IS NULL OR operand1 IS NULL"* if the Policy of RexNode‘s SqlKind
> is ANY。
> This simplification leads to the expression will not calculate the real
> value of *f(operand0, operand1)* (eg.. '(f0 + f1)' in my case ），but '(f0 +
> f1)' maybe overflows after operation.
> {code:java}
> //org.apache.calcite.rex.RexSimplify.java
> private RexNode simplifyIsNull(RexNode a) {
> // Simplify the argument first,
> // call ourselves recursively to see whether we can make more progress.
> // For example, given
> // "(CASE WHEN FALSE THEN 1 ELSE 2) IS NULL" we first simplify the
> // argument to "2", and only then we can simplify "2 IS NULL" to "FALSE".
> a = simplify(a, UNKNOWN);
> if (!a.getType().isNullable() && isSafeExpression(a)) {
> return rexBuilder.makeLiteral(false);
> }
> if (RexUtil.isNull(a)) {
> return rexBuilder.makeLiteral(true);
> }
> if (a.getKind() == SqlKind.CAST) {
> return null;
> }
> switch (Strong.policy(a.getKind())) {
> case NOT_NULL:
> return rexBuilder.makeLiteral(false);
> case ANY:
> // "f" is a strong operator, so "f(operand0, operand1) IS NULL" simplifies
> // to "operand0 IS NULL OR operand1 IS NULL"
> final List operands = new ArrayList<>();
> for (RexNode operand : ((RexCall) a).getOperands()) {
> final RexNode simplified = simplifyIsNull(operand);
> if (simplified == null) {
> operands.add(
> rexBuilder.makeCall(SqlStdOperatorTable.IS_NULL, operand));
> } else {
> operands.add(simplified);
> }
> }
> return RexUtil.composeDisjunction(rexBuilder, operands, false);
> case AS_IS:
> default:
> return null;
> }
> }{code}
> And most of calculating SqlKinds are assigned *Policy.ANY* at present.
> {code:java}
> //org.apache.calcite.plan.Strong.java
> public static Policy policy(SqlKind kind) {
> return MAP.getOrDefault(kind, Policy.AS_IS);
> }
>
> map.put(SqlKind.PLUS, Policy.ANY);
> map.put(SqlKind.PLUS_PREFIX, Policy.ANY);
> map.put(SqlKind.MINUS, Policy.ANY);
> map.put(SqlKind.MINUS_PREFIX, Policy.ANY);
> map.put(SqlKind.TIMES, Policy.ANY);
> map.put(SqlKind.DIVIDE, Policy.ANY);
> * that operator evaluates to null. */
> public enum Policy {
> /** This kind of expression is never null. No need to look at its arguments,
>* if it has any. */
> NOT_NULL,
> /** This kind of expression has its own particular rules about whether it
>* is null. */
> CUSTOM,
> /** This kind of expression is null if and only if at least one of its
>* arguments is null. */
> ANY,
> /** This kind of expression may be null. There is no way to rewrite. */
> AS_IS,
> }{code}
>
> It may be an obvious nonequivalent simplification in SQL. And this issue come
> from Flink （FLINK-14030）.
> [~danny0405]， Could you have a look at this?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-3379) Support expand atom expression in table to relational node conversion
[ https://issues.apache.org/jira/browse/CALCITE-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated CALCITE-3379: Description: Now there are 2 ways to convert a RelOptTable to LogicalTableScan: 1. One way is to open the Config#isConvertTableAccess[1] flag and the SqlToRelConverter would invoke the #toRel method which transforms the table to a node returned by the user(Usually a table scan). 2. Another way is to use the LogicalTableScan rule, this rule would invoke RelOptTable#toRel and wrap the returned node with a LogicalTableScan. The difference between 1 and 2 is that, 2 happens in the planning rule but 1 happens in sql-to-rel conversion, 1 also supports to expand the table columns based on the defined default values expressions, see InitializerExpressionFactory#newColumnDefaultValue. The problem with the InitializerExpressionFactory#newColumnDefaultValue is that it uses InitializerContext#convertExpression to convert a SqlNode, but if the SqlNode is not validated, we always got a RexCall with SqlUnresolvedFunction. We should give the user chance to validate their SqlNode or even we can support pure string expressions which can be used to persist. Another problem with #toRel is that after the expressions applied as a projection, user has no chance to apply any other relational nodes if they want, we can actually support this, the same way as we support the column expressions. [1]https://github.com/apache/calcite/blob/2dc97e6723e1b5bf762540f87b5cd1a848a1/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L5605 was: Now there are 2 ways to convert a RelOptTable to LogicalTableScan: 1. One way is tp open the Config#sConvertTableAccess[1] flag and the SqlToRelConverter would invoke the #toRel method which transforms the table to a node returned by the user(Usually a table scan). 2. Another way it to use the LogicalTableScan rule, this rule would invoke RelOptTable#toRel and wrap the returned node with a LogicalTableScan. The different between 1 and 2 is that, 2 happens in the planning rule but 1 happens in sql-to-rel conversion, 1 also supports to expand the table columns based on the defined default values expressions, see InitializerExpressionFactory#newColumnDefaultValue. The problem with the InitializerExpressionFactory#newColumnDefaultValue is that it use InitializerContext#convertExpression to convert a SqlNode, if the SqlNode is not validated, we always got a RexCall with SqlUnresolvedFunction. We should give the user change to validate their SqlNode or even we can support pure string expressions because they can be used to persist. Another problem with #toRel is that after the expressions applied as a projection, use has no change to apply any other rel nodes if they want, we can actually support this, the same way as we support the column expressions. [1]https://github.com/apache/calcite/blob/2dc97e6723e1b5bf762540f87b5cd1a848a1/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L5605 > Support expand atom expression in table to relational node conversion > - > > Key: CALCITE-3379 > URL: https://issues.apache.org/jira/browse/CALCITE-3379 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.21.0 >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 1.22.0 > > Time Spent: 2h > Remaining Estimate: 0h > > Now there are 2 ways to convert a RelOptTable to LogicalTableScan: > 1. One way is to open the Config#isConvertTableAccess[1] flag and the > SqlToRelConverter would invoke the #toRel method which transforms the table > to a node returned by the user(Usually a table scan). > 2. Another way is to use the LogicalTableScan rule, this rule would invoke > RelOptTable#toRel and wrap the returned node with a LogicalTableScan. > > The difference between 1 and 2 is that, 2 happens in the planning rule but 1 > happens in sql-to-rel conversion, 1 also supports to expand the table columns > based on the defined default values expressions, see > InitializerExpressionFactory#newColumnDefaultValue. > > The problem with the InitializerExpressionFactory#newColumnDefaultValue is > that it uses InitializerContext#convertExpression to convert a SqlNode, but > if the SqlNode is not validated, we always got a RexCall with > SqlUnresolvedFunction. We should give the user chance to validate their > SqlNode or even we can support pure string expressions which can be used to > persist. > > Another problem with #toRel is that after the expressions applied as a > projection, user has no chance to apply any other relational nodes if the
[jira] [Updated] (CALCITE-3379) Support expand STRING column expression in table during sql-to-rel conversion
[ https://issues.apache.org/jira/browse/CALCITE-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated CALCITE-3379: Summary: Support expand STRING column expression in table during sql-to-rel conversion (was: Support expand atom expression in table to relational node conversion) > Support expand STRING column expression in table during sql-to-rel conversion > - > > Key: CALCITE-3379 > URL: https://issues.apache.org/jira/browse/CALCITE-3379 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.21.0 >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 1.22.0 > > Time Spent: 2h > Remaining Estimate: 0h > > Now there are 2 ways to convert a RelOptTable to LogicalTableScan: > 1. One way is to open the Config#isConvertTableAccess[1] flag and the > SqlToRelConverter would invoke the #toRel method which transforms the table > to a node returned by the user(Usually a table scan). > 2. Another way is to use the LogicalTableScan rule, this rule would invoke > RelOptTable#toRel and wrap the returned node with a LogicalTableScan. > > The difference between 1 and 2 is that, 2 happens in the planning rule but 1 > happens in sql-to-rel conversion, 1 also supports to expand the table columns > based on the defined default values expressions, see > InitializerExpressionFactory#newColumnDefaultValue. > > The problem with the InitializerExpressionFactory#newColumnDefaultValue is > that it uses InitializerContext#convertExpression to convert a SqlNode, but > if the SqlNode is not validated, we always got a RexCall with > SqlUnresolvedFunction. We should give the user chance to validate their > SqlNode or even we can support pure string expressions which can be used to > persist. > > Another problem with #toRel is that after the expressions applied as a > projection, user has no chance to apply any other relational nodes if they > want, we can actually support this, the same way as we support the column > expressions. > > [1]https://github.com/apache/calcite/blob/2dc97e6723e1b5bf762540f87b5cd1a848a1/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L5605 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3379) Support expand STRING column expression in table during sql-to-rel conversion
[ https://issues.apache.org/jira/browse/CALCITE-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16946450#comment-16946450 ] Danny Chen commented on CALCITE-3379: - [~julianhyde] [~hyuan],yes, the linked PR is not relevant with this JIRA, maybe i mis-clicked. Let me re-state the use case: Now user can declare the column of a table in DDL with special strategy(known as column constraints in other sql engines[1] [2]), as we enumerate in ColumnStrategy[3], then with the SqlNode got from InitializerExpressionFactory#newColumnDefaultValue, SqlToRelConverter would translate it to a RexCall and put it the project list above the table. InitializerExpressionFactory#newColumnDefaultValue use the InitializerContext to convert the SqlNode, the problem is that: # SqlNode got from the DDL is not validated(we have no validation for DDL nodes yet, and i think we don'tneed to) and we always got a RexCall with SqlUnresolvedFunction. # The SqlNode can not be serialized directly, many times the DDL definition need to persist into the Catalog as a metadata(we may unparse it to pure sql but that is not intuitive and we still need use to parse it back to SqlNode when deserializing) So, what i proposed is that add the InitializerContext#convertExpression(String) and deprecate the #convertExpression(SqlNode), Calcite is the role to parse the string to SqlNode then validate and translate it to RexCall. What the user see is always a string and it can be persisted. > Support expand STRING column expression in table during sql-to-rel conversion > - > > Key: CALCITE-3379 > URL: https://issues.apache.org/jira/browse/CALCITE-3379 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.21.0 >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 1.22.0 > > Time Spent: 2h > Remaining Estimate: 0h > > Now there are 2 ways to convert a RelOptTable to LogicalTableScan: > 1. One way is to open the Config#isConvertTableAccess[1] flag and the > SqlToRelConverter would invoke the #toRel method which transforms the table > to a node returned by the user(Usually a table scan). > 2. Another way is to use the LogicalTableScan rule, this rule would invoke > RelOptTable#toRel and wrap the returned node with a LogicalTableScan. > > The difference between 1 and 2 is that, 2 happens in the planning rule but 1 > happens in sql-to-rel conversion, 1 also supports to expand the table columns > based on the defined default values expressions, see > InitializerExpressionFactory#newColumnDefaultValue. > > The problem with the InitializerExpressionFactory#newColumnDefaultValue is > that it uses InitializerContext#convertExpression to convert a SqlNode, but > if the SqlNode is not validated, we always got a RexCall with > SqlUnresolvedFunction. We should give the user chance to validate their > SqlNode or even we can support pure string expressions which can be used to > persist. > > Another problem with #toRel is that after the expressions applied as a > projection, user has no chance to apply any other relational nodes if they > want, we can actually support this, the same way as we support the column > expressions. > > [1]https://github.com/apache/calcite/blob/2dc97e6723e1b5bf762540f87b5cd1a848a1/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L5605 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-3379) Support expand STRING column expression in table during sql-to-rel conversion
[ https://issues.apache.org/jira/browse/CALCITE-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16946450#comment-16946450 ] Danny Chen edited comment on CALCITE-3379 at 10/8/19 3:37 AM: -- [~julianhyde] [~hyuan],yes, the linked PR is not relevant with this JIRA, maybe i mis-clicked. Let me re-state the use case: Now user can declare the column of a table in DDL with special strategy(known as column constraints in other sql engines[1] [2]), as we enumerate in ColumnStrategy[3], then with the SqlNode got from InitializerExpressionFactory#newColumnDefaultValue, SqlToRelConverter would translate it to a RexCall and put it the project list above the table. InitializerExpressionFactory#newColumnDefaultValue use the InitializerContext to convert the SqlNode, the problem is that: # SqlNode got from the DDL is not validated(we have no validation for DDL nodes yet, and i think we don'tneed to) and we always got a RexCall with SqlUnresolvedFunction. # The SqlNode can not be serialized directly, many times the DDL definition need to persist into the Catalog as a metadata(we may unparse it to pure sql but that is not intuitive and we still need use to parse it back to SqlNode when deserializing) So, what i proposed is that add the InitializerContext#convertExpression(String) and deprecate the #convertExpression(SqlNode), Calcite is the role to parse the string to SqlNode then validate and translate it to RexCall. What the user see is always a string and it can be persisted. [1][https://dev.mysql.com/doc/refman/5.7/en/create-table-generated-columns.html] [2][https://oracle-base.com/articles/11g/virtual-columns-11gr1] [3][https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/schema/ColumnStrategy.java] was (Author: danny0405): [~julianhyde] [~hyuan],yes, the linked PR is not relevant with this JIRA, maybe i mis-clicked. Let me re-state the use case: Now user can declare the column of a table in DDL with special strategy(known as column constraints in other sql engines[1] [2]), as we enumerate in ColumnStrategy[3], then with the SqlNode got from InitializerExpressionFactory#newColumnDefaultValue, SqlToRelConverter would translate it to a RexCall and put it the project list above the table. InitializerExpressionFactory#newColumnDefaultValue use the InitializerContext to convert the SqlNode, the problem is that: # SqlNode got from the DDL is not validated(we have no validation for DDL nodes yet, and i think we don'tneed to) and we always got a RexCall with SqlUnresolvedFunction. # The SqlNode can not be serialized directly, many times the DDL definition need to persist into the Catalog as a metadata(we may unparse it to pure sql but that is not intuitive and we still need use to parse it back to SqlNode when deserializing) So, what i proposed is that add the InitializerContext#convertExpression(String) and deprecate the #convertExpression(SqlNode), Calcite is the role to parse the string to SqlNode then validate and translate it to RexCall. What the user see is always a string and it can be persisted. > Support expand STRING column expression in table during sql-to-rel conversion > - > > Key: CALCITE-3379 > URL: https://issues.apache.org/jira/browse/CALCITE-3379 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.21.0 >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 1.22.0 > > Time Spent: 2h > Remaining Estimate: 0h > > Now there are 2 ways to convert a RelOptTable to LogicalTableScan: > 1. One way is to open the Config#isConvertTableAccess[1] flag and the > SqlToRelConverter would invoke the #toRel method which transforms the table > to a node returned by the user(Usually a table scan). > 2. Another way is to use the LogicalTableScan rule, this rule would invoke > RelOptTable#toRel and wrap the returned node with a LogicalTableScan. > > The difference between 1 and 2 is that, 2 happens in the planning rule but 1 > happens in sql-to-rel conversion, 1 also supports to expand the table columns > based on the defined default values expressions, see > InitializerExpressionFactory#newColumnDefaultValue. > > The problem with the InitializerExpressionFactory#newColumnDefaultValue is > that it uses InitializerContext#convertExpression to convert a SqlNode, but > if the SqlNode is not validated, we always got a RexCall with > SqlUnresolvedFunction. We should give the user chance to validate their > SqlNode or even we can support pure string expressions which can be used to > persist. > > Another problem with #toRel is t
[jira] [Created] (CALCITE-3392) Column expression in DDL should be validated before converting to RexNode
Danny Chen created CALCITE-3392: --- Summary: Column expression in DDL should be validated before converting to RexNode Key: CALCITE-3392 URL: https://issues.apache.org/jira/browse/CALCITE-3392 Project: Calcite Issue Type: Sub-task Components: core Affects Versions: 1.21.0 Reporter: Danny Chen Assignee: Danny Chen Fix For: 1.22.0 As described in CALCITE-3379, we should validate the column expression so that the expression can be converted correctly. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-2221) RelRoot.project does not take into account flattened record type
[
https://issues.apache.org/jira/browse/CALCITE-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947598#comment-16947598
]
Danny Chen commented on CALCITE-2221:
-
Thanks for the reminder [~IhorHuzenko] !
> RelRoot.project does not take into account flattened record type
>
>
> Key: CALCITE-2221
> URL: https://issues.apache.org/jira/browse/CALCITE-2221
> Project: Calcite
> Issue Type: Task
> Components: core
>Reporter: Shuyi Chen
>Assignee: Igor Guzenko
>Priority: Major
> Fix For: 1.21.0
>
>
> To reproduce, simply run:
> @Test public void testFlattenRecords() {
> sql("select employees[1] from dept_nested").ok();
> }
> It will yield:
> LogicalProject(EXPR$0=[$0])
> LogicalProject(EXPR$0$0=[ITEM($2, 1).EMPNO], EXPR$0$1=[ITEM($2, 1).ENAME],
> EXPR$0$2=[ITEM($2, 1).DETAIL])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT_NESTED]])
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (CALCITE-3368) Some problems simplifying ‘expression IS NULL’
[
https://issues.apache.org/jira/browse/CALCITE-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen reassigned CALCITE-3368:
---
Assignee: Leonard Xu (was: Danny Chen)
> Some problems simplifying ‘expression IS NULL’
> --
>
> Key: CALCITE-3368
> URL: https://issues.apache.org/jira/browse/CALCITE-3368
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.21.0
>Reporter: Leonard Xu
>Assignee: Leonard Xu
>Priority: Major
>
> 'is null' expression in SQL may be optimized incorrectly in the underlying
> implementation.
>
> When I write a Fink SQL to test overflow just like
> {code:java}
> select
>case when (f0 + f1) is null then 'null' else 'not null' end
> from testTable
> {code}
> , I found expression '(f0 + f1) is null ' has been optimized by Calcite, and
> the optimization may be incorrect.
>
> The underlying implementation is that Calcite's simplification logic of
> isNull expression in SQL will convert from
> *"f(operand0, operand1) IS NULL"* to
> *"operand0 IS NULL OR operand1 IS NULL"* if the Policy of RexNode‘s SqlKind
> is ANY。
> This simplification leads to the expression will not calculate the real
> value of *f(operand0, operand1)* (eg.. '(f0 + f1)' in my case ），but '(f0 +
> f1)' maybe overflows after operation.
> {code:java}
> //org.apache.calcite.rex.RexSimplify.java
> private RexNode simplifyIsNull(RexNode a) {
> // Simplify the argument first,
> // call ourselves recursively to see whether we can make more progress.
> // For example, given
> // "(CASE WHEN FALSE THEN 1 ELSE 2) IS NULL" we first simplify the
> // argument to "2", and only then we can simplify "2 IS NULL" to "FALSE".
> a = simplify(a, UNKNOWN);
> if (!a.getType().isNullable() && isSafeExpression(a)) {
> return rexBuilder.makeLiteral(false);
> }
> if (RexUtil.isNull(a)) {
> return rexBuilder.makeLiteral(true);
> }
> if (a.getKind() == SqlKind.CAST) {
> return null;
> }
> switch (Strong.policy(a.getKind())) {
> case NOT_NULL:
> return rexBuilder.makeLiteral(false);
> case ANY:
> // "f" is a strong operator, so "f(operand0, operand1) IS NULL" simplifies
> // to "operand0 IS NULL OR operand1 IS NULL"
> final List operands = new ArrayList<>();
> for (RexNode operand : ((RexCall) a).getOperands()) {
> final RexNode simplified = simplifyIsNull(operand);
> if (simplified == null) {
> operands.add(
> rexBuilder.makeCall(SqlStdOperatorTable.IS_NULL, operand));
> } else {
> operands.add(simplified);
> }
> }
> return RexUtil.composeDisjunction(rexBuilder, operands, false);
> case AS_IS:
> default:
> return null;
> }
> }{code}
> And most of calculating SqlKinds are assigned *Policy.ANY* at present.
> {code:java}
> //org.apache.calcite.plan.Strong.java
> public static Policy policy(SqlKind kind) {
> return MAP.getOrDefault(kind, Policy.AS_IS);
> }
>
> map.put(SqlKind.PLUS, Policy.ANY);
> map.put(SqlKind.PLUS_PREFIX, Policy.ANY);
> map.put(SqlKind.MINUS, Policy.ANY);
> map.put(SqlKind.MINUS_PREFIX, Policy.ANY);
> map.put(SqlKind.TIMES, Policy.ANY);
> map.put(SqlKind.DIVIDE, Policy.ANY);
> * that operator evaluates to null. */
> public enum Policy {
> /** This kind of expression is never null. No need to look at its arguments,
>* if it has any. */
> NOT_NULL,
> /** This kind of expression has its own particular rules about whether it
>* is null. */
> CUSTOM,
> /** This kind of expression is null if and only if at least one of its
>* arguments is null. */
> ANY,
> /** This kind of expression may be null. There is no way to rewrite. */
> AS_IS,
> }{code}
>
> It may be an obvious nonequivalent simplification in SQL. And this issue come
> from Flink （FLINK-14030）.
> [~danny0405]， Could you have a look at this?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3368) Some problems simplifying ‘expression IS NULL’
[
https://issues.apache.org/jira/browse/CALCITE-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947635#comment-16947635
]
Danny Chen commented on CALCITE-3368:
-
Thanks for taking this issue, [~Leonard Xu] !
> Some problems simplifying ‘expression IS NULL’
> --
>
> Key: CALCITE-3368
> URL: https://issues.apache.org/jira/browse/CALCITE-3368
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.21.0
>Reporter: Leonard Xu
>Assignee: Leonard Xu
>Priority: Major
>
> 'is null' expression in SQL may be optimized incorrectly in the underlying
> implementation.
>
> When I write a Fink SQL to test overflow just like
> {code:java}
> select
>case when (f0 + f1) is null then 'null' else 'not null' end
> from testTable
> {code}
> , I found expression '(f0 + f1) is null ' has been optimized by Calcite, and
> the optimization may be incorrect.
>
> The underlying implementation is that Calcite's simplification logic of
> isNull expression in SQL will convert from
> *"f(operand0, operand1) IS NULL"* to
> *"operand0 IS NULL OR operand1 IS NULL"* if the Policy of RexNode‘s SqlKind
> is ANY。
> This simplification leads to the expression will not calculate the real
> value of *f(operand0, operand1)* (eg.. '(f0 + f1)' in my case ），but '(f0 +
> f1)' maybe overflows after operation.
> {code:java}
> //org.apache.calcite.rex.RexSimplify.java
> private RexNode simplifyIsNull(RexNode a) {
> // Simplify the argument first,
> // call ourselves recursively to see whether we can make more progress.
> // For example, given
> // "(CASE WHEN FALSE THEN 1 ELSE 2) IS NULL" we first simplify the
> // argument to "2", and only then we can simplify "2 IS NULL" to "FALSE".
> a = simplify(a, UNKNOWN);
> if (!a.getType().isNullable() && isSafeExpression(a)) {
> return rexBuilder.makeLiteral(false);
> }
> if (RexUtil.isNull(a)) {
> return rexBuilder.makeLiteral(true);
> }
> if (a.getKind() == SqlKind.CAST) {
> return null;
> }
> switch (Strong.policy(a.getKind())) {
> case NOT_NULL:
> return rexBuilder.makeLiteral(false);
> case ANY:
> // "f" is a strong operator, so "f(operand0, operand1) IS NULL" simplifies
> // to "operand0 IS NULL OR operand1 IS NULL"
> final List operands = new ArrayList<>();
> for (RexNode operand : ((RexCall) a).getOperands()) {
> final RexNode simplified = simplifyIsNull(operand);
> if (simplified == null) {
> operands.add(
> rexBuilder.makeCall(SqlStdOperatorTable.IS_NULL, operand));
> } else {
> operands.add(simplified);
> }
> }
> return RexUtil.composeDisjunction(rexBuilder, operands, false);
> case AS_IS:
> default:
> return null;
> }
> }{code}
> And most of calculating SqlKinds are assigned *Policy.ANY* at present.
> {code:java}
> //org.apache.calcite.plan.Strong.java
> public static Policy policy(SqlKind kind) {
> return MAP.getOrDefault(kind, Policy.AS_IS);
> }
>
> map.put(SqlKind.PLUS, Policy.ANY);
> map.put(SqlKind.PLUS_PREFIX, Policy.ANY);
> map.put(SqlKind.MINUS, Policy.ANY);
> map.put(SqlKind.MINUS_PREFIX, Policy.ANY);
> map.put(SqlKind.TIMES, Policy.ANY);
> map.put(SqlKind.DIVIDE, Policy.ANY);
> * that operator evaluates to null. */
> public enum Policy {
> /** This kind of expression is never null. No need to look at its arguments,
>* if it has any. */
> NOT_NULL,
> /** This kind of expression has its own particular rules about whether it
>* is null. */
> CUSTOM,
> /** This kind of expression is null if and only if at least one of its
>* arguments is null. */
> ANY,
> /** This kind of expression may be null. There is no way to rewrite. */
> AS_IS,
> }{code}
>
> It may be an obvious nonequivalent simplification in SQL. And this issue come
> from Flink （FLINK-14030）.
> [~danny0405]， Could you have a look at this?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3394) Can not register implementation of a UDF with RexImpTable and use the same with RelBuilder
[
https://issues.apache.org/jira/browse/CALCITE-3394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948122#comment-16948122
]
Danny Chen commented on CALCITE-3394:
-
[~vaishnavee] This is because
SqlStdOperatorTable.instance()
did some initialization work for the registered functions. So an invoke of
#register after it would not work as expected.
The correct way is to use ListSqlOperatorTable and chained it with the
StdSqlOperatorTable with ChainedSqlOperatorTable, the presudo code may like
this:
{code:java}
ListSqlOperatorTable listOpTable = new ListSqlOperatorTable();
listOpTable.add(my_udf);
ChainedSqlOperatorTable chainedOpTable =
ChainedSqlOperatorTable.of(listOpTable, SqlStdOperatorTable.instance());
// then use this chainedOpTable
// If you want to use a special dialect operators, you can code like this
SqlOperatorTable optable = SqlLibraryOperatorTableFactory.INSTANCE
.getOperatorTable(SqlLibrary.STANDARD, SqlLibrary.POSTGRESQL);{code}
> Can not register implementation of a UDF with RexImpTable and use the same
> with RelBuilder
> --
>
> Key: CALCITE-3394
> URL: https://issues.apache.org/jira/browse/CALCITE-3394
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.21.0
>Reporter: Vaishnavee Kulkarni
>Priority: Major
>
> I am trying to register a simple UDF that returns the length of input string.
> I can do this with prepareStatement approach -
> {code:java}
> public static class MyUdf1 {
>public Integer eval(String a) {
> return a.length();
>}
> }
> public void testUDF() {
> final String strLenSql = "select STRLEN('SampleString') from emp";
> ScalarFunction strLenFunction = ScalarFunctionImpl.create(MyUdf1.class,
> "eval");
> calciteConnection.getRootSchema().add("STRLEN", strLenFunction);
> ResultSet resultSet =
> calciteConnection.prepareStatement(strLenSql).executeQuery();
> resultSet.next();
> System.out.println(resultSet.getString(1));
> }
> {code}
>
> When I try the similar steps with _RelBuilder_, I can successfully register
> the _SqlOperator_; but am unable to refer to the implementation of this
> operator. The builder refers to _RexImpTable_'s maps for the function table
> implementation and there is no public/protected api exposed for these maps.
> {code:java}
> SqlFunction length = new SqlFunction("STRLEN",
> SqlKind.OTHER_FUNCTION,
> ReturnTypes.INTEGER,
> null,
> OperandTypes.STRING,
> SqlFunctionCategory.USER_DEFINED_FUNCTION);
> SqlStdOperatorTable sqlStdOperatorTable = SqlStdOperatorTable.instance();
> sqlStdOperatorTable.register(length);
> FrameworkConfig frameworkConfig = Frameworks.newConfigBuilder()
> .parserConfig(SqlParser.Config.DEFAULT)
> .defaultSchema(connection.getRootSchema().getSubSchema("SYSTEM"))
> .programs(Programs.sequence(Programs.ofRules(Programs.RULE_SET),
> Programs.CALC_PROGRAM))
> .operatorTable(sqlStdOperatorTable)
> .build();
> final RelBuilder builder = RelBuilder.create(frameworkConfig);
> RelNode udfRelNode = builder
> .scan("EMP")
> .project(builder.call(length,builder.literal("SampleString")))
> .build();
> ResultSet resultSet = RelRunners.run(udfRelNode).executeQuery();
> {code}
>
> This code throws exception -
> {code:java}
> Caused by: java.lang.RuntimeException: cannot translate call
> STRLEN($t3)Caused by: java.lang.RuntimeException: cannot translate call
> STRLEN($t3) at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.translateCall(RexToLixTranslator.java:756)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.translate0(RexToLixTranslator.java:730)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.translate(RexToLixTranslator.java:199)
> at
> org.apache.calcite.adapter.enumerable.RexToLixTranslator.translate0(RexToLixTranslator.java:684)
> {code}
> There are no junits that show this working with _RelBuilder_. Is it possible
> currently to register and use the udfs with RelBuilder?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3395) add BuiltinMethod for Substring(String, int)
[ https://issues.apache.org/jira/browse/CALCITE-3395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948127#comment-16948127 ] Danny Chen commented on CALCITE-3395: - Calcite support 2 style of subString function: # One is SUBSTRING(str FROM pos FOR len) defined in SqlStdOperatorTable[1] # Another is SUBSTR(string, position [, substringLength ]) defined in SqlLibraryOperators as a ORACLE dialect operator[2] [1] https://github.com/apache/calcite/blob/e43595482617b1bb991407800c068a0c727e5c17/core/src/main/java/org/apache/calcite/sql/fun/SqlStdOperatorTable.java#L1443 [2] https://github.com/apache/calcite/blob/e43595482617b1bb991407800c068a0c727e5c17/core/src/main/java/org/apache/calcite/sql/fun/SqlLibraryOperators.java#L118 > add BuiltinMethod for Substring(String, int) > > > Key: CALCITE-3395 > URL: https://issues.apache.org/jira/browse/CALCITE-3395 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Youjun Yuan >Priority: Minor > Original Estimate: 4h > Remaining Estimate: 4h > > substring function has two versoins: > 1, Substring(String, int, int) > 2, Substring(String, int) > currently in BuiltinMethod.java, only the first one is defined. Need to > define the second one as well, so that we can use both of them. > Apache Flink(FunctionGenerator.scala), reference the BuiltinMethod, since > Calcite only defines 1, not 2, Flink always resolve SUBSTRING to > Substring(String, int, int) even if there is only 2 parameters. This problem > happens to be covered by method overloading of java, but it's still a > potential problem. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CALCITE-3392) Column expression in DDL should be validated before converting to RexNode
[ https://issues.apache.org/jira/browse/CALCITE-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen resolved CALCITE-3392. - Resolution: Fixed Fixed in [aafa583|https://github.com/apache/calcite/commit/aafa583234ef160f521e4facb5648d3838f37067] ! > Column expression in DDL should be validated before converting to RexNode > - > > Key: CALCITE-3392 > URL: https://issues.apache.org/jira/browse/CALCITE-3392 > Project: Calcite > Issue Type: Sub-task > Components: core >Affects Versions: 1.21.0 >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 1.22.0 > > Time Spent: 20m > Remaining Estimate: 0h > > As described in CALCITE-3379, we should validate the column expression so > that the expression can be converted correctly. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3379) Support expand STRING column expression in table during sql-to-rel conversion
[ https://issues.apache.org/jira/browse/CALCITE-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948187#comment-16948187 ] Danny Chen commented on CALCITE-3379: - Yes, column expression need to be validated, this is resolved in CALCITE-3392. > Support expand STRING column expression in table during sql-to-rel conversion > - > > Key: CALCITE-3379 > URL: https://issues.apache.org/jira/browse/CALCITE-3379 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.21.0 >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 1.22.0 > > Time Spent: 2h > Remaining Estimate: 0h > > Now there are 2 ways to convert a RelOptTable to LogicalTableScan: > 1. One way is to open the Config#isConvertTableAccess[1] flag and the > SqlToRelConverter would invoke the #toRel method which transforms the table > to a node returned by the user(Usually a table scan). > 2. Another way is to use the LogicalTableScan rule, this rule would invoke > RelOptTable#toRel and wrap the returned node with a LogicalTableScan. > > The difference between 1 and 2 is that, 2 happens in the planning rule but 1 > happens in sql-to-rel conversion, 1 also supports to expand the table columns > based on the defined default values expressions, see > InitializerExpressionFactory#newColumnDefaultValue. > > The problem with the InitializerExpressionFactory#newColumnDefaultValue is > that it uses InitializerContext#convertExpression to convert a SqlNode, but > if the SqlNode is not validated, we always got a RexCall with > SqlUnresolvedFunction. We should give the user chance to validate their > SqlNode or even we can support pure string expressions which can be used to > persist. > > Another problem with #toRel is that after the expressions applied as a > projection, user has no chance to apply any other relational nodes if they > want, we can actually support this, the same way as we support the column > expressions. > > [1]https://github.com/apache/calcite/blob/2dc97e6723e1b5bf762540f87b5cd1a848a1/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L5605 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3379) Support expand STRING column expression in table during sql-to-rel conversion
[ https://issues.apache.org/jira/browse/CALCITE-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948269#comment-16948269 ] Danny Chen commented on CALCITE-3379: - > And exception occurs when trying to query the table. [~yanlin-Lynn] What do you mean by an exception throws ? The query in the test case is valid. > Support expand STRING column expression in table during sql-to-rel conversion > - > > Key: CALCITE-3379 > URL: https://issues.apache.org/jira/browse/CALCITE-3379 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.21.0 >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 1.22.0 > > Time Spent: 2h > Remaining Estimate: 0h > > Now there are 2 ways to convert a RelOptTable to LogicalTableScan: > 1. One way is to open the Config#isConvertTableAccess[1] flag and the > SqlToRelConverter would invoke the #toRel method which transforms the table > to a node returned by the user(Usually a table scan). > 2. Another way is to use the LogicalTableScan rule, this rule would invoke > RelOptTable#toRel and wrap the returned node with a LogicalTableScan. > > The difference between 1 and 2 is that, 2 happens in the planning rule but 1 > happens in sql-to-rel conversion, 1 also supports to expand the table columns > based on the defined default values expressions, see > InitializerExpressionFactory#newColumnDefaultValue. > > The problem with the InitializerExpressionFactory#newColumnDefaultValue is > that it uses InitializerContext#convertExpression to convert a SqlNode, but > if the SqlNode is not validated, we always got a RexCall with > SqlUnresolvedFunction. We should give the user chance to validate their > SqlNode or even we can support pure string expressions which can be used to > persist. > > Another problem with #toRel is that after the expressions applied as a > projection, user has no chance to apply any other relational nodes if they > want, we can actually support this, the same way as we support the column > expressions. > > [1]https://github.com/apache/calcite/blob/2dc97e6723e1b5bf762540f87b5cd1a848a1/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L5605 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-3379) Support expand STRING column expression in table during sql-to-rel conversion
[ https://issues.apache.org/jira/browse/CALCITE-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948269#comment-16948269 ] Danny Chen edited comment on CALCITE-3379 at 10/10/19 7:19 AM: --- > And exception occurs when trying to query the table. [~yanlin-Lynn] What do you mean by an exception throws ? The query in the test case is valid. And why do you think the usage of trims is invalid. was (Author: danny0405): > And exception occurs when trying to query the table. [~yanlin-Lynn] What do you mean by an exception throws ? The query in the test case is valid. > Support expand STRING column expression in table during sql-to-rel conversion > - > > Key: CALCITE-3379 > URL: https://issues.apache.org/jira/browse/CALCITE-3379 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.21.0 >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 1.22.0 > > Time Spent: 2h > Remaining Estimate: 0h > > Now there are 2 ways to convert a RelOptTable to LogicalTableScan: > 1. One way is to open the Config#isConvertTableAccess[1] flag and the > SqlToRelConverter would invoke the #toRel method which transforms the table > to a node returned by the user(Usually a table scan). > 2. Another way is to use the LogicalTableScan rule, this rule would invoke > RelOptTable#toRel and wrap the returned node with a LogicalTableScan. > > The difference between 1 and 2 is that, 2 happens in the planning rule but 1 > happens in sql-to-rel conversion, 1 also supports to expand the table columns > based on the defined default values expressions, see > InitializerExpressionFactory#newColumnDefaultValue. > > The problem with the InitializerExpressionFactory#newColumnDefaultValue is > that it uses InitializerContext#convertExpression to convert a SqlNode, but > if the SqlNode is not validated, we always got a RexCall with > SqlUnresolvedFunction. We should give the user chance to validate their > SqlNode or even we can support pure string expressions which can be used to > persist. > > Another problem with #toRel is that after the expressions applied as a > projection, user has no chance to apply any other relational nodes if they > want, we can actually support this, the same way as we support the column > expressions. > > [1]https://github.com/apache/calcite/blob/2dc97e6723e1b5bf762540f87b5cd1a848a1/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L5605 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3379) Support expand STRING column expression in table during sql-to-rel conversion
[ https://issues.apache.org/jira/browse/CALCITE-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948295#comment-16948295 ] Danny Chen commented on CALCITE-3379: - Okey, got your idea, personally, i'm inclined to validate the nodes when the query really happens, just like many other sql engines do to VIEWs [1]. The computed column actually is a VIEW. [1] https://docs.microsoft.com/en-us/sql/t-sql/statements/create-view-transact-sql?view=sql-server-2017#remarks > Support expand STRING column expression in table during sql-to-rel conversion > - > > Key: CALCITE-3379 > URL: https://issues.apache.org/jira/browse/CALCITE-3379 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.21.0 >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 1.22.0 > > Time Spent: 2h > Remaining Estimate: 0h > > Now there are 2 ways to convert a RelOptTable to LogicalTableScan: > 1. One way is to open the Config#isConvertTableAccess[1] flag and the > SqlToRelConverter would invoke the #toRel method which transforms the table > to a node returned by the user(Usually a table scan). > 2. Another way is to use the LogicalTableScan rule, this rule would invoke > RelOptTable#toRel and wrap the returned node with a LogicalTableScan. > > The difference between 1 and 2 is that, 2 happens in the planning rule but 1 > happens in sql-to-rel conversion, 1 also supports to expand the table columns > based on the defined default values expressions, see > InitializerExpressionFactory#newColumnDefaultValue. > > The problem with the InitializerExpressionFactory#newColumnDefaultValue is > that it uses InitializerContext#convertExpression to convert a SqlNode, but > if the SqlNode is not validated, we always got a RexCall with > SqlUnresolvedFunction. We should give the user chance to validate their > SqlNode or even we can support pure string expressions which can be used to > persist. > > Another problem with #toRel is that after the expressions applied as a > projection, user has no chance to apply any other relational nodes if they > want, we can actually support this, the same way as we support the column > expressions. > > [1]https://github.com/apache/calcite/blob/2dc97e6723e1b5bf762540f87b5cd1a848a1/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L5605 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3379) Support expand STRING column expression in table during sql-to-rel conversion
[ https://issues.apache.org/jira/browse/CALCITE-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948301#comment-16948301 ] Danny Chen commented on CALCITE-3379: - Finally i decide not to support a pure string column expression, if user want to persist their expressions, they can firstly transform them to sql strings like "a + 1" or "my_udf(a)", then use the SqlParser to parse it back to SqlNode and transform it to RexNode with the InitializerContext. That means, use should control the expressions se-de by themselves. Personally i think the SqlToRelConverter should not see the parser thing anymore. So i would just close this issue. > Support expand STRING column expression in table during sql-to-rel conversion > - > > Key: CALCITE-3379 > URL: https://issues.apache.org/jira/browse/CALCITE-3379 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.21.0 >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 1.22.0 > > Time Spent: 2h > Remaining Estimate: 0h > > Now there are 2 ways to convert a RelOptTable to LogicalTableScan: > 1. One way is to open the Config#isConvertTableAccess[1] flag and the > SqlToRelConverter would invoke the #toRel method which transforms the table > to a node returned by the user(Usually a table scan). > 2. Another way is to use the LogicalTableScan rule, this rule would invoke > RelOptTable#toRel and wrap the returned node with a LogicalTableScan. > > The difference between 1 and 2 is that, 2 happens in the planning rule but 1 > happens in sql-to-rel conversion, 1 also supports to expand the table columns > based on the defined default values expressions, see > InitializerExpressionFactory#newColumnDefaultValue. > > The problem with the InitializerExpressionFactory#newColumnDefaultValue is > that it uses InitializerContext#convertExpression to convert a SqlNode, but > if the SqlNode is not validated, we always got a RexCall with > SqlUnresolvedFunction. We should give the user chance to validate their > SqlNode or even we can support pure string expressions which can be used to > persist. > > Another problem with #toRel is that after the expressions applied as a > projection, user has no chance to apply any other relational nodes if they > want, we can actually support this, the same way as we support the column > expressions. > > [1]https://github.com/apache/calcite/blob/2dc97e6723e1b5bf762540f87b5cd1a848a1/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L5605 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-3379) Support expand STRING column expression in table during sql-to-rel conversion
[ https://issues.apache.org/jira/browse/CALCITE-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948295#comment-16948295 ] Danny Chen edited comment on CALCITE-3379 at 10/11/19 1:43 AM: --- Okey, got your idea, personally, i'm inclined to validate the nodes when the query really happens, just like many other sql engines do to VIEWs [1]. The computed column actually is a VIEW. I did check the MS-SQL 2017 and it validates for create table, so maybe we also need a validation phrase for DDL nodes, we may need 2 kinds of validation: # Syntax check for the DDLs # Expression validation for the table columns [1] https://docs.microsoft.com/en-us/sql/t-sql/statements/create-view-transact-sql?view=sql-server-2017#remarks was (Author: danny0405): Okey, got your idea, personally, i'm inclined to validate the nodes when the query really happens, just like many other sql engines do to VIEWs [1]. The computed column actually is a VIEW. [1] https://docs.microsoft.com/en-us/sql/t-sql/statements/create-view-transact-sql?view=sql-server-2017#remarks > Support expand STRING column expression in table during sql-to-rel conversion > - > > Key: CALCITE-3379 > URL: https://issues.apache.org/jira/browse/CALCITE-3379 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.21.0 >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 1.22.0 > > Time Spent: 2h > Remaining Estimate: 0h > > Now there are 2 ways to convert a RelOptTable to LogicalTableScan: > 1. One way is to open the Config#isConvertTableAccess[1] flag and the > SqlToRelConverter would invoke the #toRel method which transforms the table > to a node returned by the user(Usually a table scan). > 2. Another way is to use the LogicalTableScan rule, this rule would invoke > RelOptTable#toRel and wrap the returned node with a LogicalTableScan. > > The difference between 1 and 2 is that, 2 happens in the planning rule but 1 > happens in sql-to-rel conversion, 1 also supports to expand the table columns > based on the defined default values expressions, see > InitializerExpressionFactory#newColumnDefaultValue. > > The problem with the InitializerExpressionFactory#newColumnDefaultValue is > that it uses InitializerContext#convertExpression to convert a SqlNode, but > if the SqlNode is not validated, we always got a RexCall with > SqlUnresolvedFunction. We should give the user chance to validate their > SqlNode or even we can support pure string expressions which can be used to > persist. > > Another problem with #toRel is that after the expressions applied as a > projection, user has no chance to apply any other relational nodes if they > want, we can actually support this, the same way as we support the column > expressions. > > [1]https://github.com/apache/calcite/blob/2dc97e6723e1b5bf762540f87b5cd1a848a1/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L5605 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-963) Hoist literals
[
https://issues.apache.org/jira/browse/CALCITE-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16949073#comment-16949073
]
Danny Chen commented on CALCITE-963:
[~ScottReynolds] Sorry to bother, can you explain more about
EnumerableRelImplementaor#stash ? I only see one direct invoke[1]
> This method takes an object stores it in DataContext and returns an
> Expression that calls DataContext#get to pull out the object.
I didn't see a logic like what you described. It seems that
EnumerableRelImplementaor#stash did some expression cache, but for constant, it
always returns a constant expression directly.
> Hoist literals
> --
>
> Key: CALCITE-963
> URL: https://issues.apache.org/jira/browse/CALCITE-963
> Project: Calcite
> Issue Type: Bug
>Reporter: Julian Hyde
>Priority: Major
> Labels: pull-request-available
> Attachments: HoistedVariables.png
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Convert literals into (internal) bind variables so that statements that
> differ only in literal values can be executed using the same plan.
> In [mail
> thread|http://mail-archives.apache.org/mod_mbox/calcite-dev/201511.mbox/%3c56437bf8.70...@gmail.com%3E]
> Homer wrote:
> {quote}Imagine that it is common to run a large number of very similar
> machine generated queries that just change the literals in the sql query.
> For example (the real queries would be much more complex):
> {code}Select * from emp where empno = 1;
> Select * from emp where empno = 2;
> etc.{code}
> The plan that is likely being generated for these kind of queries is going to
> be very much the same each time, so to save some time, I would like to
> recognize that the literals are all that have changed in a query and use the
> previously optimized execution plan and just replace the literals.{quote}
> I think this could be done as a transform on the initial RelNode tree. It
> would find literals (RexLiteral), replace them with bind variables
> (RexDynamicParam) and write the value into a pool. The next statement would
> go through the same process and the RelNode tree would be identical, but with
> possibly different values for the bind variables.
> The bind variables are of course internal; not visible from JDBC. When the
> statement is executed, the bind variables are implicitly bound.
> Statements would be held in a Guava cache.
> This would be enabled by a config parameter. Unfortunately I don't think we
> could do this by default -- we'd lose optimization power because we would no
> longer be able to do constant reduction.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-3397) AssertionError for interpreter multiset
[
https://issues.apache.org/jira/browse/CALCITE-3397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen updated CALCITE-3397:
Component/s: core
> AssertionError for interpreter multiset
> ---
>
> Key: CALCITE-3397
> URL: https://issues.apache.org/jira/browse/CALCITE-3397
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Wang Yanlin
>Assignee: Danny Chen
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.22.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> when interpretering sql
> *select multiset['a', 'b', 'c']*
> got,
> {code:java}
> java.lang.AssertionError: interpreter: no implementation for class
> org.apache.calcite.rel.core.Collect
> at
> org.apache.calcite.interpreter.Interpreter$CompilerImpl.visit(Interpreter.java:460)
> at
> org.apache.calcite.interpreter.Nodes$CoreCompiler.visit(Nodes.java:43)
> at org.apache.calcite.rel.BiRel.childrenAccept(BiRel.java:46)
> at
> org.apache.calcite.interpreter.Interpreter$CompilerImpl.visit(Interpreter.java:447)
> at
> org.apache.calcite.interpreter.Nodes$CoreCompiler.visit(Nodes.java:43)
> at org.apache.calcite.rel.SingleRel.childrenAccept(SingleRel.java:72)
> at
> org.apache.calcite.interpreter.Interpreter$CompilerImpl.visit(Interpreter.java:447)
> at
> org.apache.calcite.interpreter.Nodes$CoreCompiler.visit(Nodes.java:43)
> at
> org.apache.calcite.interpreter.Interpreter$CompilerImpl.visitRoot(Interpreter.java:405)
> at
> org.apache.calcite.interpreter.Interpreter.(Interpreter.java:88)
> at
> org.apache.calcite.test.InterpreterTest.testInterpretMultiset(InterpreterTest.java:127)
> {code}
> Reproduce this with test case in InterpreterTest
> {code:java}
> @Test public void testInterpretMultiset() throws Exception {
> final String sql = "select multiset['a', 'b', 'c']";
> SqlNode parse = planner.parse(sql);
> SqlNode validate = planner.validate(parse);
> RelNode convert = planner.rel(validate).project();
> final Interpreter interpreter = new Interpreter(dataContext, convert);
> assertRows(interpreter, "[[a, b, c]]");
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-3397) AssertionError for interpreter multiset
[
https://issues.apache.org/jira/browse/CALCITE-3397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen resolved CALCITE-3397.
-
Fix Version/s: 1.22.0
Assignee: Danny Chen
Resolution: Fixed
Fixed in
[f7ff1c9|https://github.com/apache/calcite/commit/f7ff1c906212b34a41599c5009527639ad2e8718],
thanks for your PR, [~yanlin-Lynn] !
> AssertionError for interpreter multiset
> ---
>
> Key: CALCITE-3397
> URL: https://issues.apache.org/jira/browse/CALCITE-3397
> Project: Calcite
> Issue Type: Bug
>Reporter: Wang Yanlin
>Assignee: Danny Chen
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.22.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> when interpretering sql
> *select multiset['a', 'b', 'c']*
> got,
> {code:java}
> java.lang.AssertionError: interpreter: no implementation for class
> org.apache.calcite.rel.core.Collect
> at
> org.apache.calcite.interpreter.Interpreter$CompilerImpl.visit(Interpreter.java:460)
> at
> org.apache.calcite.interpreter.Nodes$CoreCompiler.visit(Nodes.java:43)
> at org.apache.calcite.rel.BiRel.childrenAccept(BiRel.java:46)
> at
> org.apache.calcite.interpreter.Interpreter$CompilerImpl.visit(Interpreter.java:447)
> at
> org.apache.calcite.interpreter.Nodes$CoreCompiler.visit(Nodes.java:43)
> at org.apache.calcite.rel.SingleRel.childrenAccept(SingleRel.java:72)
> at
> org.apache.calcite.interpreter.Interpreter$CompilerImpl.visit(Interpreter.java:447)
> at
> org.apache.calcite.interpreter.Nodes$CoreCompiler.visit(Nodes.java:43)
> at
> org.apache.calcite.interpreter.Interpreter$CompilerImpl.visitRoot(Interpreter.java:405)
> at
> org.apache.calcite.interpreter.Interpreter.(Interpreter.java:88)
> at
> org.apache.calcite.test.InterpreterTest.testInterpretMultiset(InterpreterTest.java:127)
> {code}
> Reproduce this with test case in InterpreterTest
> {code:java}
> @Test public void testInterpretMultiset() throws Exception {
> final String sql = "select multiset['a', 'b', 'c']";
> SqlNode parse = planner.parse(sql);
> SqlNode validate = planner.validate(parse);
> RelNode convert = planner.rel(validate).project();
> final Interpreter interpreter = new Interpreter(dataContext, convert);
> assertRows(interpreter, "[[a, b, c]]");
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3368) Some problems simplifying ‘expression IS NULL’
[
https://issues.apache.org/jira/browse/CALCITE-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16949168#comment-16949168
]
Danny Chen commented on CALCITE-3368:
-
Personally, i'm + 1 to add a switch for the PLUS/MINUS/TIMES is null
simplification, because these operators are so common, we may lost many valid
simplification just because of the overflow/unsafe corner cases.
[~kgyrtkirk] What do you think about this idea ?
> Some problems simplifying ‘expression IS NULL’
> --
>
> Key: CALCITE-3368
> URL: https://issues.apache.org/jira/browse/CALCITE-3368
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.21.0
>Reporter: Leonard Xu
>Assignee: Leonard Xu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> 'is null' expression in SQL may be optimized incorrectly in the underlying
> implementation.
>
> When I write a Fink SQL to test overflow just like
> {code:java}
> select
>case when (f0 + f1) is null then 'null' else 'not null' end
> from testTable
> {code}
> , I found expression '(f0 + f1) is null ' has been optimized by Calcite, and
> the optimization may be incorrect.
>
> The underlying implementation is that Calcite's simplification logic of
> isNull expression in SQL will convert from
> *"f(operand0, operand1) IS NULL"* to
> *"operand0 IS NULL OR operand1 IS NULL"* if the Policy of RexNode‘s SqlKind
> is ANY。
> This simplification leads to the expression will not calculate the real
> value of *f(operand0, operand1)* (eg.. '(f0 + f1)' in my case ），but '(f0 +
> f1)' maybe overflows after operation.
> {code:java}
> //org.apache.calcite.rex.RexSimplify.java
> private RexNode simplifyIsNull(RexNode a) {
> // Simplify the argument first,
> // call ourselves recursively to see whether we can make more progress.
> // For example, given
> // "(CASE WHEN FALSE THEN 1 ELSE 2) IS NULL" we first simplify the
> // argument to "2", and only then we can simplify "2 IS NULL" to "FALSE".
> a = simplify(a, UNKNOWN);
> if (!a.getType().isNullable() && isSafeExpression(a)) {
> return rexBuilder.makeLiteral(false);
> }
> if (RexUtil.isNull(a)) {
> return rexBuilder.makeLiteral(true);
> }
> if (a.getKind() == SqlKind.CAST) {
> return null;
> }
> switch (Strong.policy(a.getKind())) {
> case NOT_NULL:
> return rexBuilder.makeLiteral(false);
> case ANY:
> // "f" is a strong operator, so "f(operand0, operand1) IS NULL" simplifies
> // to "operand0 IS NULL OR operand1 IS NULL"
> final List operands = new ArrayList<>();
> for (RexNode operand : ((RexCall) a).getOperands()) {
> final RexNode simplified = simplifyIsNull(operand);
> if (simplified == null) {
> operands.add(
> rexBuilder.makeCall(SqlStdOperatorTable.IS_NULL, operand));
> } else {
> operands.add(simplified);
> }
> }
> return RexUtil.composeDisjunction(rexBuilder, operands, false);
> case AS_IS:
> default:
> return null;
> }
> }{code}
> And most of calculating SqlKinds are assigned *Policy.ANY* at present.
> {code:java}
> //org.apache.calcite.plan.Strong.java
> public static Policy policy(SqlKind kind) {
> return MAP.getOrDefault(kind, Policy.AS_IS);
> }
>
> map.put(SqlKind.PLUS, Policy.ANY);
> map.put(SqlKind.PLUS_PREFIX, Policy.ANY);
> map.put(SqlKind.MINUS, Policy.ANY);
> map.put(SqlKind.MINUS_PREFIX, Policy.ANY);
> map.put(SqlKind.TIMES, Policy.ANY);
> map.put(SqlKind.DIVIDE, Policy.ANY);
> * that operator evaluates to null. */
> public enum Policy {
> /** This kind of expression is never null. No need to look at its arguments,
>* if it has any. */
> NOT_NULL,
> /** This kind of expression has its own particular rules about whether it
>* is null. */
> CUSTOM,
> /** This kind of expression is null if and only if at least one of its
>* arguments is null. */
> ANY,
> /** This kind of expression may be null. There is no way to rewrite. */
> AS_IS,
> }{code}
>
> It may be an obvious nonequivalent simplification in SQL. And this issue come
> from Flink （FLINK-14030）.
> [~danny0405]， Could you have a look at this?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-3379) Support expand STRING column expression of table during sql-to-rel conversion
[ https://issues.apache.org/jira/browse/CALCITE-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated CALCITE-3379: Summary: Support expand STRING column expression of table during sql-to-rel conversion (was: Support expand STRING column expression in table during sql-to-rel conversion) > Support expand STRING column expression of table during sql-to-rel conversion > - > > Key: CALCITE-3379 > URL: https://issues.apache.org/jira/browse/CALCITE-3379 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.21.0 >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 1.22.0 > > Time Spent: 2h > Remaining Estimate: 0h > > Now there are 2 ways to convert a RelOptTable to LogicalTableScan: > 1. One way is to open the Config#isConvertTableAccess[1] flag and the > SqlToRelConverter would invoke the #toRel method which transforms the table > to a node returned by the user(Usually a table scan). > 2. Another way is to use the LogicalTableScan rule, this rule would invoke > RelOptTable#toRel and wrap the returned node with a LogicalTableScan. > > The difference between 1 and 2 is that, 2 happens in the planning rule but 1 > happens in sql-to-rel conversion, 1 also supports to expand the table columns > based on the defined default values expressions, see > InitializerExpressionFactory#newColumnDefaultValue. > > The problem with the InitializerExpressionFactory#newColumnDefaultValue is > that it uses InitializerContext#convertExpression to convert a SqlNode, but > if the SqlNode is not validated, we always got a RexCall with > SqlUnresolvedFunction. We should give the user chance to validate their > SqlNode or even we can support pure string expressions which can be used to > persist. > > Another problem with #toRel is that after the expressions applied as a > projection, user has no chance to apply any other relational nodes if they > want, we can actually support this, the same way as we support the column > expressions. > > [1]https://github.com/apache/calcite/blob/2dc97e6723e1b5bf762540f87b5cd1a848a1/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L5605 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-3368) PLUS, MUNUS and TIMES should be unsafe when simplifying ‘expression IS NULL’
[
https://issues.apache.org/jira/browse/CALCITE-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen updated CALCITE-3368:
Summary: PLUS, MUNUS and TIMES should be unsafe when simplifying
‘expression IS NULL’ (was: Some problems simplifying ‘expression IS NULL’)
> PLUS, MUNUS and TIMES should be unsafe when simplifying ‘expression IS NULL’
>
>
> Key: CALCITE-3368
> URL: https://issues.apache.org/jira/browse/CALCITE-3368
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.21.0
>Reporter: Leonard Xu
>Assignee: Leonard Xu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> 'is null' expression in SQL may be optimized incorrectly in the underlying
> implementation.
>
> When I write a Fink SQL to test overflow just like
> {code:java}
> select
>case when (f0 + f1) is null then 'null' else 'not null' end
> from testTable
> {code}
> , I found expression '(f0 + f1) is null ' has been optimized by Calcite, and
> the optimization may be incorrect.
>
> The underlying implementation is that Calcite's simplification logic of
> isNull expression in SQL will convert from
> *"f(operand0, operand1) IS NULL"* to
> *"operand0 IS NULL OR operand1 IS NULL"* if the Policy of RexNode‘s SqlKind
> is ANY。
> This simplification leads to the expression will not calculate the real
> value of *f(operand0, operand1)* (eg.. '(f0 + f1)' in my case ），but '(f0 +
> f1)' maybe overflows after operation.
> {code:java}
> //org.apache.calcite.rex.RexSimplify.java
> private RexNode simplifyIsNull(RexNode a) {
> // Simplify the argument first,
> // call ourselves recursively to see whether we can make more progress.
> // For example, given
> // "(CASE WHEN FALSE THEN 1 ELSE 2) IS NULL" we first simplify the
> // argument to "2", and only then we can simplify "2 IS NULL" to "FALSE".
> a = simplify(a, UNKNOWN);
> if (!a.getType().isNullable() && isSafeExpression(a)) {
> return rexBuilder.makeLiteral(false);
> }
> if (RexUtil.isNull(a)) {
> return rexBuilder.makeLiteral(true);
> }
> if (a.getKind() == SqlKind.CAST) {
> return null;
> }
> switch (Strong.policy(a.getKind())) {
> case NOT_NULL:
> return rexBuilder.makeLiteral(false);
> case ANY:
> // "f" is a strong operator, so "f(operand0, operand1) IS NULL" simplifies
> // to "operand0 IS NULL OR operand1 IS NULL"
> final List operands = new ArrayList<>();
> for (RexNode operand : ((RexCall) a).getOperands()) {
> final RexNode simplified = simplifyIsNull(operand);
> if (simplified == null) {
> operands.add(
> rexBuilder.makeCall(SqlStdOperatorTable.IS_NULL, operand));
> } else {
> operands.add(simplified);
> }
> }
> return RexUtil.composeDisjunction(rexBuilder, operands, false);
> case AS_IS:
> default:
> return null;
> }
> }{code}
> And most of calculating SqlKinds are assigned *Policy.ANY* at present.
> {code:java}
> //org.apache.calcite.plan.Strong.java
> public static Policy policy(SqlKind kind) {
> return MAP.getOrDefault(kind, Policy.AS_IS);
> }
>
> map.put(SqlKind.PLUS, Policy.ANY);
> map.put(SqlKind.PLUS_PREFIX, Policy.ANY);
> map.put(SqlKind.MINUS, Policy.ANY);
> map.put(SqlKind.MINUS_PREFIX, Policy.ANY);
> map.put(SqlKind.TIMES, Policy.ANY);
> map.put(SqlKind.DIVIDE, Policy.ANY);
> * that operator evaluates to null. */
> public enum Policy {
> /** This kind of expression is never null. No need to look at its arguments,
>* if it has any. */
> NOT_NULL,
> /** This kind of expression has its own particular rules about whether it
>* is null. */
> CUSTOM,
> /** This kind of expression is null if and only if at least one of its
>* arguments is null. */
> ANY,
> /** This kind of expression may be null. There is no way to rewrite. */
> AS_IS,
> }{code}
>
> It may be an obvious nonequivalent simplification in SQL. And this issue come
> from Flink （FLINK-14030）.
> [~danny0405]， Could you have a look at this?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-3379) Support expand STRING column expression of table during sql-to-rel conversion
[ https://issues.apache.org/jira/browse/CALCITE-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen resolved CALCITE-3379. - Resolution: Fixed Finally i changed my idea and provide a default impl to support string column expression, Fixed in [22c76fb|https://github.com/apache/calcite/commit/22c76fb8e0a3ec363332caf7eb8e46071ace798e] ! > Support expand STRING column expression of table during sql-to-rel conversion > - > > Key: CALCITE-3379 > URL: https://issues.apache.org/jira/browse/CALCITE-3379 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.21.0 >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 1.22.0 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > Now there are 2 ways to convert a RelOptTable to LogicalTableScan: > 1. One way is to open the Config#isConvertTableAccess[1] flag and the > SqlToRelConverter would invoke the #toRel method which transforms the table > to a node returned by the user(Usually a table scan). > 2. Another way is to use the LogicalTableScan rule, this rule would invoke > RelOptTable#toRel and wrap the returned node with a LogicalTableScan. > > The difference between 1 and 2 is that, 2 happens in the planning rule but 1 > happens in sql-to-rel conversion, 1 also supports to expand the table columns > based on the defined default values expressions, see > InitializerExpressionFactory#newColumnDefaultValue. > > The problem with the InitializerExpressionFactory#newColumnDefaultValue is > that it uses InitializerContext#convertExpression to convert a SqlNode, but > if the SqlNode is not validated, we always got a RexCall with > SqlUnresolvedFunction. We should give the user chance to validate their > SqlNode or even we can support pure string expressions which can be used to > persist. > > Another problem with #toRel is that after the expressions applied as a > projection, user has no chance to apply any other relational nodes if they > want, we can actually support this, the same way as we support the column > expressions. > > [1]https://github.com/apache/calcite/blob/2dc97e6723e1b5bf762540f87b5cd1a848a1/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L5605 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CALCITE-3217) Support "SELECT NULL"
[
https://issues.apache.org/jira/browse/CALCITE-3217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen resolved CALCITE-3217.
-
Fix Version/s: 1.21.0
Assignee: Danny Chen
Resolution: Duplicate
Mark as duplicated because CALCITE-2302 already covered this issue.
> Support "SELECT NULL"
> -
>
> Key: CALCITE-3217
> URL: https://issues.apache.org/jira/browse/CALCITE-3217
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Xurenhe
>Assignee: Danny Chen
>Priority: Minor
> Fix For: 1.21.0
>
> Attachments: Jietu20190728-221651.jpg
>
>
>
> Should Calcite need to support "select null", such as:
> {code:java}
> select null;{code}
> In the version of "1.20.0", it throws exception in
> `org.apache.calcite.sql.validate.SqlValidatorImpl#inferUnknownTypes`.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3404) Promote plan for AggregateExpandDistinctAggregatesRule when all the agg expressions are distinct and have same arguments
Danny Chen created CALCITE-3404:
---
Summary: Promote plan for AggregateExpandDistinctAggregatesRule
when all the agg expressions are distinct and have same arguments
Key: CALCITE-3404
URL: https://issues.apache.org/jira/browse/CALCITE-3404
Project: Calcite
Issue Type: Improvement
Components: core
Affects Versions: 1.21.0
Reporter: Danny Chen
Assignee: Danny Chen
Fix For: 1.22.0
After CALCITE-3159, the distinct constraint for MAX/MIN/BIT_OR/BIT_AND are
removed automatically, so if we have a query:
{code:sql}
select sum(distinct deptno), count(distinct deptno), max(deptno) from emp
{code}
Than plan has regression from
{code:xml}
LogicalAggregate(group=[{}], EXPR$0=[SUM($0)], EXPR$1=[COUNT($0)],
EXPR$2=[MAX($0)])
LogicalAggregate(group=[{0}])
LogicalProject(DEPTNO=[$7])
LogicalTableScan(table=[[CATALOG, SALES, EMP]])
{code}
to
{code:xml}
LogicalAggregate(group=[{}], EXPR$0=[SUM($0) FILTER $2], EXPR$1=[COUNT($0)
FILTER $2], EXPR$2=[MIN($1) FILTER $3])
LogicalProject(DEPTNO=[$0], EXPR$2=[$1], $g_0=[=($2, 0)], $g_1=[=($2, 1)])
LogicalAggregate(group=[{0}], groups=[[{0}, {}]], EXPR$2=[MAX($0)],
$g=[GROUPING($0)])
LogicalProject(DEPTNO=[$7])
LogicalTableScan(table=[[CATALOG, SALES, EMP]])
{code}
The distinct trait actually can be reused.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3404) Promote plan for AggregateExpandDistinctAggregatesRule when all the agg expressions are distinct and have same arguments
[
https://issues.apache.org/jira/browse/CALCITE-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16949924#comment-16949924
]
Danny Chen commented on CALCITE-3404:
-
Thanks for the feedback Julian, working now ~
> Promote plan for AggregateExpandDistinctAggregatesRule when all the agg
> expressions are distinct and have same arguments
>
>
> Key: CALCITE-3404
> URL: https://issues.apache.org/jira/browse/CALCITE-3404
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.21.0
>Reporter: Danny Chen
>Assignee: Danny Chen
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.22.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> After CALCITE-3159, the distinct constraint for MAX/MIN/BIT_OR/BIT_AND are
> removed automatically, so if we have a query:
> {code:sql}
> select sum(distinct deptno), count(distinct deptno), max(deptno) from emp
> {code}
> Than plan has regression from
> {code:xml}
> LogicalAggregate(group=[{}], EXPR$0=[SUM($0)], EXPR$1=[COUNT($0)],
> EXPR$2=[MAX($0)])
> LogicalAggregate(group=[{0}])
> LogicalProject(DEPTNO=[$7])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {code}
> to
> {code:xml}
> LogicalAggregate(group=[{}], EXPR$0=[SUM($0) FILTER $2], EXPR$1=[COUNT($0)
> FILTER $2], EXPR$2=[MIN($1) FILTER $3])
> LogicalProject(DEPTNO=[$0], EXPR$2=[$1], $g_0=[=($2, 0)], $g_1=[=($2, 1)])
> LogicalAggregate(group=[{0}], groups=[[{0}, {}]], EXPR$2=[MAX($0)],
> $g=[GROUPING($0)])
> LogicalProject(DEPTNO=[$7])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {code}
> The distinct trait actually can be reused.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-3404) Treat agg expressions that can ignore distinct constraint as distinct in AggregateExpandDistinctAggregatesRule when all the other agg expressions are distinct and have
[
https://issues.apache.org/jira/browse/CALCITE-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen updated CALCITE-3404:
Summary: Treat agg expressions that can ignore distinct constraint as
distinct in AggregateExpandDistinctAggregatesRule when all the other agg
expressions are distinct and have same arguments (was: Promote plan for
AggregateExpandDistinctAggregatesRule when all the agg expressions are distinct
and have same arguments)
> Treat agg expressions that can ignore distinct constraint as distinct in
> AggregateExpandDistinctAggregatesRule when all the other agg expressions are
> distinct and have same arguments
> --
>
> Key: CALCITE-3404
> URL: https://issues.apache.org/jira/browse/CALCITE-3404
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.21.0
>Reporter: Danny Chen
>Assignee: Danny Chen
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.22.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> After CALCITE-3159, the distinct constraint for MAX/MIN/BIT_OR/BIT_AND are
> removed automatically, so if we have a query:
> {code:sql}
> select sum(distinct deptno), count(distinct deptno), max(deptno) from emp
> {code}
> Than plan has regression from
> {code:xml}
> LogicalAggregate(group=[{}], EXPR$0=[SUM($0)], EXPR$1=[COUNT($0)],
> EXPR$2=[MAX($0)])
> LogicalAggregate(group=[{0}])
> LogicalProject(DEPTNO=[$7])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {code}
> to
> {code:xml}
> LogicalAggregate(group=[{}], EXPR$0=[SUM($0) FILTER $2], EXPR$1=[COUNT($0)
> FILTER $2], EXPR$2=[MIN($1) FILTER $3])
> LogicalProject(DEPTNO=[$0], EXPR$2=[$1], $g_0=[=($2, 0)], $g_1=[=($2, 1)])
> LogicalAggregate(group=[{0}], groups=[[{0}, {}]], EXPR$2=[MAX($0)],
> $g=[GROUPING($0)])
> LogicalProject(DEPTNO=[$7])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {code}
> The distinct trait actually can be reused.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-3404) Treat agg expressions that can ignore distinct constraint as distinct in AggregateExpandDistinctAggregatesRule when all the other agg expressions are distinct and have
[
https://issues.apache.org/jira/browse/CALCITE-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen updated CALCITE-3404:
Description:
After CALCITE-3159, the distinct constraint for MAX/MIN/BIT_OR/BIT_AND are
removed automatically, so if we have a query:
{code:sql}
select sum(distinct deptno), count(distinct deptno), max(deptno) from emp
{code}
The plan has regression from
{code:xml}
LogicalAggregate(group=[{}], EXPR$0=[SUM($0)], EXPR$1=[COUNT($0)],
EXPR$2=[MAX($0)])
LogicalAggregate(group=[{0}])
LogicalProject(DEPTNO=[$7])
LogicalTableScan(table=[[CATALOG, SALES, EMP]])
{code}
to
{code:xml}
LogicalAggregate(group=[{}], EXPR$0=[SUM($0) FILTER $2], EXPR$1=[COUNT($0)
FILTER $2], EXPR$2=[MIN($1) FILTER $3])
LogicalProject(DEPTNO=[$0], EXPR$2=[$1], $g_0=[=($2, 0)], $g_1=[=($2, 1)])
LogicalAggregate(group=[{0}], groups=[[{0}, {}]], EXPR$2=[MAX($0)],
$g=[GROUPING($0)])
LogicalProject(DEPTNO=[$7])
LogicalTableScan(table=[[CATALOG, SALES, EMP]])
{code}
The distinct trait actually can be reused.
was:
After CALCITE-3159, the distinct constraint for MAX/MIN/BIT_OR/BIT_AND are
removed automatically, so if we have a query:
{code:sql}
select sum(distinct deptno), count(distinct deptno), max(deptno) from emp
{code}
Than plan has regression from
{code:xml}
LogicalAggregate(group=[{}], EXPR$0=[SUM($0)], EXPR$1=[COUNT($0)],
EXPR$2=[MAX($0)])
LogicalAggregate(group=[{0}])
LogicalProject(DEPTNO=[$7])
LogicalTableScan(table=[[CATALOG, SALES, EMP]])
{code}
to
{code:xml}
LogicalAggregate(group=[{}], EXPR$0=[SUM($0) FILTER $2], EXPR$1=[COUNT($0)
FILTER $2], EXPR$2=[MIN($1) FILTER $3])
LogicalProject(DEPTNO=[$0], EXPR$2=[$1], $g_0=[=($2, 0)], $g_1=[=($2, 1)])
LogicalAggregate(group=[{0}], groups=[[{0}, {}]], EXPR$2=[MAX($0)],
$g=[GROUPING($0)])
LogicalProject(DEPTNO=[$7])
LogicalTableScan(table=[[CATALOG, SALES, EMP]])
{code}
The distinct trait actually can be reused.
> Treat agg expressions that can ignore distinct constraint as distinct in
> AggregateExpandDistinctAggregatesRule when all the other agg expressions are
> distinct and have same arguments
> --
>
> Key: CALCITE-3404
> URL: https://issues.apache.org/jira/browse/CALCITE-3404
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.21.0
>Reporter: Danny Chen
>Assignee: Danny Chen
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.22.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> After CALCITE-3159, the distinct constraint for MAX/MIN/BIT_OR/BIT_AND are
> removed automatically, so if we have a query:
> {code:sql}
> select sum(distinct deptno), count(distinct deptno), max(deptno) from emp
> {code}
> The plan has regression from
> {code:xml}
> LogicalAggregate(group=[{}], EXPR$0=[SUM($0)], EXPR$1=[COUNT($0)],
> EXPR$2=[MAX($0)])
> LogicalAggregate(group=[{0}])
> LogicalProject(DEPTNO=[$7])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {code}
> to
> {code:xml}
> LogicalAggregate(group=[{}], EXPR$0=[SUM($0) FILTER $2], EXPR$1=[COUNT($0)
> FILTER $2], EXPR$2=[MIN($1) FILTER $3])
> LogicalProject(DEPTNO=[$0], EXPR$2=[$1], $g_0=[=($2, 0)], $g_1=[=($2, 1)])
> LogicalAggregate(group=[{0}], groups=[[{0}, {}]], EXPR$2=[MAX($0)],
> $g=[GROUPING($0)])
> LogicalProject(DEPTNO=[$7])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {code}
> The distinct trait actually can be reused.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3404) Treat agg expressions that can ignore distinct constraint as distinct in AggregateExpandDistinctAggregatesRule when all the other agg expressions are distinct and hav
[
https://issues.apache.org/jira/browse/CALCITE-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16949954#comment-16949954
]
Danny Chen commented on CALCITE-3404:
-
Julian, i have did some refactoring for the code, hopes it worked.
I also noticed that there are so many code to use checkPlanning in
RelOptRulesTest, maybe it needs a refactoring just like CALCITE-3327.
> Treat agg expressions that can ignore distinct constraint as distinct in
> AggregateExpandDistinctAggregatesRule when all the other agg expressions are
> distinct and have same arguments
> --
>
> Key: CALCITE-3404
> URL: https://issues.apache.org/jira/browse/CALCITE-3404
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.21.0
>Reporter: Danny Chen
>Assignee: Danny Chen
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.22.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> After CALCITE-3159, the distinct constraint for MAX/MIN/BIT_OR/BIT_AND are
> removed automatically, so if we have a query:
> {code:sql}
> select sum(distinct deptno), count(distinct deptno), max(deptno) from emp
> {code}
> The plan has regression from
> {code:xml}
> LogicalAggregate(group=[{}], EXPR$0=[SUM($0)], EXPR$1=[COUNT($0)],
> EXPR$2=[MAX($0)])
> LogicalAggregate(group=[{0}])
> LogicalProject(DEPTNO=[$7])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {code}
> to
> {code:xml}
> LogicalAggregate(group=[{}], EXPR$0=[SUM($0) FILTER $2], EXPR$1=[COUNT($0)
> FILTER $2], EXPR$2=[MIN($1) FILTER $3])
> LogicalProject(DEPTNO=[$0], EXPR$2=[$1], $g_0=[=($2, 0)], $g_1=[=($2, 1)])
> LogicalAggregate(group=[{0}], groups=[[{0}, {}]], EXPR$2=[MAX($0)],
> $g=[GROUPING($0)])
> LogicalProject(DEPTNO=[$7])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {code}
> The distinct trait actually can be reused.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-3404) In AggregateExpandDistinctAggregatesRule, treat all the agg expressions as distinct if they have the same arguments and the non-distinct expressions distinct constraint
[
https://issues.apache.org/jira/browse/CALCITE-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen updated CALCITE-3404:
Summary: In AggregateExpandDistinctAggregatesRule, treat all the agg
expressions as distinct if they have the same arguments and the non-distinct
expressions distinct constraints can be ignored (was: Treat agg expressions
that can ignore distinct constraint as distinct in
AggregateExpandDistinctAggregatesRule when all the other agg expressions are
distinct and have same arguments)
> In AggregateExpandDistinctAggregatesRule, treat all the agg expressions as
> distinct if they have the same arguments and the non-distinct expressions
> distinct constraints can be ignored
>
>
> Key: CALCITE-3404
> URL: https://issues.apache.org/jira/browse/CALCITE-3404
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.21.0
>Reporter: Danny Chen
>Assignee: Danny Chen
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.22.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> After CALCITE-3159, the distinct constraint for MAX/MIN/BIT_OR/BIT_AND are
> removed automatically, so if we have a query:
> {code:sql}
> select sum(distinct deptno), count(distinct deptno), max(deptno) from emp
> {code}
> The plan has regression from
> {code:xml}
> LogicalAggregate(group=[{}], EXPR$0=[SUM($0)], EXPR$1=[COUNT($0)],
> EXPR$2=[MAX($0)])
> LogicalAggregate(group=[{0}])
> LogicalProject(DEPTNO=[$7])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {code}
> to
> {code:xml}
> LogicalAggregate(group=[{}], EXPR$0=[SUM($0) FILTER $2], EXPR$1=[COUNT($0)
> FILTER $2], EXPR$2=[MIN($1) FILTER $3])
> LogicalProject(DEPTNO=[$0], EXPR$2=[$1], $g_0=[=($2, 0)], $g_1=[=($2, 1)])
> LogicalAggregate(group=[{0}], groups=[[{0}, {}]], EXPR$2=[MAX($0)],
> $g=[GROUPING($0)])
> LogicalProject(DEPTNO=[$7])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {code}
> The distinct trait actually can be reused.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-3404) In AggregateExpandDistinctAggregatesRule, treat all the agg expressions as distinct if they have the same arguments and the non-distinct expressions distinct constrain
[
https://issues.apache.org/jira/browse/CALCITE-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen resolved CALCITE-3404.
-
Resolution: Fixed
Fixed in
[8f5f251|https://github.com/apache/calcite/commit/8f5f251123ee9cc036cada902c4795db3869ffa0]
!
> In AggregateExpandDistinctAggregatesRule, treat all the agg expressions as
> distinct if they have the same arguments and the non-distinct expressions
> distinct constraints can be ignored
>
>
> Key: CALCITE-3404
> URL: https://issues.apache.org/jira/browse/CALCITE-3404
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.21.0
>Reporter: Danny Chen
>Assignee: Danny Chen
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.22.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> After CALCITE-3159, the distinct constraint for MAX/MIN/BIT_OR/BIT_AND are
> removed automatically, so if we have a query:
> {code:sql}
> select sum(distinct deptno), count(distinct deptno), max(deptno) from emp
> {code}
> The plan has regression from
> {code:xml}
> LogicalAggregate(group=[{}], EXPR$0=[SUM($0)], EXPR$1=[COUNT($0)],
> EXPR$2=[MAX($0)])
> LogicalAggregate(group=[{0}])
> LogicalProject(DEPTNO=[$7])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {code}
> to
> {code:xml}
> LogicalAggregate(group=[{}], EXPR$0=[SUM($0) FILTER $2], EXPR$1=[COUNT($0)
> FILTER $2], EXPR$2=[MIN($1) FILTER $3])
> LogicalProject(DEPTNO=[$0], EXPR$2=[$1], $g_0=[=($2, 0)], $g_1=[=($2, 1)])
> LogicalAggregate(group=[{0}], groups=[[{0}, {}]], EXPR$2=[MAX($0)],
> $g=[GROUPING($0)])
> LogicalProject(DEPTNO=[$7])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {code}
> The distinct trait actually can be reused.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (CALCITE-2829) Use consistent types when processing ranges
[ https://issues.apache.org/jira/browse/CALCITE-2829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen closed CALCITE-2829. --- Fix Version/s: 1.21.0 Assignee: Danny Chen (was: Juhwan Kim) Resolution: Fixed Close this issue because it is contained by CALCITE-2302. > Use consistent types when processing ranges > --- > > Key: CALCITE-2829 > URL: https://issues.apache.org/jira/browse/CALCITE-2829 > Project: Calcite > Issue Type: Bug >Reporter: Siddharth Teotia >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 1.21.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Range expressions like = 'literal' AND < 'literal' trigger > ClassCastException as literal are implicitly casted differently between =/<> > operators and other comparison operators. Apply the same casting rules for > comparison to =/<> calls when processing ranges, so that all the terms have > the same type for literals. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (CALCITE-3326) SQL with invalid function throws NullPointerException if typeCoercion is disabled
[
https://issues.apache.org/jira/browse/CALCITE-3326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen closed CALCITE-3326.
---
Fix Version/s: 1.22.0
Assignee: Danny Chen
Resolution: Fixed
Close and duplicated by CALCITE-3360.
> SQL with invalid function throws NullPointerException if typeCoercion is
> disabled
> -
>
> Key: CALCITE-3326
> URL: https://issues.apache.org/jira/browse/CALCITE-3326
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.21.0
>Reporter: Julian Hyde
>Assignee: Danny Chen
>Priority: Major
> Fix For: 1.22.0
>
>
> If you have a SQL query with a function that does not exist, and if
> typeCoercion is disabled, the validator throws a NullPointerException; it
> should throw a validation exception, same as if typeCoercion is enabled.
> Here is a testcase:
> {noformat}
> diff --git a/core/src/test/java/org/apache/calcite/test/SqlValidatorTest.java
> b/core/src/test/java/org/apache/calcite/test/SqlValidatorTest.java
> index 43621a7d6..227cc28e9 100644
> --- a/core/src/test/java/org/apache/calcite/test/SqlValidatorTest.java
> +++ b/core/src/test/java/org/apache/calcite/test/SqlValidatorTest.java
> @@ -1361,6 +1361,8 @@ public void _testLikeAndSimilarFails() {
>
>@Test public void testInvalidFunction() {
> checkWholeExpFails("foo()", "No match found for function signature
> FOO..");
> +checkFails("select foo()", "No match found for function signature FOO..",
> +false);
> checkWholeExpFails("mod(123)",
> "Invalid number of arguments to function 'MOD'. Was expecting 2
> arguments");
>}
> {noformat}
> throws
> {noformat}
> at java.base/java.util.Objects.requireNonNull(Objects.java:221)
> at org.apache.calcite.sql.SqlBasicCall.setOperator(SqlBasicCall.java:67)
> at org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:297)
> at org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:216)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:5626)
> {noformat}
> It's clear that the flow at [SqlFunction line
> 274|https://github.com/apache/calcite/blob/955d4ea7b30b85519529c1c77662fa04141bfa89/core/src/main/java/org/apache/calcite/sql/SqlFunction.java#L274]
> will let through {{function = null}} if coercion is disabled. This was
> caused by CALCITE-2302.
> Are there other possible paths through that block where {{function}} ends up
> null at the end? It's not clear from looking at the code.
> [~danny0405], Can you please take a look? Cc: [~fib-seq].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (CALCITE-3217) Support "SELECT NULL"
[
https://issues.apache.org/jira/browse/CALCITE-3217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen closed CALCITE-3217.
---
> Support "SELECT NULL"
> -
>
> Key: CALCITE-3217
> URL: https://issues.apache.org/jira/browse/CALCITE-3217
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Xurenhe
>Assignee: Danny Chen
>Priority: Minor
> Fix For: 1.21.0
>
> Attachments: Jietu20190728-221651.jpg
>
>
>
> Should Calcite need to support "select null", such as:
> {code:java}
> select null;{code}
> In the version of "1.20.0", it throws exception in
> `org.apache.calcite.sql.validate.SqlValidatorImpl#inferUnknownTypes`.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3081) Literal NULL should be generated in SqlDialect
[
https://issues.apache.org/jira/browse/CALCITE-3081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16950665#comment-16950665
]
Danny Chen commented on CALCITE-3081:
-
Hi, [~donnyzone], Calcite has supported "select null" now in CALCITE-2302, can
you go on to push this issue ?
> Literal NULL should be generated in SqlDialect
> --
>
> Key: CALCITE-3081
> URL: https://issues.apache.org/jira/browse/CALCITE-3081
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.19.0
>Reporter: Feng Zhu
>Priority: Minor
>
> In Calcite, this simple query will throw exception during validation, even it
> is ok in many databases.
> {code:java}
> Query:
> final String query = "select NULL as col "
> + "from \"foodmart\".\"product\"";
> Exception
> org.apache.calcite.tools.ValidationException:
> org.apache.calcite.runtime.CalciteContextException: From line 1, column 8 to
> line 1, column 11: Illegal use of 'NULL'
> {code}
> The right way to use 'NULL' in Calcite is:
> {code:java}
> final String query = "select cast(NULL as integer) as col "
> + "from \"foodmart\".\"product\"";
> {code}
> However, the converted query by *RelToSqlConverter* is illegal in Calcite.
> {code:java}
> SELECT NULL AS \"COL\"
> FROM \"foodmart\".\"product\"
> {code}
> The issue is trivial, but it is against to general sense. Maybe we can
> generate NULL literal in SqlDialect?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-3410) Simplify RelOptRulesTest and HepPlannerTest by making test methods fluent
Danny Chen created CALCITE-3410: --- Summary: Simplify RelOptRulesTest and HepPlannerTest by making test methods fluent Key: CALCITE-3410 URL: https://issues.apache.org/jira/browse/CALCITE-3410 Project: Calcite Issue Type: Improvement Components: core Affects Versions: 1.21.0 Reporter: Danny Chen Assignee: Danny Chen Fix For: 1.22.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CALCITE-3410) Simplify RelOptRulesTest and HepPlannerTest by making test methods fluent
[ https://issues.apache.org/jira/browse/CALCITE-3410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen resolved CALCITE-3410. - Resolution: Fixed Fixed in [8da131c|https://github.com/apache/calcite/commit/8da131c08afe67861ced49bc6180a20f66be52fb] ! > Simplify RelOptRulesTest and HepPlannerTest by making test methods fluent > - > > Key: CALCITE-3410 > URL: https://issues.apache.org/jira/browse/CALCITE-3410 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.21.0 >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 1.22.0 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3412) Response Error for FLOOR TIMESTAMP TO WEEK
[
https://issues.apache.org/jira/browse/CALCITE-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16951666#comment-16951666
]
Danny Chen commented on CALCITE-3412:
-
I'm curious what is the right start day of a week, Sunday ?
> Response Error for FLOOR TIMESTAMP TO WEEK
> ---
>
> Key: CALCITE-3412
> URL: https://issues.apache.org/jira/browse/CALCITE-3412
> Project: Calcite
> Issue Type: Bug
> Components: core
> Environment: mac,linux
>Reporter: huaicui
>Priority: Major
> Attachments: image-2019-10-15-13-33-34-896.png
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> The start day of the week is not Sunday or Monday, calcite use Tuesday to
> split a week, like this:
> select FLOOR(CAST('2017-01-28' AS TIMESTAMP) TO WEEK);
> The response is :
> 2017-01-26 00:00:00.0
> 2017-01-26 is Tuesday.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3414) Unify Expression'type cast and conversion as a robust one
[
https://issues.apache.org/jira/browse/CALCITE-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16951893#comment-16951893
]
Danny Chen commented on CALCITE-3414:
-
I'm inclined to move the Types#castIfNecessary into Calcite core, and use the
RexToLixTranslator#convert as the last fallback invocation. Then we should
always use Types#castIfNecessary instead of RexToLixTranslator#convert.
> Unify Expression'type cast and conversion as a robust one
> -
>
> Key: CALCITE-3414
> URL: https://issues.apache.org/jira/browse/CALCITE-3414
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.21.0
>Reporter: Feng Zhu
>Assignee: Feng Zhu
>Priority: Major
> Labels: pull-request-available
> Attachments: RexToLixTranslator.png, TypeConversion.txt, Types.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Current now, there are two functions in calcite that can be used to
> cast/convert Expression to a specific Type.
> *_Types.castIfNecessary_* and _*RexToLixTranslator.convert*_.
> We make a deep investigation on their implementations and demonstrate them as
> below.
> {color:#ff}
> !RexToLixTranslator.png!{color}
> {color:#ff}
> *RexToLixTranslator.convert*{color}
> !Types.png!
> {color:#ff}
> *Types.castIfNecessary*{color}
> It can be seen that:
> (1) They have a lot of overlaps;
> (2) *_RexToLixTranslator.cast_* can cover more cases with tools like
> _SqlFunctions_ and etc.
> (3) Both of them have limitations and may generate incorrect code, which is
> listed in attachment(TypeConversion.txt).
> Multiple choices usually bring confusion to developers and resulting to the
> misuse of them.
> For example, CALCITE-3245 exposes that Types.castIfNecessary cannot cast the
> Expression to BigDecimal.class.
> Fixing the issue in *_Types.castIfNecessary_* directly seems to be not a
> good idea.
> On one hand, it is not convenient to call _SqlFunctions_ in linq4j. One the
> other hand, it will brings duplicate with _*RexToLixTranslator.cast*_.
> However, due to some unique logic in _*Types.castIfNecessary*_, we cannot
> replace it as _*RexToLixTranslator.cast*_ neither.
> Therefore, it is a good idea to integrate implementations into
> RexToLixTranslator.cast.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3415) Cannot parse REGEXP_SUBSTR in BigQuery
[
https://issues.apache.org/jira/browse/CALCITE-3415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16952442#comment-16952442
]
Danny Chen commented on CALCITE-3415:
-
Thanks [~pranay.parmar] for reporting this, i checked that there indeed don't
have a REGEXP_SUBSTR function in Calcite now, i found that Oracle has such
function as you described, maybe we can add one for the Oracle dialect. You can
add one in SqlLibraryOperators[2]
[1] https://docs.oracle.com/cd/B12037_01/server.101/b10759/functions116.htm
[2]
https://github.com/apache/calcite/blob/3cbbafa941128dc5097c2a26711f5751f764e12d/core/src/main/java/org/apache/calcite/sql/fun/SqlLibraryOperators.java#L173
> Cannot parse REGEXP_SUBSTR in BigQuery
> --
>
> Key: CALCITE-3415
> URL: https://issues.apache.org/jira/browse/CALCITE-3415
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Pranay Parmar
>Priority: Minor
> Fix For: 1.22.0
>
>
> REGEXP_SUBSTR error :
> {code:java}
> No match found for function signature REGEXP_SUBSTR(, ,
> [, , ]){code}
>
> Example query:
> {code:sql}
> SELECT REGEXP_SUBSTR('chocolate Chip cookies', 'c+.{2}', 1, product_id, 'i')
> FROM public.account{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3416) SQL Dialects "DEFAULT"s should be more extensible
[
https://issues.apache.org/jira/browse/CALCITE-3416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16952457#comment-16952457
]
Danny Chen commented on CALCITE-3416:
-
Well, i think [~swtalbot]'s request is that how to config or fine tweak a
sub-class SqlDialect, for example, how we can change the unquotedCasing for
BigQuerySqlDialect, this may be the code for current codes:
{code:java}
SqlDialect dialect = new BigQuerySqlDialect(
EMPTY_CONTEXT
.withDatabaseProduct(SqlDialect.DatabaseProduct.BIG_QUERY)
.withLiteralQuoteString("'")
.withLiteralEscapedQuoteString("\\'")
.withIdentifierQuoteString("`")
.withNullCollation(NullCollation.LOW)
.withUnquotedCasing(Casing.UPPER_CASE) // what we have changed
.withQuotedCasing(Casing.UNCHANGED)
.withCaseSensitive(false));
{code}
It would be nice if we do not copy these redundant codes.
> SQL Dialects "DEFAULT"s should be more extensible
> -
>
> Key: CALCITE-3416
> URL: https://issues.apache.org/jira/browse/CALCITE-3416
> Project: Calcite
> Issue Type: Improvement
>Reporter: Steven Talbot
>Priority: Minor
>
> The behavior of SQLDialect is partly governed by the methods defined on the
> given dialect subclass and partly governed by options passed in in a Context
> object. So every dialect subclass exposes a "DEFAULT" instance that has been
> initialized with the correct Context.
> However, if you then wish to extend the dialect for one reason or another,
> you must create a new instance to extend, or of course create a whole new
> subclass. In either case, you lose the options from the Context passed into
> the default, which governs important behavior of the dialect. You can
> copy-paste the relevant Context out of the Calcite code, but then you lose
> future improvements or fixes that might land in mainline Calcite.
> It would be nice if each dialect exposed the DEFAULT_CONTEXT that it passed
> into its DEFAULT instance as a public final member. Then, when extending the
> dialect, you simply initialize your extension with the DEFAULT_CONTEXT, and
> if any customization needs to happens on the Context options that's easy to
> do with the Context's API.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3368) PLUS, MUNUS and TIMES should be unsafe when simplifying ‘expression IS NULL’
[
https://issues.apache.org/jira/browse/CALCITE-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16952471#comment-16952471
]
Danny Chen commented on CALCITE-3368:
-
I tested these queries in MYSQL 5.6:
{code:sql}
select -2147483648/0.25 from t; -- returns -8589934592
select -2147483648/0 from t; -- returns null ORACLE and MS-SQL throws exception
select 2147483647 + 2147483647 from t; -- returns 4294967294
select -2147483648 + -2147483648 from t; -- returns -4294967296
select -2147483648 * 2147483647 from t; -- returns -4611686016279904000
{code}
> PLUS, MUNUS and TIMES should be unsafe when simplifying ‘expression IS NULL’
>
>
> Key: CALCITE-3368
> URL: https://issues.apache.org/jira/browse/CALCITE-3368
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.21.0
>Reporter: Leonard Xu
>Assignee: Leonard Xu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> 'is null' expression in SQL may be optimized incorrectly in the underlying
> implementation.
>
> When I write a Fink SQL to test overflow just like
> {code:java}
> select
>case when (f0 + f1) is null then 'null' else 'not null' end
> from testTable
> {code}
> , I found expression '(f0 + f1) is null ' has been optimized by Calcite, and
> the optimization may be incorrect.
>
> The underlying implementation is that Calcite's simplification logic of
> isNull expression in SQL will convert from
> *"f(operand0, operand1) IS NULL"* to
> *"operand0 IS NULL OR operand1 IS NULL"* if the Policy of RexNode‘s SqlKind
> is ANY。
> This simplification leads to the expression will not calculate the real
> value of *f(operand0, operand1)* (eg.. '(f0 + f1)' in my case ），but '(f0 +
> f1)' maybe overflows after operation.
> {code:java}
> //org.apache.calcite.rex.RexSimplify.java
> private RexNode simplifyIsNull(RexNode a) {
> // Simplify the argument first,
> // call ourselves recursively to see whether we can make more progress.
> // For example, given
> // "(CASE WHEN FALSE THEN 1 ELSE 2) IS NULL" we first simplify the
> // argument to "2", and only then we can simplify "2 IS NULL" to "FALSE".
> a = simplify(a, UNKNOWN);
> if (!a.getType().isNullable() && isSafeExpression(a)) {
> return rexBuilder.makeLiteral(false);
> }
> if (RexUtil.isNull(a)) {
> return rexBuilder.makeLiteral(true);
> }
> if (a.getKind() == SqlKind.CAST) {
> return null;
> }
> switch (Strong.policy(a.getKind())) {
> case NOT_NULL:
> return rexBuilder.makeLiteral(false);
> case ANY:
> // "f" is a strong operator, so "f(operand0, operand1) IS NULL" simplifies
> // to "operand0 IS NULL OR operand1 IS NULL"
> final List operands = new ArrayList<>();
> for (RexNode operand : ((RexCall) a).getOperands()) {
> final RexNode simplified = simplifyIsNull(operand);
> if (simplified == null) {
> operands.add(
> rexBuilder.makeCall(SqlStdOperatorTable.IS_NULL, operand));
> } else {
> operands.add(simplified);
> }
> }
> return RexUtil.composeDisjunction(rexBuilder, operands, false);
> case AS_IS:
> default:
> return null;
> }
> }{code}
> And most of calculating SqlKinds are assigned *Policy.ANY* at present.
> {code:java}
> //org.apache.calcite.plan.Strong.java
> public static Policy policy(SqlKind kind) {
> return MAP.getOrDefault(kind, Policy.AS_IS);
> }
>
> map.put(SqlKind.PLUS, Policy.ANY);
> map.put(SqlKind.PLUS_PREFIX, Policy.ANY);
> map.put(SqlKind.MINUS, Policy.ANY);
> map.put(SqlKind.MINUS_PREFIX, Policy.ANY);
> map.put(SqlKind.TIMES, Policy.ANY);
> map.put(SqlKind.DIVIDE, Policy.ANY);
> * that operator evaluates to null. */
> public enum Policy {
> /** This kind of expression is never null. No need to look at its arguments,
>* if it has any. */
> NOT_NULL,
> /** This kind of expression has its own particular rules about whether it
>* is null. */
> CUSTOM,
> /** This kind of expression is null if and only if at least one of its
>* arguments is null. */
> ANY,
> /** This kind of expression may be null. There is no way to rewrite. */
> AS_IS,
> }{code}
>
> It may be an obvious nonequivalent simplification in SQL. And this issue come
> from Flink （FLINK-14030）.
> [~danny0405]， Could you have a look at this?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-3368) PLUS, MUNUS and TIMES should be unsafe when simplifying ‘expression IS NULL’
[
https://issues.apache.org/jira/browse/CALCITE-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16952471#comment-16952471
]
Danny Chen edited comment on CALCITE-3368 at 10/16/19 3:14 AM:
---
I tested these queries in MYSQL 5.6 and MS-SQL 2017:
{code:sql}
select -2147483648/0.25 from t; -- returns -8589934592
select -2147483648/0 from t; -- returns null ORACLE and MS-SQL throws exception
select 2147483647 + 2147483647 from t; -- returns 4294967294
select -2147483648 + -2147483648 from t; -- returns -4294967296
select -2147483648 * 2147483647 from t; -- returns -4611686016279904000
{code}
was (Author: danny0405):
I tested these queries in MYSQL 5.6:
{code:sql}
select -2147483648/0.25 from t; -- returns -8589934592
select -2147483648/0 from t; -- returns null ORACLE and MS-SQL throws exception
select 2147483647 + 2147483647 from t; -- returns 4294967294
select -2147483648 + -2147483648 from t; -- returns -4294967296
select -2147483648 * 2147483647 from t; -- returns -4611686016279904000
{code}
> PLUS, MUNUS and TIMES should be unsafe when simplifying ‘expression IS NULL’
>
>
> Key: CALCITE-3368
> URL: https://issues.apache.org/jira/browse/CALCITE-3368
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.21.0
>Reporter: Leonard Xu
>Assignee: Leonard Xu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> 'is null' expression in SQL may be optimized incorrectly in the underlying
> implementation.
>
> When I write a Fink SQL to test overflow just like
> {code:java}
> select
>case when (f0 + f1) is null then 'null' else 'not null' end
> from testTable
> {code}
> , I found expression '(f0 + f1) is null ' has been optimized by Calcite, and
> the optimization may be incorrect.
>
> The underlying implementation is that Calcite's simplification logic of
> isNull expression in SQL will convert from
> *"f(operand0, operand1) IS NULL"* to
> *"operand0 IS NULL OR operand1 IS NULL"* if the Policy of RexNode‘s SqlKind
> is ANY。
> This simplification leads to the expression will not calculate the real
> value of *f(operand0, operand1)* (eg.. '(f0 + f1)' in my case ），but '(f0 +
> f1)' maybe overflows after operation.
> {code:java}
> //org.apache.calcite.rex.RexSimplify.java
> private RexNode simplifyIsNull(RexNode a) {
> // Simplify the argument first,
> // call ourselves recursively to see whether we can make more progress.
> // For example, given
> // "(CASE WHEN FALSE THEN 1 ELSE 2) IS NULL" we first simplify the
> // argument to "2", and only then we can simplify "2 IS NULL" to "FALSE".
> a = simplify(a, UNKNOWN);
> if (!a.getType().isNullable() && isSafeExpression(a)) {
> return rexBuilder.makeLiteral(false);
> }
> if (RexUtil.isNull(a)) {
> return rexBuilder.makeLiteral(true);
> }
> if (a.getKind() == SqlKind.CAST) {
> return null;
> }
> switch (Strong.policy(a.getKind())) {
> case NOT_NULL:
> return rexBuilder.makeLiteral(false);
> case ANY:
> // "f" is a strong operator, so "f(operand0, operand1) IS NULL" simplifies
> // to "operand0 IS NULL OR operand1 IS NULL"
> final List operands = new ArrayList<>();
> for (RexNode operand : ((RexCall) a).getOperands()) {
> final RexNode simplified = simplifyIsNull(operand);
> if (simplified == null) {
> operands.add(
> rexBuilder.makeCall(SqlStdOperatorTable.IS_NULL, operand));
> } else {
> operands.add(simplified);
> }
> }
> return RexUtil.composeDisjunction(rexBuilder, operands, false);
> case AS_IS:
> default:
> return null;
> }
> }{code}
> And most of calculating SqlKinds are assigned *Policy.ANY* at present.
> {code:java}
> //org.apache.calcite.plan.Strong.java
> public static Policy policy(SqlKind kind) {
> return MAP.getOrDefault(kind, Policy.AS_IS);
> }
>
> map.put(SqlKind.PLUS, Policy.ANY);
> map.put(SqlKind.PLUS_PREFIX, Policy.ANY);
> map.put(SqlKind.MINUS, Policy.ANY);
> map.put(SqlKind.MINUS_PREFIX, Policy.ANY);
> map.put(SqlKind.TIMES, Policy.ANY);
> map.put(SqlKind.DIVIDE, Policy.ANY);
> * that operator evaluates to null. */
> public enum Policy {
> /** This kind of expression is never null. No need to look at its arguments,
>* if it has any. */
> NOT_NULL,
> /** This kind of expression has its own particular rules about whether it
>* is null. */
> CUSTOM,
> /** This kind of expression is null if and only if at least one of its
>* arguments is null. */
> ANY,
> /** This kind of expression may be null. There is no way to rewrite. */
> AS_IS,
> }{code}
>
> It may be an obvious nonequivalent simplification in SQL. And this issue come
> from Flink （FLINK-14030）.
> [~danny0405]， Cou
[jira] [Comment Edited] (CALCITE-3368) PLUS, MUNUS and TIMES should be unsafe when simplifying ‘expression IS NULL’
[
https://issues.apache.org/jira/browse/CALCITE-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16952471#comment-16952471
]
Danny Chen edited comment on CALCITE-3368 at 10/16/19 3:19 AM:
---
I tested these queries in MYSQL 5.6 and MS-SQL 2017:
{code:sql}
select -2147483648/0.25 from t; -- returns -8589934592
select -2147483648/0 from t; -- returns null ORACLE and MS-SQL throws exception
select 2147483647 + 2147483647 from t; -- returns 4294967294
select -2147483648 + -2147483648 from t; -- returns -4294967296
select -2147483648 * 2147483647 from t; -- returns -4611686016279904000
{code}
It seems that INTEGER + INTEGER does not return INTEGER for these engines,
which is different with Calcite.
was (Author: danny0405):
I tested these queries in MYSQL 5.6 and MS-SQL 2017:
{code:sql}
select -2147483648/0.25 from t; -- returns -8589934592
select -2147483648/0 from t; -- returns null ORACLE and MS-SQL throws exception
select 2147483647 + 2147483647 from t; -- returns 4294967294
select -2147483648 + -2147483648 from t; -- returns -4294967296
select -2147483648 * 2147483647 from t; -- returns -4611686016279904000
{code}
> PLUS, MUNUS and TIMES should be unsafe when simplifying ‘expression IS NULL’
>
>
> Key: CALCITE-3368
> URL: https://issues.apache.org/jira/browse/CALCITE-3368
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.21.0
>Reporter: Leonard Xu
>Assignee: Leonard Xu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> 'is null' expression in SQL may be optimized incorrectly in the underlying
> implementation.
>
> When I write a Fink SQL to test overflow just like
> {code:java}
> select
>case when (f0 + f1) is null then 'null' else 'not null' end
> from testTable
> {code}
> , I found expression '(f0 + f1) is null ' has been optimized by Calcite, and
> the optimization may be incorrect.
>
> The underlying implementation is that Calcite's simplification logic of
> isNull expression in SQL will convert from
> *"f(operand0, operand1) IS NULL"* to
> *"operand0 IS NULL OR operand1 IS NULL"* if the Policy of RexNode‘s SqlKind
> is ANY。
> This simplification leads to the expression will not calculate the real
> value of *f(operand0, operand1)* (eg.. '(f0 + f1)' in my case ），but '(f0 +
> f1)' maybe overflows after operation.
> {code:java}
> //org.apache.calcite.rex.RexSimplify.java
> private RexNode simplifyIsNull(RexNode a) {
> // Simplify the argument first,
> // call ourselves recursively to see whether we can make more progress.
> // For example, given
> // "(CASE WHEN FALSE THEN 1 ELSE 2) IS NULL" we first simplify the
> // argument to "2", and only then we can simplify "2 IS NULL" to "FALSE".
> a = simplify(a, UNKNOWN);
> if (!a.getType().isNullable() && isSafeExpression(a)) {
> return rexBuilder.makeLiteral(false);
> }
> if (RexUtil.isNull(a)) {
> return rexBuilder.makeLiteral(true);
> }
> if (a.getKind() == SqlKind.CAST) {
> return null;
> }
> switch (Strong.policy(a.getKind())) {
> case NOT_NULL:
> return rexBuilder.makeLiteral(false);
> case ANY:
> // "f" is a strong operator, so "f(operand0, operand1) IS NULL" simplifies
> // to "operand0 IS NULL OR operand1 IS NULL"
> final List operands = new ArrayList<>();
> for (RexNode operand : ((RexCall) a).getOperands()) {
> final RexNode simplified = simplifyIsNull(operand);
> if (simplified == null) {
> operands.add(
> rexBuilder.makeCall(SqlStdOperatorTable.IS_NULL, operand));
> } else {
> operands.add(simplified);
> }
> }
> return RexUtil.composeDisjunction(rexBuilder, operands, false);
> case AS_IS:
> default:
> return null;
> }
> }{code}
> And most of calculating SqlKinds are assigned *Policy.ANY* at present.
> {code:java}
> //org.apache.calcite.plan.Strong.java
> public static Policy policy(SqlKind kind) {
> return MAP.getOrDefault(kind, Policy.AS_IS);
> }
>
> map.put(SqlKind.PLUS, Policy.ANY);
> map.put(SqlKind.PLUS_PREFIX, Policy.ANY);
> map.put(SqlKind.MINUS, Policy.ANY);
> map.put(SqlKind.MINUS_PREFIX, Policy.ANY);
> map.put(SqlKind.TIMES, Policy.ANY);
> map.put(SqlKind.DIVIDE, Policy.ANY);
> * that operator evaluates to null. */
> public enum Policy {
> /** This kind of expression is never null. No need to look at its arguments,
>* if it has any. */
> NOT_NULL,
> /** This kind of expression has its own particular rules about whether it
>* is null. */
> CUSTOM,
> /** This kind of expression is null if and only if at least one of its
>* arguments is null. */
> ANY,
> /** This kind of expression may be null. There is no way to rewrite. */
> AS_IS,
> }{code}
>
>
[jira] [Commented] (CALCITE-3414) Unify Expression'type cast and conversion as a robust one
[
https://issues.apache.org/jira/browse/CALCITE-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16952480#comment-16952480
]
Danny Chen commented on CALCITE-3414:
-
Thanks Julian for clarifying the strategy, then i think keep
Types.castIfNecessary in linq4j is the right way to choose, but we should give
more doc to it and states that in which case it can be used.
Instead of copy code from Types.castIfNecessary into
RexToLixTranslator.convert, why not reuse and invoke Types.castIfNecessary
directly from inside RexToLixTranslator.convert ?
> Unify Expression'type cast and conversion as a robust one
> -
>
> Key: CALCITE-3414
> URL: https://issues.apache.org/jira/browse/CALCITE-3414
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.21.0
>Reporter: Feng Zhu
>Assignee: Feng Zhu
>Priority: Major
> Labels: pull-request-available
> Attachments: RexToLixTranslator.png, TypeConversion.txt, Types.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Current now, there are two functions in calcite that can be used to
> cast/convert Expression to a specific Type.
> *_Types.castIfNecessary_* and _*RexToLixTranslator.convert*_.
> We make a deep investigation on their implementations and demonstrate them as
> below.
> {color:#ff}
> !RexToLixTranslator.png!{color}
> {color:#ff}
> *RexToLixTranslator.convert*{color}
> !Types.png!
> {color:#ff}
> *Types.castIfNecessary*{color}
> It can be seen that:
> (1) They have a lot of overlaps;
> (2) *_RexToLixTranslator.cast_* can cover more cases with tools like
> _SqlFunctions_ and etc.
> (3) Both of them have limitations and may generate incorrect code, which is
> listed in attachment(TypeConversion.txt).
> Multiple choices usually bring confusion to developers and resulting to the
> misuse of them.
> For example, CALCITE-3245 exposes that Types.castIfNecessary cannot cast the
> Expression to BigDecimal.class.
> Fixing the issue in *_Types.castIfNecessary_* directly seems to be not a
> good idea.
> On one hand, it is not convenient to call _SqlFunctions_ in linq4j. One the
> other hand, it will brings duplicate with _*RexToLixTranslator.cast*_.
> However, due to some unique logic in _*Types.castIfNecessary*_, we cannot
> replace it as _*RexToLixTranslator.cast*_ neither.
> Therefore, it is a good idea to integrate implementations into
> RexToLixTranslator.cast.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-1075) UDF Error of Calcite 1.6
[ https://issues.apache.org/jira/browse/CALCITE-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16952481#comment-16952481 ] Danny Chen commented on CALCITE-1075: - I think it is because internally Calcite use Long to represent the datetime data type values, including DATE and TIMESTAMP. > UDF Error of Calcite 1.6 > - > > Key: CALCITE-1075 > URL: https://issues.apache.org/jira/browse/CALCITE-1075 > Project: Calcite > Issue Type: Bug >Affects Versions: 1.6.0 >Reporter: Wei Hu >Priority: Major > Attachments: 0001-test-cases-for-CALCITE-1075.patch > > > UDF Error In Calcite 1.6, may be Regression bugs? > schema：Table:hr, A String， B String， C String， D Timestamp > UDF： DATE_TRUNC（Timestamp D, String X） returns a timestamp value > there a several regression bugs. 1.5 is ok, but 1.6 will throw exception: > 1. UDF in Group_by clause, > sql：select min(m0), max(m0) from (select DATE_TRUNC(D, 'month') as d1, > COUNT(A) as m0 from \"hr\" group by DATE_TRUNC(D, 'month')) > Error Stack： > Caused by: java.lang.ClassCastException: java.sql.Timestamp cannot be cast to > java.lang.Long > at Baz$2.apply(Unknown Source) > at Baz$2.apply(Unknown Source) > at > org.apache.calcite.linq4j.EnumerableDefaults.groupBy_(EnumerableDefaults.java:821) > at > org.apache.calcite.linq4j.EnumerableDefaults.groupBy(EnumerableDefaults.java:754) > at > org.apache.calcite.linq4j.DefaultEnumerable.groupBy(DefaultEnumerable.java:302) > at Baz.bind(Unknown Source) > at > org.apache.calcite.jdbc.CalcitePrepare$CalciteSignature.enumerable(CalcitePrepare.java:326) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.enumerable(CalciteConnectionImpl.java:281) > at > org.apache.calcite.jdbc.CalciteMetaImpl._createIterable(CalciteMetaImpl.java:545) > at > org.apache.calcite.jdbc.CalciteMetaImpl.createIterable(CalciteMetaImpl.java:536) > at > org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:187) > at > org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:65) > at > org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:44) > at > org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:566) > at > org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:578) > at > org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:571) > at > org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:135) > ... 2 more > > > 2. select UDF > sql: select D, DATE_TRUNC(D, 'month') from \"hr\" group by D > throw exception when call resultSet.getTimestamp(1) > Error stack: > Exception in thread "main" java.lang.ClassCastException: java.sql.Timestamp > cannot be cast to java.lang.Number > at > org.apache.calcite.avatica.util.AbstractCursor$NumberAccessor.getNumber(AbstractCursor.java:696) > at > org.apache.calcite.avatica.util.AbstractCursor$TimestampFromNumberAccessor.getTimestamp(AbstractCursor.java:958) > at > org.apache.calcite.avatica.AvaticaResultSet.getTimestamp(AvaticaResultSet.java:285) > at testPs.TestPs.main(TestPs.java:150) > > 3. UDFs with same name and different arg types > UDF： DATE_TRUNC（Timestamp D, String x） returns a timestamp value > DATE_TRUNC(Date D, String x) returns a date value > > sql: select D, DATE_TRUNC(D, 'month') from \"hr\" group by D > Error stack： > Caused by: org.apache.calcite.runtime.CalciteContextException: From line 1, > column 11 to line 1, column 32: Cannot apply 'DATE_TRUNC' to arguments of > type 'DATE_TRUNC(, )'. Supported form(s): > 'DATE_TRUNC(, )' > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:526) > at > org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:405) > at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:715) > at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:703) > at > org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError(SqlValidatorImpl.java:3932) > at > org.apache.calcite.sql.SqlCallBinding.newValidationSignatureError(SqlCallBinding.java:275) > at > org.apache.calcite.sql.type.FamilyOperandTypeChecker.checkSingleOperandType(FamilyOperandTypeChecker.java:92) > at > org.apache.calcite.sql.type.FamilyOperandTypeChecker.checkO
[jira] [Commented] (CALCITE-3412) Response Error for FLOOR TIMESTAMP TO WEEK
[
https://issues.apache.org/jira/browse/CALCITE-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16952489#comment-16952489
]
Danny Chen commented on CALCITE-3412:
-
I didn't find any engine to support 'FLOOR(... TO XXX)' [1][2], this is the
grammar i found in SQL standard 2011:
{code:sql}
::=
FLOOR
::=
|
|
{code}
[1] https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions058.htm
[2]
https://docs.microsoft.com/en-us/sql/t-sql/functions/floor-transact-sql?view=sql-server-ver15
> Response Error for FLOOR TIMESTAMP TO WEEK
> ---
>
> Key: CALCITE-3412
> URL: https://issues.apache.org/jira/browse/CALCITE-3412
> Project: Calcite
> Issue Type: Bug
> Components: core
> Environment: mac,linux
>Reporter: huaicui
>Priority: Major
> Attachments: image-2019-10-15-13-33-34-896.png
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Floor timestamp to week exist error:
> The start day of the week is not Sunday or Monday, calcite use Thursday to
> split a week.
> Example:
> sql: select FLOOR(CAST('2017-01-28' AS TIMESTAMP) TO WEEK);
> Response:
> 2017-01-26 00:00:00.0
>
> 2017-01-26 is Tuesday that is not our expect result. As result, 2017-01-22 is
> the first day that we expect.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3416) SQL Dialects "DEFAULT"s should be more extensible
[ https://issues.apache.org/jira/browse/CALCITE-3416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16952594#comment-16952594 ] Danny Chen commented on CALCITE-3416: - Yes, we may need a way to give user a "default" initial context to let them customize. > SQL Dialects "DEFAULT"s should be more extensible > - > > Key: CALCITE-3416 > URL: https://issues.apache.org/jira/browse/CALCITE-3416 > Project: Calcite > Issue Type: Improvement >Reporter: Steven Talbot >Priority: Minor > > The behavior of SQLDialect is partly governed by the methods defined on the > given dialect subclass and partly governed by options passed in in a Context > object. So every dialect subclass exposes a "DEFAULT" instance that has been > initialized with the correct Context. > However, if you then wish to extend the dialect for one reason or another, > you must create a new instance to extend, or of course create a whole new > subclass. In either case, you lose the options from the Context passed into > the default, which governs important behavior of the dialect. You can > copy-paste the relevant Context out of the Calcite code, but then you lose > future improvements or fixes that might land in mainline Calcite. > It would be nice if each dialect exposed the DEFAULT_CONTEXT that it passed > into its DEFAULT instance as a public final member. Then, when extending the > dialect, you simply initialize your extension with the DEFAULT_CONTEXT, and > if any customization needs to happens on the Context options that's easy to > do with the Context's API. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3414) Unify Expression'type cast and conversion as a robust one
[
https://issues.apache.org/jira/browse/CALCITE-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16952595#comment-16952595
]
Danny Chen commented on CALCITE-3414:
-
For (1) i think we can make a fix, for (2) could we make another method for the
unique logic in linq4j then Types.castIfNecessary and
RexToLixTranslator.convert can both reuse them ?
> Unify Expression'type cast and conversion as a robust one
> -
>
> Key: CALCITE-3414
> URL: https://issues.apache.org/jira/browse/CALCITE-3414
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.21.0
>Reporter: Feng Zhu
>Assignee: Feng Zhu
>Priority: Major
> Labels: pull-request-available
> Attachments: RexToLixTranslator.png, TypeConversion.txt, Types.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Current now, there are two functions in calcite that can be used to
> cast/convert Expression to a specific Type.
> *_Types.castIfNecessary_* and _*RexToLixTranslator.convert*_.
> We make a deep investigation on their implementations and demonstrate them as
> below.
> {color:#ff}
> !RexToLixTranslator.png!{color}
> {color:#ff}
> *RexToLixTranslator.convert*{color}
> !Types.png!
> {color:#ff}
> *Types.castIfNecessary*{color}
> It can be seen that:
> (1) They have a lot of overlaps;
> (2) *_RexToLixTranslator.cast_* can cover more cases with tools like
> _SqlFunctions_ and etc.
> (3) Both of them have limitations and may generate incorrect code, which is
> listed in attachment(TypeConversion.txt).
> Multiple choices usually bring confusion to developers and resulting to the
> misuse of them.
> For example, CALCITE-3245 exposes that Types.castIfNecessary cannot cast the
> Expression to BigDecimal.class.
> Fixing the issue in *_Types.castIfNecessary_* directly seems to be not a
> good idea.
> On one hand, it is not convenient to call _SqlFunctions_ in linq4j. One the
> other hand, it will brings duplicate with _*RexToLixTranslator.cast*_.
> However, due to some unique logic in _*Types.castIfNecessary*_, we cannot
> replace it as _*RexToLixTranslator.cast*_ neither.
> Therefore, it is a good idea to integrate implementations into
> RexToLixTranslator.cast.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (CALCITE-3415) Cannot parse REGEXP_SUBSTR in BigQuery
[
https://issues.apache.org/jira/browse/CALCITE-3415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen reassigned CALCITE-3415:
---
Assignee: (was: Danny Chen)
> Cannot parse REGEXP_SUBSTR in BigQuery
> --
>
> Key: CALCITE-3415
> URL: https://issues.apache.org/jira/browse/CALCITE-3415
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.21.0
>Reporter: Pranay Parmar
>Priority: Minor
>
> REGEXP_SUBSTR error :
> {code:java}
> No match found for function signature REGEXP_SUBSTR(, ,
> [, , ]){code}
>
> Example query:
> {code:sql}
> SELECT REGEXP_SUBSTR('chocolate Chip cookies', 'c+.{2}', 1, product_id, 'i')
> FROM public.account{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3415) Cannot parse REGEXP_SUBSTR in BigQuery
[
https://issues.apache.org/jira/browse/CALCITE-3415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16952643#comment-16952643
]
Danny Chen commented on CALCITE-3415:
-
Sorry for cannot doing that because you have no contrition permission for the
JIRA cases, you can apply for it by sending a mail to DEV mailing list.
> Cannot parse REGEXP_SUBSTR in BigQuery
> --
>
> Key: CALCITE-3415
> URL: https://issues.apache.org/jira/browse/CALCITE-3415
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.21.0
>Reporter: Pranay Parmar
>Priority: Minor
>
> REGEXP_SUBSTR error :
> {code:java}
> No match found for function signature REGEXP_SUBSTR(, ,
> [, , ]){code}
>
> Example query:
> {code:sql}
> SELECT REGEXP_SUBSTR('chocolate Chip cookies', 'c+.{2}', 1, product_id, 'i')
> FROM public.account{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (CALCITE-3415) Cannot parse REGEXP_SUBSTR in BigQuery
[
https://issues.apache.org/jira/browse/CALCITE-3415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen reassigned CALCITE-3415:
---
Assignee: Danny Chen
> Cannot parse REGEXP_SUBSTR in BigQuery
> --
>
> Key: CALCITE-3415
> URL: https://issues.apache.org/jira/browse/CALCITE-3415
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.21.0
>Reporter: Pranay Parmar
>Assignee: Danny Chen
>Priority: Minor
>
> REGEXP_SUBSTR error :
> {code:java}
> No match found for function signature REGEXP_SUBSTR(, ,
> [, , ]){code}
>
> Example query:
> {code:sql}
> SELECT REGEXP_SUBSTR('chocolate Chip cookies', 'c+.{2}', 1, product_id, 'i')
> FROM public.account{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3419) Tools for converting List to SQL expression string
[ https://issues.apache.org/jira/browse/CALCITE-3419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16952685#comment-16952685 ] Danny Chen commented on CALCITE-3419: - You can use SqlImplementor#toSql[1] to convert RexNode to SqlNode then use SqlNode#toSqlString to convert it to SQL-like string. [1] https://github.com/apache/calcite/blob/3cbbafa941128dc5097c2a26711f5751f764e12d/core/src/main/java/org/apache/calcite/rel/rel2sql/SqlImplementor.java#L480 > Tools for converting List to SQL expression string > > > Key: CALCITE-3419 > URL: https://issues.apache.org/jira/browse/CALCITE-3419 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: jamie12221 >Priority: Trivial > > 'org.apache.calcite.schema.ProjectableFilterableTable' Is a simple tool that > can apply filter expressions.On a datasource that uses SQL as a query > command, the org.apache.calcite.schema.ProjectableFilterableTable#scan method > applies a filter expression, and the filter expression may be converted to > SQL. I have not found a way to convert List to SQL. In particular, > OR expressions may involve some complicated operations.Is there a better > solution? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-3420) NullPointerException throws for implicit type coercion of nested SET operations
Danny Chen created CALCITE-3420:
---
Summary: NullPointerException throws for implicit type coercion of
nested SET operations
Key: CALCITE-3420
URL: https://issues.apache.org/jira/browse/CALCITE-3420
Project: Calcite
Issue Type: Improvement
Components: core
Affects Versions: 1.21.0
Reporter: Danny Chen
Assignee: Danny Chen
Fix For: 1.22.0
Check this sql in TypeCoercionTest:
{code:sql}
select t1_varchar20 from t1
union select t2_int from t2
union select t1_int from t1
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3300) Add a star identifier as operand to "SqlCountAggFunction" if no operands given while in the method "createCall"
[
https://issues.apache.org/jira/browse/CALCITE-3300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16953325#comment-16953325
]
Danny Chen commented on CALCITE-3300:
-
[~wangweidong], i am also inclined that we should not "fix" the
SqlCountAggFunction, just fix the RelToSqlConverter is okey for this issue. You
can match the count agg call, and customize the "count(*)" grammar when you
found that its arguments are empty. You can take [1] as a reference.
The "exec0" method name is kind of confusing and conflict with the exists name
"exec", if you want a SqlNode converted back from RelNode, can you just defined
a method by yourself, it seems that only you have such requests.
[1]
https://github.com/apache/calcite/blob/3cbbafa941128dc5097c2a26711f5751f764e12d/core/src/main/java/org/apache/calcite/sql/dialect/MssqlSqlDialect.java#L143
> Add a star identifier as operand to "SqlCountAggFunction" if no operands
> given while in the method "createCall"
> ---
>
> Key: CALCITE-3300
> URL: https://issues.apache.org/jira/browse/CALCITE-3300
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.21.0
>Reporter: Wang Weidong
>Assignee: Wang Weidong
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> h4. Procedure to reproduce the problem:
> - create table {{~^t^~}}
> - parse query select count(*) from t to SqlNode
> - convert SqlNode to RelNode
> - convert RelNode to SqlNode
> - validate the result SqlNode
> test code is like this
> {code:java}
> // code placeholder
> @Test
> public void testSelectCountAll() throws Exception {
> try (Statement s = parserContext.getStatement()) {
> s.execute("create table if not exists t (i int not null)");
> String sql = "select count(*) from t";
> SqlNode sqlNode = parserContext.parseStmt(sql);
> parserContext.getSqlValidator().validate(sqlNode);
> RelNode relNode =
> parserContext.getSqlToRelConverter().convertQuery(sqlNode, true, true).rel;
> SqlNode sqlNodeNew = toSqlNode(relNode);
> parserContext.getSqlValidator().validate(sqlNodeNew);
> }
> }
> {code}
> Finally we will get an exception as follow:
> {code:java}
> // code placeholder
> org.apache.calcite.runtime.CalciteContextException: At line 0, column 0:
> Invalid number of arguments to function 'COUNT'. Was expecting 1
> argumentsorg.apache.calcite.runtime.CalciteContextException: At line 0,
> column 0: Invalid number of arguments to function 'COUNT'. Was expecting 1
> arguments at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method) at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at
> org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:463)
> at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:783) at
> org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:768) at
> org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError(SqlValidatorImpl.java:4753)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.handleUnresolvedFunction(SqlValidatorImpl.java:1699)
> at org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:270) at
> org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:215) at
> org.apache.calcite.sql.fun.SqlCountAggFunction.deriveType(SqlCountAggFunction.java:83)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:5477)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:5464)
> at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:138) at
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1629)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1614)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.expandSelectItem(SqlValidatorImpl.java:457)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelectList(SqlValidatorImpl.java:4017)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3280)
> at
> org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:967)
> at
> org.apache.calcite.sql.validate.Sq
[jira] [Resolved] (CALCITE-3420) NullPointerException throws for implicit type coercion of nested SET operations
[
https://issues.apache.org/jira/browse/CALCITE-3420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen resolved CALCITE-3420.
-
Resolution: Fixed
Fixed in
[2ac4415|https://github.com/apache/calcite/commit/2ac44153c4a521694c4dc975dd37b097184fc998]
!
> NullPointerException throws for implicit type coercion of nested SET
> operations
> ---
>
> Key: CALCITE-3420
> URL: https://issues.apache.org/jira/browse/CALCITE-3420
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.21.0
>Reporter: Danny Chen
>Assignee: Danny Chen
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.22.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Check this sql in TypeCoercionTest:
> {code:sql}
> select t1_varchar20 from t1
> union select t2_int from t2
> union select t1_int from t1
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3415) Cannot parse REGEXP_SUBSTR in BigQuery
[
https://issues.apache.org/jira/browse/CALCITE-3415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16953342#comment-16953342
]
Danny Chen commented on CALCITE-3415:
-
Thanks for the contribution, hoping for your PR.
> Cannot parse REGEXP_SUBSTR in BigQuery
> --
>
> Key: CALCITE-3415
> URL: https://issues.apache.org/jira/browse/CALCITE-3415
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.21.0
>Reporter: Pranay Parmar
>Priority: Minor
>
> REGEXP_SUBSTR error :
> {code:java}
> No match found for function signature REGEXP_SUBSTR(, ,
> [, , ]){code}
>
> Example query:
> {code:sql}
> SELECT REGEXP_SUBSTR('chocolate Chip cookies', 'c+.{2}', 1, product_id, 'i')
> FROM public.account{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-3412) FLOOR(timestamp TO WEEK) gives wrong result
[
https://issues.apache.org/jira/browse/CALCITE-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen updated CALCITE-3412:
Description:
Floor timestamp to week exist error:
The start day of the week is not Sunday or Monday, calcite use Thursday to
split a week.
Example:
sql: select FLOOR(CAST('2017-01-28' AS TIMESTAMP) TO WEEK);
Response:
2017-01-26 00:00:00.0
2017-01-26 is Thursday that is not our expect result. As result, 2017-01-22 is
the first day that we expect.
was:
Floor timestamp to week exist error:
The start day of the week is not Sunday or Monday, calcite use Thursday to
split a week.
Example:
sql: select FLOOR(CAST('2017-01-28' AS TIMESTAMP) TO WEEK);
Response:
2017-01-26 00:00:00.0
2017-01-26 is Tuesday that is not our expect result. As result, 2017-01-22 is
the first day that we expect.
> FLOOR(timestamp TO WEEK) gives wrong result
> ---
>
> Key: CALCITE-3412
> URL: https://issues.apache.org/jira/browse/CALCITE-3412
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: huaicui
>Assignee: Julian Hyde
>Priority: Major
> Fix For: 1.22.0
>
> Attachments: image-2019-10-15-13-33-34-896.png
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Floor timestamp to week exist error:
> The start day of the week is not Sunday or Monday, calcite use Thursday to
> split a week.
> Example:
> sql: select FLOOR(CAST('2017-01-28' AS TIMESTAMP) TO WEEK);
> Response:
> 2017-01-26 00:00:00.0
>
> 2017-01-26 is Thursday that is not our expect result. As result, 2017-01-22
> is the first day that we expect.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-2303) In EXTRACT function, support MICROSECONDS, MILLISECONDS, EPOCH, ISODOW, ISOYEAR and DECADE time units
[
https://issues.apache.org/jira/browse/CALCITE-2303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16954211#comment-16954211
]
Danny Chen commented on CALCITE-2303:
-
Thanks for the fixing, Julian ~
> In EXTRACT function, support MICROSECONDS, MILLISECONDS, EPOCH, ISODOW,
> ISOYEAR and DECADE time units
> -
>
> Key: CALCITE-2303
> URL: https://issues.apache.org/jira/browse/CALCITE-2303
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Sergey Nuyanzin
>Assignee: Julian Hyde
>Priority: Major
> Fix For: 1.17.0, avatica-1.13.0
>
>
> Here CALCITE-1177 were supported new units
> however such test
> {code:java}
> @Test public void testDecadeFunction() throws Exception {
> ExpressionChecker checker = new ExpressionChecker()
> .addExpr("EXTRACT(DECADE FROM ts)", 199L)
> ;
> checker.buildRunAndCheck();
> }
> {code}
> failed like
> Extract for time unit: DECADE not supported!
> {noformat}
> SQL:>
> SELECT EXTRACT(DECADE FROM ts) FROM PCOLLECTION
> May 08, 2018 1:34:58 PM
> org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner
> validateAndConvert
> INFO: SQL:
> SELECT EXTRACT(DECADE FROM `PCOLLECTION`.`ts`)
> FROM `PCOLLECTION` AS `PCOLLECTION`
> May 08, 2018 1:34:58 PM
> org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner
> convertToBeamRel
> INFO: SQLPlan>
> LogicalProject(EXPR$0=[EXTRACT(FLAG(DECADE), $0)])
> BeamIOSourceRel(table=[[PCOLLECTION]])
> java.lang.RuntimeException:
> org.apache.beam.sdk.Pipeline$PipelineExecutionException:
> java.lang.UnsupportedOperationException: Extract for time unit: DECADE not
> supported!
> at
> org.apache.beam.sdk.extensions.sql.integrationtest.BeamSqlBuiltinFunctionsIntegrationTestBase$ExpressionChecker.buildRunAndCheck(BeamSqlBuiltinFunctionsIntegrationTestBase.java:167)
> at
> org.apache.beam.sdk.extensions.sql.integrationtest.BeamSqlDateFunctionsIntegrationTest.testDecadeFunction(BeamSqlDateFunctionsIntegrationTest.java:66)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:317)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
> at
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> Caused by: org.apache.beam.sdk.Pipeline$PipelineExecutionException:
> java.lang.UnsupportedOperationException: Extract for time unit: DECADE not
> supported!
> at
> org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:349)
> at
> org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:319)
> at
> org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:210)
> at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:66)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:311)
> at org.apache.beam.sdk.testin
[jira] [Commented] (CALCITE-3412) FLOOR(timestamp TO WEEK) gives wrong result
[
https://issues.apache.org/jira/browse/CALCITE-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16954213#comment-16954213
]
Danny Chen commented on CALCITE-3412:
-
Thanks for the fixing, Julian ~
> FLOOR(timestamp TO WEEK) gives wrong result
> ---
>
> Key: CALCITE-3412
> URL: https://issues.apache.org/jira/browse/CALCITE-3412
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: huaicui
>Assignee: Julian Hyde
>Priority: Major
> Fix For: 1.22.0
>
> Attachments: image-2019-10-15-13-33-34-896.png
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Floor timestamp to week exist error:
> The start day of the week is not Sunday or Monday, calcite use Thursday to
> split a week.
> Example:
> sql: select FLOOR(CAST('2017-01-28' AS TIMESTAMP) TO WEEK);
> Response:
> 2017-01-26 00:00:00.0
>
> 2017-01-26 is Thursday that is not our expect result. As result, 2017-01-22
> is the first day that we expect.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3425) Inconsistent behavior of MetadataProvider in RelOptCluster
[
https://issues.apache.org/jira/browse/CALCITE-3425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16954388#comment-16954388
]
Danny Chen commented on CALCITE-3425:
-
Thanks [~hyuan] for firing this issue, let's continue the discussion on the DEV
mailing list.
> Inconsistent behavior of MetadataProvider in RelOptCluster
> --
>
> Key: CALCITE-3425
> URL: https://issues.apache.org/jira/browse/CALCITE-3425
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
>
> To use customized metadata provider, we can do the following:
> {code:java}
> RelMetadataQuery.THREAD_PROVIDERS.set(
> JaninoRelMetadataProvider.of(xxxmetadataProvider));
> {code}
> It only works for builtin metadata type, but for customized metadata, we
> still get exception when retrieve the metadata using reflection. Because when
> the RelOptCluster is created, it always use the default metadata provider,
> instead of the customized one.
> {code:java}
> setMetadataProvider(DefaultRelMetadataProvider.INSTANCE);
> {code}
> It causes confusing. We have to set the provider in 2 places. Should we unify
> them in a single place?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-3318) Preserving CAST of STRING operands in comparison operator
[
https://issues.apache.org/jira/browse/CALCITE-3318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen resolved CALCITE-3318.
-
Fix Version/s: 1.22.0
Assignee: Danny Chen
Resolution: Fixed
Fixed in
[a39cab3|https://github.com/apache/calcite/commit/a39cab31bf202645743138b8f632607e1776a3a1],
thanks for your PR, [~Soma.Mondal] !
> Preserving CAST of STRING operands in comparison operator
> -
>
> Key: CALCITE-3318
> URL: https://issues.apache.org/jira/browse/CALCITE-3318
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.20.0
>Reporter: Soma Mondal
>Assignee: Danny Chen
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.22.0
>
> Time Spent: 5h 40m
> Remaining Estimate: 0h
>
> We have a REL which has this information
> select * from employee where employee_id = cast('12' as float);
> but Calcite removes the CAST from the STRING literal('12' in our case).
> select * from employee where employee_id = '12';
> There are dialects which needs explicit casting in the above case and we need
> to maintain the CAST in our dialect.
> Calcite removes the cast in SqlImplementor's stripCastFromString() method.
> I'm thinking of having the dialect intercept this and decide whether or not
> to remove the cast.
>
> I have created a PR [https://github.com/apache/calcite/pull/1437]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-3318) Preserving CAST of VARCHAR operand in binary comparison for BigQuery
[
https://issues.apache.org/jira/browse/CALCITE-3318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen updated CALCITE-3318:
Summary: Preserving CAST of VARCHAR operand in binary comparison for
BigQuery (was: Preserving CAST of STRING operands in comparison operator)
> Preserving CAST of VARCHAR operand in binary comparison for BigQuery
>
>
> Key: CALCITE-3318
> URL: https://issues.apache.org/jira/browse/CALCITE-3318
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.20.0
>Reporter: Soma Mondal
>Assignee: Danny Chen
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.22.0
>
> Time Spent: 5h 40m
> Remaining Estimate: 0h
>
> We have a REL which has this information
> select * from employee where employee_id = cast('12' as float);
> but Calcite removes the CAST from the STRING literal('12' in our case).
> select * from employee where employee_id = '12';
> There are dialects which needs explicit casting in the above case and we need
> to maintain the CAST in our dialect.
> Calcite removes the cast in SqlImplementor's stripCastFromString() method.
> I'm thinking of having the dialect intercept this and decide whether or not
> to remove the cast.
>
> I have created a PR [https://github.com/apache/calcite/pull/1437]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-3318) Preserving CAST of VARCHAR operand in binary comparison for BigQuery
[
https://issues.apache.org/jira/browse/CALCITE-3318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16954453#comment-16954453
]
Danny Chen edited comment on CALCITE-3318 at 10/18/19 11:04 AM:
Fixed in
[cd24cae|https://github.com/apache/calcite/commit/cd24cae77072e56e4333d10114bf380be79709f1],
thanks for your PR, [~Soma.Mondal] !
was (Author: danny0405):
Fixed in
[a39cab3|https://github.com/apache/calcite/commit/a39cab31bf202645743138b8f632607e1776a3a1],
thanks for your PR, [~Soma.Mondal] !
> Preserving CAST of VARCHAR operand in binary comparison for BigQuery
>
>
> Key: CALCITE-3318
> URL: https://issues.apache.org/jira/browse/CALCITE-3318
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.20.0
>Reporter: Soma Mondal
>Assignee: Danny Chen
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.22.0
>
> Time Spent: 5h 40m
> Remaining Estimate: 0h
>
> We have a REL which has this information
> select * from employee where employee_id = cast('12' as float);
> but Calcite removes the CAST from the STRING literal('12' in our case).
> select * from employee where employee_id = '12';
> There are dialects which needs explicit casting in the above case and we need
> to maintain the CAST in our dialect.
> Calcite removes the cast in SqlImplementor's stripCastFromString() method.
> I'm thinking of having the dialect intercept this and decide whether or not
> to remove the cast.
>
> I have created a PR [https://github.com/apache/calcite/pull/1437]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-3426) Complement validConstant type in RexLiteral
[ https://issues.apache.org/jira/browse/CALCITE-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated CALCITE-3426: Summary: Complement validConstant type in RexLiteral (was: compensate validConstant type in RexLiteral.) > Complement validConstant type in RexLiteral > --- > > Key: CALCITE-3426 > URL: https://issues.apache.org/jira/browse/CALCITE-3426 > Project: Calcite > Issue Type: Wish >Reporter: xzh_dz >Priority: Minor > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > compensate validConstant type in RexLiteral. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3414) Unify Expression'type cast and conversion as a robust one
[
https://issues.apache.org/jira/browse/CALCITE-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16955692#comment-16955692
]
Danny Chen commented on CALCITE-3414:
-
Reviewing now ~
> Unify Expression'type cast and conversion as a robust one
> -
>
> Key: CALCITE-3414
> URL: https://issues.apache.org/jira/browse/CALCITE-3414
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.21.0
>Reporter: Feng Zhu
>Assignee: Feng Zhu
>Priority: Major
> Labels: pull-request-available
> Attachments: RexToLixTranslator.png, TypeConversion.txt, Types.png
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Current now, there are two functions in calcite that can be used to
> cast/convert Expression to a specific Type.
> *_Types.castIfNecessary_* and _*RexToLixTranslator.convert*_.
> We make a deep investigation on their implementations and demonstrate them as
> below.
> {color:#ff}
> !RexToLixTranslator.png!{color}
> {color:#ff}
> *RexToLixTranslator.convert*{color}
> !Types.png!
> {color:#ff}
> *Types.castIfNecessary*{color}
> It can be seen that:
> (1) They have a lot of overlaps;
> (2) *_RexToLixTranslator.cast_* can cover more cases with tools like
> _SqlFunctions_ and etc.
> (3) Both of them have limitations and may generate incorrect code, which is
> listed in attachment(TypeConversion.txt).
> Multiple choices usually bring confusion to developers and resulting to the
> misuse of them.
> For example, CALCITE-3245 exposes that Types.castIfNecessary cannot cast the
> Expression to BigDecimal.class.
> Fixing the issue in *_Types.castIfNecessary_* directly seems to be not a
> good idea.
> On one hand, it is not convenient to call _SqlFunctions_ in linq4j. One the
> other hand, it will brings duplicate with _*RexToLixTranslator.cast*_.
> However, due to some unique logic in _*Types.castIfNecessary*_, we cannot
> replace it as _*RexToLixTranslator.cast*_ neither.
> Therefore, it is a good idea to integrate implementations into
> RexToLixTranslator.cast.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (CALCITE-3432) RexBuilder construct method with the default of RelDataTypeFactory.
[ https://issues.apache.org/jira/browse/CALCITE-3432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen closed CALCITE-3432. --- Resolution: Won't Do > RexBuilder construct method with the default of RelDataTypeFactory. > --- > > Key: CALCITE-3432 > URL: https://issues.apache.org/jira/browse/CALCITE-3432 > Project: Calcite > Issue Type: Wish >Reporter: xzh_dz >Priority: Minor > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > RexBuilder construct method with the default of RelDataTypeFactory. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-3415) Supports REGEXP_SUBSTR function
[
https://issues.apache.org/jira/browse/CALCITE-3415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen updated CALCITE-3415:
Summary: Supports REGEXP_SUBSTR function (was: Cannot parse REGEXP_SUBSTR
in BigQuery)
> Supports REGEXP_SUBSTR function
> ---
>
> Key: CALCITE-3415
> URL: https://issues.apache.org/jira/browse/CALCITE-3415
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.21.0
>Reporter: Pranay Parmar
>Priority: Minor
>
> REGEXP_SUBSTR error :
> {code:java}
> No match found for function signature REGEXP_SUBSTR(, ,
> [, , ]){code}
>
> Example query:
> {code:sql}
> SELECT REGEXP_SUBSTR('chocolate Chip cookies', 'c+.{2}', 1, product_id, 'i')
> FROM public.account{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-3414) [CALCITE-3414] In calcite-core, use RexToLixTranslator.convert for type conversion code generation uniformly
[
https://issues.apache.org/jira/browse/CALCITE-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen updated CALCITE-3414:
Summary: [CALCITE-3414] In calcite-core, use RexToLixTranslator.convert for
type conversion code generation uniformly (was: Unify Expression'type cast and
conversion as a robust one)
> [CALCITE-3414] In calcite-core, use RexToLixTranslator.convert for type
> conversion code generation uniformly
>
>
> Key: CALCITE-3414
> URL: https://issues.apache.org/jira/browse/CALCITE-3414
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.21.0
>Reporter: Feng Zhu
>Assignee: Feng Zhu
>Priority: Major
> Labels: pull-request-available
> Attachments: RexToLixTranslator.png, TypeConversion.txt, Types.png
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Current now, there are two functions in calcite that can be used to
> cast/convert Expression to a specific Type.
> *_Types.castIfNecessary_* and _*RexToLixTranslator.convert*_.
> We make a deep investigation on their implementations and demonstrate them as
> below.
> {color:#ff}
> !RexToLixTranslator.png!{color}
> {color:#ff}
> *RexToLixTranslator.convert*{color}
> !Types.png!
> {color:#ff}
> *Types.castIfNecessary*{color}
> It can be seen that:
> (1) They have a lot of overlaps;
> (2) *_RexToLixTranslator.cast_* can cover more cases with tools like
> _SqlFunctions_ and etc.
> (3) Both of them have limitations and may generate incorrect code, which is
> listed in attachment(TypeConversion.txt).
> Multiple choices usually bring confusion to developers and resulting to the
> misuse of them.
> For example, CALCITE-3245 exposes that Types.castIfNecessary cannot cast the
> Expression to BigDecimal.class.
> Fixing the issue in *_Types.castIfNecessary_* directly seems to be not a
> good idea.
> On one hand, it is not convenient to call _SqlFunctions_ in linq4j. One the
> other hand, it will brings duplicate with _*RexToLixTranslator.cast*_.
> However, due to some unique logic in _*Types.castIfNecessary*_, we cannot
> replace it as _*RexToLixTranslator.cast*_ neither.
> Therefore, it is a good idea to integrate implementations into
> RexToLixTranslator.cast.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-3414) In calcite-core, use RexToLixTranslator.convert for type conversion code generation uniformly
[
https://issues.apache.org/jira/browse/CALCITE-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen updated CALCITE-3414:
Summary: In calcite-core, use RexToLixTranslator.convert for type
conversion code generation uniformly (was: [CALCITE-3414] In calcite-core, use
RexToLixTranslator.convert for type conversion code generation uniformly)
> In calcite-core, use RexToLixTranslator.convert for type conversion code
> generation uniformly
> -
>
> Key: CALCITE-3414
> URL: https://issues.apache.org/jira/browse/CALCITE-3414
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.21.0
>Reporter: Feng Zhu
>Assignee: Feng Zhu
>Priority: Major
> Labels: pull-request-available
> Attachments: RexToLixTranslator.png, TypeConversion.txt, Types.png
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Current now, there are two functions in calcite that can be used to
> cast/convert Expression to a specific Type.
> *_Types.castIfNecessary_* and _*RexToLixTranslator.convert*_.
> We make a deep investigation on their implementations and demonstrate them as
> below.
> {color:#ff}
> !RexToLixTranslator.png!{color}
> {color:#ff}
> *RexToLixTranslator.convert*{color}
> !Types.png!
> {color:#ff}
> *Types.castIfNecessary*{color}
> It can be seen that:
> (1) They have a lot of overlaps;
> (2) *_RexToLixTranslator.cast_* can cover more cases with tools like
> _SqlFunctions_ and etc.
> (3) Both of them have limitations and may generate incorrect code, which is
> listed in attachment(TypeConversion.txt).
> Multiple choices usually bring confusion to developers and resulting to the
> misuse of them.
> For example, CALCITE-3245 exposes that Types.castIfNecessary cannot cast the
> Expression to BigDecimal.class.
> Fixing the issue in *_Types.castIfNecessary_* directly seems to be not a
> good idea.
> On one hand, it is not convenient to call _SqlFunctions_ in linq4j. One the
> other hand, it will brings duplicate with _*RexToLixTranslator.cast*_.
> However, due to some unique logic in _*Types.castIfNecessary*_, we cannot
> replace it as _*RexToLixTranslator.cast*_ neither.
> Therefore, it is a good idea to integrate implementations into
> RexToLixTranslator.cast.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (CALCITE-3415) Supports REGEXP_SUBSTR function
[
https://issues.apache.org/jira/browse/CALCITE-3415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen reassigned CALCITE-3415:
---
Assignee: Danny Chen
> Supports REGEXP_SUBSTR function
> ---
>
> Key: CALCITE-3415
> URL: https://issues.apache.org/jira/browse/CALCITE-3415
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.21.0
>Reporter: Pranay Parmar
>Assignee: Danny Chen
>Priority: Minor
>
> REGEXP_SUBSTR error :
> {code:java}
> No match found for function signature REGEXP_SUBSTR(, ,
> [, , ]){code}
>
> Example query:
> {code:sql}
> SELECT REGEXP_SUBSTR('chocolate Chip cookies', 'c+.{2}', 1, product_id, 'i')
> FROM public.account{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3293) Add strcmp function
[ https://issues.apache.org/jira/browse/CALCITE-3293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16955816#comment-16955816 ] Danny Chen commented on CALCITE-3293: - I found one in the Oracle doc: https://docs.oracle.com/goldengate/1212/gg-winux/GWURF/column_conversion_functions024.htm#GWURF814 > Add strcmp function > --- > > Key: CALCITE-3293 > URL: https://issues.apache.org/jira/browse/CALCITE-3293 > Project: Calcite > Issue Type: Improvement >Reporter: xzh_dz >Priority: Minor > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > Add strcmp function. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3293) Add strcmp function
[ https://issues.apache.org/jira/browse/CALCITE-3293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16955817#comment-16955817 ] Danny Chen commented on CALCITE-3293: - Sorry, my fault, the Oracle one seems with "@" as prefix. > Add strcmp function > --- > > Key: CALCITE-3293 > URL: https://issues.apache.org/jira/browse/CALCITE-3293 > Project: Calcite > Issue Type: Improvement >Reporter: xzh_dz >Priority: Minor > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > Add strcmp function. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (CALCITE-3416) SQL Dialects "DEFAULT"s should be more extensible
[ https://issues.apache.org/jira/browse/CALCITE-3416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen reassigned CALCITE-3416: --- Assignee: Danny Chen > SQL Dialects "DEFAULT"s should be more extensible > - > > Key: CALCITE-3416 > URL: https://issues.apache.org/jira/browse/CALCITE-3416 > Project: Calcite > Issue Type: Improvement >Reporter: Steven Talbot >Assignee: Danny Chen >Priority: Minor > > The behavior of SQLDialect is partly governed by the methods defined on the > given dialect subclass and partly governed by options passed in in a Context > object. So every dialect subclass exposes a "DEFAULT" instance that has been > initialized with the correct Context. > However, if you then wish to extend the dialect for one reason or another, > you must create a new instance to extend, or of course create a whole new > subclass. In either case, you lose the options from the Context passed into > the default, which governs important behavior of the dialect. You can > copy-paste the relevant Context out of the Calcite code, but then you lose > future improvements or fixes that might land in mainline Calcite. > It would be nice if each dialect exposed the DEFAULT_CONTEXT that it passed > into its DEFAULT instance as a public final member. Then, when extending the > dialect, you simply initialize your extension with the DEFAULT_CONTEXT, and > if any customization needs to happens on the Context options that's easy to > do with the Context's API. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3416) SQL Dialects "DEFAULT"s should be more extensible
[ https://issues.apache.org/jira/browse/CALCITE-3416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956586#comment-16956586 ] Danny Chen commented on CALCITE-3416: - Thanks, i kindly agree, how about we move these contexts into each sql dialect as a protected final one ? > SQL Dialects "DEFAULT"s should be more extensible > - > > Key: CALCITE-3416 > URL: https://issues.apache.org/jira/browse/CALCITE-3416 > Project: Calcite > Issue Type: Improvement >Reporter: Steven Talbot >Assignee: Danny Chen >Priority: Minor > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > The behavior of SQLDialect is partly governed by the methods defined on the > given dialect subclass and partly governed by options passed in in a Context > object. So every dialect subclass exposes a "DEFAULT" instance that has been > initialized with the correct Context. > However, if you then wish to extend the dialect for one reason or another, > you must create a new instance to extend, or of course create a whole new > subclass. In either case, you lose the options from the Context passed into > the default, which governs important behavior of the dialect. You can > copy-paste the relevant Context out of the Calcite code, but then you lose > future improvements or fixes that might land in mainline Calcite. > It would be nice if each dialect exposed the DEFAULT_CONTEXT that it passed > into its DEFAULT instance as a public final member. Then, when extending the > dialect, you simply initialize your extension with the DEFAULT_CONTEXT, and > if any customization needs to happens on the Context options that's easy to > do with the Context's API. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3416) SQL Dialects "DEFAULT"s should be more extensible
[ https://issues.apache.org/jira/browse/CALCITE-3416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956650#comment-16956650 ] Danny Chen commented on CALCITE-3416: - The constructor has already required an argument of Context, so i have no choice to create the context inside the constructor. I choose to make the context public static and named "DEFAULT_CONTEXT", the "DEFAULT" instance of each SqlDialect reference and create based on the "DEFAULT_CONTEXT". > SQL Dialects "DEFAULT"s should be more extensible > - > > Key: CALCITE-3416 > URL: https://issues.apache.org/jira/browse/CALCITE-3416 > Project: Calcite > Issue Type: Improvement >Reporter: Steven Talbot >Assignee: Danny Chen >Priority: Minor > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > The behavior of SQLDialect is partly governed by the methods defined on the > given dialect subclass and partly governed by options passed in in a Context > object. So every dialect subclass exposes a "DEFAULT" instance that has been > initialized with the correct Context. > However, if you then wish to extend the dialect for one reason or another, > you must create a new instance to extend, or of course create a whole new > subclass. In either case, you lose the options from the Context passed into > the default, which governs important behavior of the dialect. You can > copy-paste the relevant Context out of the Calcite code, but then you lose > future improvements or fixes that might land in mainline Calcite. > It would be nice if each dialect exposed the DEFAULT_CONTEXT that it passed > into its DEFAULT instance as a public final member. Then, when extending the > dialect, you simply initialize your extension with the DEFAULT_CONTEXT, and > if any customization needs to happens on the Context options that's easy to > do with the Context's API. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CALCITE-3393) RelStructuredTypeFlattener: improve support for functions with struct input
[
https://issues.apache.org/jira/browse/CALCITE-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen resolved CALCITE-3393.
-
Fix Version/s: 1.22.0
Resolution: Fixed
Fixed in
[0d1d491|https://github.com/apache/calcite/commit/0d1d4915f6b2ce56a6d4b570d352432f1cb3fe1d],
thanks for your PR [~IhorHuzenko] !
> RelStructuredTypeFlattener: improve support for functions with struct input
> ---
>
> Key: CALCITE-3393
> URL: https://issues.apache.org/jira/browse/CALCITE-3393
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Igor Guzenko
>Assignee: Igor Guzenko
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.22.0
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> 1.
> [RelStructuredTypeFlattener|https://github.com/apache/calcite/blob/148bfd329413c0272395cc0b7c322b3c5a34b667/core/src/main/java/org/apache/calcite/sql2rel/RelStructuredTypeFlattener.java#L376]
> doesn't support aggregate functions for struct type column.
> Example test case:
> {code:java}
> @Test
> public void testAggregateFunctionWithStructInput() {
> final String sql =
> "select count(dn.skill) from sales.dept_nested dn";
> sql(sql).ok();
> }
> {code}
> 2. For other functions, except ITEM, flattener uses first nested primitive
> field from
> original struct input. Example test case:
> {code:java}
> @Test
> public void testFunctionWithStructInput() {
> final String sql =
> "select json_type(dn.skill) from sales.dept_nested dn";
> sql(sql).ok();
> }
> {code}
> Generated plan:
> {code}
> LogicalProject(EXPR$0=[JSON_TYPE($2.TYPE)])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT_NESTED]])
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3402) Allow RANGE with compoud ORDER BY clause
[
https://issues.apache.org/jira/browse/CALCITE-3402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956862#comment-16956862
]
Danny Chen commented on CALCITE-3402:
-
It's not surprise the cases you gave for RANGE and ROWS output differently.
Because RANGE filter columns by the column value range (for your example it's
[-1, 1]) but ROWS filter columns by the row numbers(for your case, always the 3
rows with current row in the middle).
> Allow RANGE with compoud ORDER BY clause
>
>
> Key: CALCITE-3402
> URL: https://issues.apache.org/jira/browse/CALCITE-3402
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.18.0, 1.19.0
>Reporter: benj
>Priority: Major
>
> It will be very useful to have the capacity to use compound ORDER BY clause
> with RANGE
> {code:sql}
> apache drill (dfs.tmp)> SELECT a
> , last_value(c) OVER(PARTITION BY a ORDER BY c, b DESC RANGE BETWEEN
> UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING)
> FROM (SELECT 1 a, 'b' b, 3 c
> UNION SELECT 2, 'c', 4
> UNION SELECT 1, 'c', 4
> /* UNION ... */
> ) x;
> Error: VALIDATION ERROR: From line 2, column 56 to line 2, column 60: RANGE
> clause cannot be used with compound ORDER BY clause
> {code}
> I know it's possible (for last_value) to rewrite with first_value with an
> reverse ORDER BY and without RANGE to obtain correct result.
> But it will become sometimes less readable and request write from other SGBDR
> will not be compatible and should be rewrite, and for some other function
> than last_value, the problem will not be solved like that.
> compound ORDER BY clause with RANGE is possible with some SGBDR like
> Postgres:
> [https://www.postgresql.org/docs/9.3/sql-expressions.html#SYNTAX-WINDOW-FUNCTIONS]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3282) HiveSqlDialect should unparse INTEGER type as INT in order to be compatible with Hive1.x
[ https://issues.apache.org/jira/browse/CALCITE-3282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16957480#comment-16957480 ] Danny Chen commented on CALCITE-3282: - Thanks for that, Julian ! > HiveSqlDialect should unparse INTEGER type as INT in order to be compatible > with Hive1.x > > > Key: CALCITE-3282 > URL: https://issues.apache.org/jira/browse/CALCITE-3282 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: feng huang >Assignee: Danny Chen >Priority: Minor > Labels: pull-request-available > Fix For: 1.22.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > > Every database might have different type or same type but different type > name, therefore making every SqlDialect unparse their own data type is a > suitable way. > For example, there is a sql “select cast(col as int) from table” change to > hive sql "select cast(col as integer) from table", but "integer" is not > allowed in hive. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CALCITE-3416) SQL Dialects "DEFAULT"s should be more extensible
[ https://issues.apache.org/jira/browse/CALCITE-3416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen resolved CALCITE-3416. - Fix Version/s: 1.22.0 Resolution: Fixed Fixed in [ff43dcb|https://github.com/apache/calcite/commit/ff43dcb95caa251e5fc8120980fc70c8fea8ac40] ! > SQL Dialects "DEFAULT"s should be more extensible > - > > Key: CALCITE-3416 > URL: https://issues.apache.org/jira/browse/CALCITE-3416 > Project: Calcite > Issue Type: Improvement >Reporter: Steven Talbot >Assignee: Danny Chen >Priority: Minor > Labels: pull-request-available > Fix For: 1.22.0 > > Time Spent: 40m > Remaining Estimate: 0h > > The behavior of SQLDialect is partly governed by the methods defined on the > given dialect subclass and partly governed by options passed in in a Context > object. So every dialect subclass exposes a "DEFAULT" instance that has been > initialized with the correct Context. > However, if you then wish to extend the dialect for one reason or another, > you must create a new instance to extend, or of course create a whole new > subclass. In either case, you lose the options from the Context passed into > the default, which governs important behavior of the dialect. You can > copy-paste the relevant Context out of the Calcite code, but then you lose > future improvements or fixes that might land in mainline Calcite. > It would be nice if each dialect exposed the DEFAULT_CONTEXT that it passed > into its DEFAULT instance as a public final member. Then, when extending the > dialect, you simply initialize your extension with the DEFAULT_CONTEXT, and > if any customization needs to happens on the Context options that's easy to > do with the Context's API. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CALCITE-3414) In calcite-core, use RexToLixTranslator.convert for type conversion code generation uniformly
[
https://issues.apache.org/jira/browse/CALCITE-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen resolved CALCITE-3414.
-
Fix Version/s: 1.22.0
Resolution: Fixed
Fixed in
[429c0d0|https://github.com/apache/calcite/commit/429c0d0e4cacc76933ae6cc651e5e4164f868383],
thanks for your PR, [~donnyzone] !
> In calcite-core, use RexToLixTranslator.convert for type conversion code
> generation uniformly
> -
>
> Key: CALCITE-3414
> URL: https://issues.apache.org/jira/browse/CALCITE-3414
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.21.0
>Reporter: Feng Zhu
>Assignee: Feng Zhu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.22.0
>
> Attachments: RexToLixTranslator.png, TypeConversion.txt, Types.png
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Current now, there are two functions in calcite that can be used to
> cast/convert Expression to a specific Type.
> *_Types.castIfNecessary_* and _*RexToLixTranslator.convert*_.
> We make a deep investigation on their implementations and demonstrate them as
> below.
> {color:#ff}
> !RexToLixTranslator.png!{color}
> {color:#ff}
> *RexToLixTranslator.convert*{color}
> !Types.png!
> {color:#ff}
> *Types.castIfNecessary*{color}
> It can be seen that:
> (1) They have a lot of overlaps;
> (2) *_RexToLixTranslator.cast_* can cover more cases with tools like
> _SqlFunctions_ and etc.
> (3) Both of them have limitations and may generate incorrect code, which is
> listed in attachment(TypeConversion.txt).
> Multiple choices usually bring confusion to developers and resulting to the
> misuse of them.
> For example, CALCITE-3245 exposes that Types.castIfNecessary cannot cast the
> Expression to BigDecimal.class.
> Fixing the issue in *_Types.castIfNecessary_* directly seems to be not a
> good idea.
> On one hand, it is not convenient to call _SqlFunctions_ in linq4j. One the
> other hand, it will brings duplicate with _*RexToLixTranslator.cast*_.
> However, due to some unique logic in _*Types.castIfNecessary*_, we cannot
> replace it as _*RexToLixTranslator.cast*_ neither.
> Therefore, it is a good idea to integrate implementations into
> RexToLixTranslator.cast.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (CALCITE-3300) Add a star identifier as operand to "SqlCountAggFunction" if no operands given while in the method "createCall"
[
https://issues.apache.org/jira/browse/CALCITE-3300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen reassigned CALCITE-3300:
---
Assignee: Danny Chen (was: Wang Weidong)
> Add a star identifier as operand to "SqlCountAggFunction" if no operands
> given while in the method "createCall"
> ---
>
> Key: CALCITE-3300
> URL: https://issues.apache.org/jira/browse/CALCITE-3300
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.21.0
>Reporter: Wang Weidong
>Assignee: Danny Chen
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> h4. Procedure to reproduce the problem:
> - create table {{~^t^~}}
> - parse query select count(*) from t to SqlNode
> - convert SqlNode to RelNode
> - convert RelNode to SqlNode
> - validate the result SqlNode
> test code is like this
> {code:java}
> // code placeholder
> @Test
> public void testSelectCountAll() throws Exception {
> try (Statement s = parserContext.getStatement()) {
> s.execute("create table if not exists t (i int not null)");
> String sql = "select count(*) from t";
> SqlNode sqlNode = parserContext.parseStmt(sql);
> parserContext.getSqlValidator().validate(sqlNode);
> RelNode relNode =
> parserContext.getSqlToRelConverter().convertQuery(sqlNode, true, true).rel;
> SqlNode sqlNodeNew = toSqlNode(relNode);
> parserContext.getSqlValidator().validate(sqlNodeNew);
> }
> }
> {code}
> Finally we will get an exception as follow:
> {code:java}
> // code placeholder
> org.apache.calcite.runtime.CalciteContextException: At line 0, column 0:
> Invalid number of arguments to function 'COUNT'. Was expecting 1
> argumentsorg.apache.calcite.runtime.CalciteContextException: At line 0,
> column 0: Invalid number of arguments to function 'COUNT'. Was expecting 1
> arguments at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method) at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at
> org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:463)
> at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:783) at
> org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:768) at
> org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError(SqlValidatorImpl.java:4753)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.handleUnresolvedFunction(SqlValidatorImpl.java:1699)
> at org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:270) at
> org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:215) at
> org.apache.calcite.sql.fun.SqlCountAggFunction.deriveType(SqlCountAggFunction.java:83)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:5477)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:5464)
> at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:138) at
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1629)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1614)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.expandSelectItem(SqlValidatorImpl.java:457)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelectList(SqlValidatorImpl.java:4017)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3280)
> at
> org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:967)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:943)
> at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:225) at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:918)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:628)
> at
> org.apache.calcite.test.OptimizeTest.testSelectCountAll(OptimizeTest.java:220)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Met
[jira] [Resolved] (CALCITE-3300) Add a star identifier as operand to "SqlCountAggFunction" if no operands given while in the method "createCall"
[
https://issues.apache.org/jira/browse/CALCITE-3300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen resolved CALCITE-3300.
-
Fix Version/s: 1.22.0
Resolution: Fixed
Fixed in
[b5ffef1|https://github.com/apache/calcite/commit/b5ffef1c05bfc35dba349ffe4874de61aa15ae0e],
thanks for your PR, [~wangweidong] !
> Add a star identifier as operand to "SqlCountAggFunction" if no operands
> given while in the method "createCall"
> ---
>
> Key: CALCITE-3300
> URL: https://issues.apache.org/jira/browse/CALCITE-3300
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.21.0
>Reporter: Wang Weidong
>Assignee: Danny Chen
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.22.0
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> h4. Procedure to reproduce the problem:
> - create table {{~^t^~}}
> - parse query select count(*) from t to SqlNode
> - convert SqlNode to RelNode
> - convert RelNode to SqlNode
> - validate the result SqlNode
> test code is like this
> {code:java}
> // code placeholder
> @Test
> public void testSelectCountAll() throws Exception {
> try (Statement s = parserContext.getStatement()) {
> s.execute("create table if not exists t (i int not null)");
> String sql = "select count(*) from t";
> SqlNode sqlNode = parserContext.parseStmt(sql);
> parserContext.getSqlValidator().validate(sqlNode);
> RelNode relNode =
> parserContext.getSqlToRelConverter().convertQuery(sqlNode, true, true).rel;
> SqlNode sqlNodeNew = toSqlNode(relNode);
> parserContext.getSqlValidator().validate(sqlNodeNew);
> }
> }
> {code}
> Finally we will get an exception as follow:
> {code:java}
> // code placeholder
> org.apache.calcite.runtime.CalciteContextException: At line 0, column 0:
> Invalid number of arguments to function 'COUNT'. Was expecting 1
> argumentsorg.apache.calcite.runtime.CalciteContextException: At line 0,
> column 0: Invalid number of arguments to function 'COUNT'. Was expecting 1
> arguments at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method) at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at
> org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:463)
> at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:783) at
> org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:768) at
> org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError(SqlValidatorImpl.java:4753)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.handleUnresolvedFunction(SqlValidatorImpl.java:1699)
> at org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:270) at
> org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:215) at
> org.apache.calcite.sql.fun.SqlCountAggFunction.deriveType(SqlCountAggFunction.java:83)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:5477)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:5464)
> at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:138) at
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1629)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1614)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.expandSelectItem(SqlValidatorImpl.java:457)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelectList(SqlValidatorImpl.java:4017)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3280)
> at
> org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:967)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:943)
> at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:225) at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:918)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:628)
> at
> org.apache.calcite.test.OptimizeTest.testSelectCountAll(OptimizeTest.java:220)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAc
[jira] [Updated] (CALCITE-3300) In JDBC adapter, when generating SQL for count star, generates the star argument of the call
[
https://issues.apache.org/jira/browse/CALCITE-3300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen updated CALCITE-3300:
Summary: In JDBC adapter, when generating SQL for count star, generates the
star argument of the call (was: Add a star identifier as operand to
"SqlCountAggFunction" if no operands given while in the method "createCall")
> In JDBC adapter, when generating SQL for count star, generates the star
> argument of the call
>
>
> Key: CALCITE-3300
> URL: https://issues.apache.org/jira/browse/CALCITE-3300
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.21.0
>Reporter: Wang Weidong
>Assignee: Danny Chen
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.22.0
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> h4. Procedure to reproduce the problem:
> - create table {{~^t^~}}
> - parse query select count(*) from t to SqlNode
> - convert SqlNode to RelNode
> - convert RelNode to SqlNode
> - validate the result SqlNode
> test code is like this
> {code:java}
> // code placeholder
> @Test
> public void testSelectCountAll() throws Exception {
> try (Statement s = parserContext.getStatement()) {
> s.execute("create table if not exists t (i int not null)");
> String sql = "select count(*) from t";
> SqlNode sqlNode = parserContext.parseStmt(sql);
> parserContext.getSqlValidator().validate(sqlNode);
> RelNode relNode =
> parserContext.getSqlToRelConverter().convertQuery(sqlNode, true, true).rel;
> SqlNode sqlNodeNew = toSqlNode(relNode);
> parserContext.getSqlValidator().validate(sqlNodeNew);
> }
> }
> {code}
> Finally we will get an exception as follow:
> {code:java}
> // code placeholder
> org.apache.calcite.runtime.CalciteContextException: At line 0, column 0:
> Invalid number of arguments to function 'COUNT'. Was expecting 1
> argumentsorg.apache.calcite.runtime.CalciteContextException: At line 0,
> column 0: Invalid number of arguments to function 'COUNT'. Was expecting 1
> arguments at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method) at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at
> org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:463)
> at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:783) at
> org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:768) at
> org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError(SqlValidatorImpl.java:4753)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.handleUnresolvedFunction(SqlValidatorImpl.java:1699)
> at org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:270) at
> org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:215) at
> org.apache.calcite.sql.fun.SqlCountAggFunction.deriveType(SqlCountAggFunction.java:83)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:5477)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:5464)
> at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:138) at
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1629)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1614)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.expandSelectItem(SqlValidatorImpl.java:457)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelectList(SqlValidatorImpl.java:4017)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3280)
> at
> org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:967)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:943)
> at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:225) at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:918)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:628)
> at
> org.apache.calcite.test.OptimizeTest.testSelectCountAll(OptimizeTest.java:220)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccesso
[jira] [Commented] (CALCITE-3368) PLUS, MINUS and TIMES should be unsafe when simplifying ‘expression IS NULL’
[
https://issues.apache.org/jira/browse/CALCITE-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16957811#comment-16957811
]
Danny Chen commented on CALCITE-3368:
-
I tried these sql:
{code:java}
create table t(
a int,
b int
);
insert into t values(2147483647, 2147483647);
select (a + b) is null from t; // outputs 0 for MySQL 5.6, out of range for
PostgreSQL 9.6 and MS-SQL 2017
{code}
So it seems that MySQL are more lenient for the semantic which is same with
what Calcite does now.
Julian, do you think we should follow the PostgreSQL style ?
> PLUS, MINUS and TIMES should be unsafe when simplifying ‘expression IS NULL’
>
>
> Key: CALCITE-3368
> URL: https://issues.apache.org/jira/browse/CALCITE-3368
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.21.0
>Reporter: Leonard Xu
>Assignee: Leonard Xu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> 'is null' expression in SQL may be optimized incorrectly in the underlying
> implementation.
>
> When I write a Fink SQL to test overflow just like
> {code:java}
> select
>case when (f0 + f1) is null then 'null' else 'not null' end
> from testTable
> {code}
> , I found expression '(f0 + f1) is null ' has been optimized by Calcite, and
> the optimization may be incorrect.
>
> The underlying implementation is that Calcite's simplification logic of
> isNull expression in SQL will convert from
> *"f(operand0, operand1) IS NULL"* to
> *"operand0 IS NULL OR operand1 IS NULL"* if the Policy of RexNode‘s SqlKind
> is ANY。
> This simplification leads to the expression will not calculate the real
> value of *f(operand0, operand1)* (eg.. '(f0 + f1)' in my case ），but '(f0 +
> f1)' maybe overflows after operation.
> {code:java}
> //org.apache.calcite.rex.RexSimplify.java
> private RexNode simplifyIsNull(RexNode a) {
> // Simplify the argument first,
> // call ourselves recursively to see whether we can make more progress.
> // For example, given
> // "(CASE WHEN FALSE THEN 1 ELSE 2) IS NULL" we first simplify the
> // argument to "2", and only then we can simplify "2 IS NULL" to "FALSE".
> a = simplify(a, UNKNOWN);
> if (!a.getType().isNullable() && isSafeExpression(a)) {
> return rexBuilder.makeLiteral(false);
> }
> if (RexUtil.isNull(a)) {
> return rexBuilder.makeLiteral(true);
> }
> if (a.getKind() == SqlKind.CAST) {
> return null;
> }
> switch (Strong.policy(a.getKind())) {
> case NOT_NULL:
> return rexBuilder.makeLiteral(false);
> case ANY:
> // "f" is a strong operator, so "f(operand0, operand1) IS NULL" simplifies
> // to "operand0 IS NULL OR operand1 IS NULL"
> final List operands = new ArrayList<>();
> for (RexNode operand : ((RexCall) a).getOperands()) {
> final RexNode simplified = simplifyIsNull(operand);
> if (simplified == null) {
> operands.add(
> rexBuilder.makeCall(SqlStdOperatorTable.IS_NULL, operand));
> } else {
> operands.add(simplified);
> }
> }
> return RexUtil.composeDisjunction(rexBuilder, operands, false);
> case AS_IS:
> default:
> return null;
> }
> }{code}
> And most of calculating SqlKinds are assigned *Policy.ANY* at present.
> {code:java}
> //org.apache.calcite.plan.Strong.java
> public static Policy policy(SqlKind kind) {
> return MAP.getOrDefault(kind, Policy.AS_IS);
> }
>
> map.put(SqlKind.PLUS, Policy.ANY);
> map.put(SqlKind.PLUS_PREFIX, Policy.ANY);
> map.put(SqlKind.MINUS, Policy.ANY);
> map.put(SqlKind.MINUS_PREFIX, Policy.ANY);
> map.put(SqlKind.TIMES, Policy.ANY);
> map.put(SqlKind.DIVIDE, Policy.ANY);
> * that operator evaluates to null. */
> public enum Policy {
> /** This kind of expression is never null. No need to look at its arguments,
>* if it has any. */
> NOT_NULL,
> /** This kind of expression has its own particular rules about whether it
>* is null. */
> CUSTOM,
> /** This kind of expression is null if and only if at least one of its
>* arguments is null. */
> ANY,
> /** This kind of expression may be null. There is no way to rewrite. */
> AS_IS,
> }{code}
>
> It may be an obvious nonequivalent simplification in SQL. And this issue come
> from Flink （FLINK-14030）.
> [~danny0405]， Could you have a look at this?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-3424) AssertionError thrown for user-defined table function with array argument
[
https://issues.apache.org/jira/browse/CALCITE-3424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen resolved CALCITE-3424.
-
Fix Version/s: 1.22.0
Resolution: Fixed
Fixed in
[8ba2d85|https://github.com/apache/calcite/commit/8ba2d8512b8f195bb0866af12308beef2b3eed9f],
thanks for your PR, [~IhorHuzenko] !
> AssertionError thrown for user-defined table function with array argument
> -
>
> Key: CALCITE-3424
> URL: https://issues.apache.org/jira/browse/CALCITE-3424
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.21.0
>Reporter: Igor Guzenko
>Assignee: Igor Guzenko
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.22.0
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> *Steps to reproduce:*
> *1.* Add method with list parameter to Smalls.java
> {code:java}
> public static final Method GENERATE_STRINGS_2_METHOD =
> Types.lookupMethod(Smalls.class, "generateStrings2", List.class);
> public static QueryableTable generateStrings2(final List list) {
> return generateStrings(list.size());
> }
> {code}
> *2.* Add test method which uses new user-defined table function to
> TableFunctionTest.java
> {code:java}
> @Test public void testTableFunction2() throws SQLException {
> try (Connection connection =
> DriverManager.getConnection("jdbc:calcite:")) {
> CalciteConnection calciteConnection =
> connection.unwrap(CalciteConnection.class);
> SchemaPlus rootSchema = calciteConnection.getRootSchema();
> SchemaPlus schema = rootSchema.add("s", new AbstractSchema());
> final TableFunction table =
> TableFunctionImpl.create(Smalls.GENERATE_STRINGS_2_METHOD);
> schema.add("GenerateStrings2", table);
> final String sql = "select *\n"
> + "from table(\"s\".\"GenerateStrings2\"(5,4,3,1,2)) as t(n, c)\n"
> + "where char_length(c) > 3";
> ResultSet resultSet = connection.createStatement().executeQuery(sql);
> assertThat(CalciteAssert.toString(resultSet),
> equalTo("N=4; C=abcd\n"));
> }
> }
> {code}
> Execution result produced by such test method is the following stack trace:
> {code:none}
> java.lang.AssertionError: use createArrayType() instead
> at
> org.apache.calcite.sql.type.SqlTypeFactoryImpl.assertBasic(SqlTypeFactoryImpl.java:221)
> at
> org.apache.calcite.sql.type.SqlTypeFactoryImpl.createSqlType(SqlTypeFactoryImpl.java:48)
> at
> org.apache.calcite.jdbc.JavaTypeFactoryImpl.toSql(JavaTypeFactoryImpl.java:255)
> at
> org.apache.calcite.prepare.CalciteCatalogReader.toSql(CalciteCatalogReader.java:381)
> at
> org.apache.calcite.prepare.CalciteCatalogReader.lambda$toSql$7(CalciteCatalogReader.java:370)
> at
> com.google.common.collect.Lists$TransformingRandomAccessList$1.transform(Lists.java:640)
> at
> com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
> at java.util.AbstractCollection.toArray(AbstractCollection.java:141)
> at
> com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:239)
> at org.apache.calcite.sql.SqlFunction.(SqlFunction.java:123)
> at
> org.apache.calcite.sql.validate.SqlUserDefinedFunction.(SqlUserDefinedFunction.java:63)
> at
> org.apache.calcite.sql.validate.SqlUserDefinedTableFunction.(SqlUserDefinedTableFunction.java:45)
> at
> org.apache.calcite.prepare.CalciteCatalogReader.toOp(CalciteCatalogReader.java:338)
> at
> org.apache.calcite.prepare.CalciteCatalogReader.toOp(CalciteCatalogReader.java:302)
> at
> org.apache.calcite.prepare.CalciteCatalogReader.lambda$lookupOperatorOverloads$3(CalciteCatalogReader.java:271)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at
> java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
> at
> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> at
> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.forEachOrdered(ReferencePipeline.java:490)
> at
> org.apache.calcite.prepare.CalciteCatalogReader.lookupOperatorOverloads(CalciteCatalogReader.java:272)
> at
> org.apache.calcite.sql.util.ChainedSqlOperatorTable.lookupOperatorOverloa
[jira] [Created] (CALCITE-3441) Remove SqlTypeExplicitPrecedenceList.COMPACT_NUMERIC_TYPES because the NULL delimiters are useless
Danny Chen created CALCITE-3441: --- Summary: Remove SqlTypeExplicitPrecedenceList.COMPACT_NUMERIC_TYPES because the NULL delimiters are useless Key: CALCITE-3441 URL: https://issues.apache.org/jira/browse/CALCITE-3441 Project: Calcite Issue Type: Improvement Components: core Affects Versions: 1.21.0 Reporter: Danny Chen Assignee: Danny Chen Fix For: 1.22.0 We do not need the "null" value as delimiter for the equivalence class because method Util.skip already did that. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CALCITE-3441) Remove SqlTypeExplicitPrecedenceList.COMPACT_NUMERIC_TYPES because the NULL delimiters are useless
[ https://issues.apache.org/jira/browse/CALCITE-3441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen resolved CALCITE-3441. - Resolution: Fixed Fixed in [d287b05|https://github.com/apache/calcite/commit/d287b0551a189fd1bf1f8a1ae3afb1b446823350] ! > Remove SqlTypeExplicitPrecedenceList.COMPACT_NUMERIC_TYPES because the NULL > delimiters are useless > -- > > Key: CALCITE-3441 > URL: https://issues.apache.org/jira/browse/CALCITE-3441 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.21.0 >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Minor > Labels: pull-request-available > Fix For: 1.22.0 > > Time Spent: 20m > Remaining Estimate: 0h > > We do not need the "null" value as delimiter for the equivalence class > because method Util.skip already did that. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-3446) Make RelMetadataQuery extendible
Danny Chen created CALCITE-3446: --- Summary: Make RelMetadataQuery extendible Key: CALCITE-3446 URL: https://issues.apache.org/jira/browse/CALCITE-3446 Project: Calcite Issue Type: Improvement Components: core Affects Versions: 1.21.0 Reporter: Danny Chen Assignee: Danny Chen Fix For: 1.22.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-3446) Make RelMetadataQuery extendible
[ https://issues.apache.org/jira/browse/CALCITE-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated CALCITE-3446: Description: As discussed in the DEV mailing list: https://lists.apache.org/list.html?d...@calcite.apache.org:lte=1M:%5BDISCUSSION%5D%20Extension%20of%20Metadata%20Query, we have made conclusion that the user can sub-class the RelMetadataQuery. > Make RelMetadataQuery extendible > > > Key: CALCITE-3446 > URL: https://issues.apache.org/jira/browse/CALCITE-3446 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.21.0 >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Fix For: 1.22.0 > > > As discussed in the DEV mailing list: > https://lists.apache.org/list.html?d...@calcite.apache.org:lte=1M:%5BDISCUSSION%5D%20Extension%20of%20Metadata%20Query, > we have made conclusion that the user can sub-class the RelMetadataQuery. -- This message was sent by Atlassian Jira (v8.3.4#803005)
