[jira] [Comment Edited] (CALCITE-2620) Simplify "coalesce(null,x)" to "x"
[ https://issues.apache.org/jira/browse/CALCITE-2620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648733#comment-16648733 ] pengzhiwei edited comment on CALCITE-2620 at 10/13/18 4:10 AM: --- [~julianhyde],In -CALCITE-2506- this issue has not fixed. I fix it in the [PR882|https://github.com/apache/calcite/pull/882] ,Please take a look. was (Author: pzw2018): [~julianhyde],In -CALCITE-2506- this issue has not fixed. I fix it in the [PR882|https://github.com/apache/calcite/pull/882] > Simplify "coalesce(null,x)" to "x" > -- > > Key: CALCITE-2620 > URL: https://issues.apache.org/jira/browse/CALCITE-2620 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.18.0 >Reporter: pengzhiwei >Assignee: Julian Hyde >Priority: Major > > "coalesce(x0,x1,.)" can be simplified if any of "xi" is null literal. > e.g. "coalesce(null,x)" can simplify to "x", "coalesce(x,null,y)" can > simplify to "coalesce(x,y)" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (CALCITE-2620) Simplify "coalesce(null,x)" to "x"
[ https://issues.apache.org/jira/browse/CALCITE-2620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648733#comment-16648733 ] pengzhiwei edited comment on CALCITE-2620 at 10/13/18 4:09 AM: --- [~julianhyde],In -CALCITE-2506- this issue has not fixed. I fix it in the [PR882|https://github.com/apache/calcite/pull/882] was (Author: pzw2018): [~julianhyde],In -CALCITE-2506- this issue has not fixed. I fix it in the [PR882|https://github.com/apache/calcite/pull/882] > Simplify "coalesce(null,x)" to "x" > -- > > Key: CALCITE-2620 > URL: https://issues.apache.org/jira/browse/CALCITE-2620 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.18.0 >Reporter: pengzhiwei >Assignee: Julian Hyde >Priority: Major > > "coalesce(x0,x1,.)" can be simplified if any of "xi" is null literal. > e.g. "coalesce(null,x)" can simplify to "x", "coalesce(x,null,y)" can > simplify to "coalesce(x,y)" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CALCITE-2620) Simplify "coalesce(null,x)" to "x"
[ https://issues.apache.org/jira/browse/CALCITE-2620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648733#comment-16648733 ] pengzhiwei commented on CALCITE-2620: - [~julianhyde],In -CALCITE-2506- this issue has not fixed. I fix it in the [PR882|[http://example.com|https://github.com/apache/calcite/pull/882]] > Simplify "coalesce(null,x)" to "x" > -- > > Key: CALCITE-2620 > URL: https://issues.apache.org/jira/browse/CALCITE-2620 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.18.0 >Reporter: pengzhiwei >Assignee: Julian Hyde >Priority: Major > > "coalesce(x0,x1,.)" can be simplified if any of "xi" is null literal. > e.g. "coalesce(null,x)" can simplify to "x", "coalesce(x,null,y)" can > simplify to "coalesce(x,y)" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (CALCITE-2620) Simplify "coalesce(null,x)" to "x"
[ https://issues.apache.org/jira/browse/CALCITE-2620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648733#comment-16648733 ] pengzhiwei edited comment on CALCITE-2620 at 10/13/18 4:09 AM: --- [~julianhyde],In -CALCITE-2506- this issue has not fixed. I fix it in the [PR882|https://github.com/apache/calcite/pull/882] was (Author: pzw2018): [~julianhyde],In -CALCITE-2506- this issue has not fixed. I fix it in the [PR882|[http://example.com|https://github.com/apache/calcite/pull/882]] > Simplify "coalesce(null,x)" to "x" > -- > > Key: CALCITE-2620 > URL: https://issues.apache.org/jira/browse/CALCITE-2620 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.18.0 >Reporter: pengzhiwei >Assignee: Julian Hyde >Priority: Major > > "coalesce(x0,x1,.)" can be simplified if any of "xi" is null literal. > e.g. "coalesce(null,x)" can simplify to "x", "coalesce(x,null,y)" can > simplify to "coalesce(x,y)" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CALCITE-2620) Simplify "coalesce(null,x)" to "x"
[
https://issues.apache.org/jira/browse/CALCITE-2620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648648#comment-16648648
]
Julian Hyde commented on CALCITE-2620:
--
In the fix to CALCITE-2506 I see {code}checkSimplify2(coalesce(gt(nullInt,
nullInt), trueLiteral),
"COALESCE(null, true)", "COALESCE(null, true)");{code}
The fact that "COALESCE(null, true)" remains in the simplified result, rather
than having been further simplified to "true", implies that this bug is not
fixed.
> Simplify "coalesce(null,x)" to "x"
> --
>
> Key: CALCITE-2620
> URL: https://issues.apache.org/jira/browse/CALCITE-2620
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.18.0
>Reporter: pengzhiwei
>Assignee: Julian Hyde
>Priority: Major
>
> "coalesce(x0,x1,.)" can be simplified if any of "xi" is null literal.
> e.g. "coalesce(null,x)" can simplify to "x", "coalesce(x,null,y)" can
> simplify to "coalesce(x,y)"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-525) Exception-handling in built-in functions
[
https://issues.apache.org/jira/browse/CALCITE-525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648644#comment-16648644
]
Julian Hyde commented on CALCITE-525:
-
If consistency is paramount for someone, they can continue to run in "fail
everything" mode. It will remain the default.
But we should do this feature, because the ability to ignore errors improves
usability. That's why it is in every major DBMS.
> Exception-handling in built-in functions
>
>
> Key: CALCITE-525
> URL: https://issues.apache.org/jira/browse/CALCITE-525
> Project: Calcite
> Issue Type: Bug
>Reporter: Julian Hyde
>Assignee: Hongze Zhang
>Priority: Major
>
> The standard calls for certain built-in functions to throw exceptions.
> Examples:
> * 1 / 0
> * MOD(1, 0)
> * OVERLAY('foo' PLACING 'x' FROM -1)
> * 'x' NOT LIKE 'x' ESCAPE 'x'
> First, these exceptions should occur at run time. They should cause the
> current value to become null, or the row to be omitted, but should not abort
> the query. (Actual behavior TBD.)
> Second, EnumerableCalc does constant reduction and generates code like
> 'static final int X = 0 / 0'. This code blows up when the class is loaded. It
> should not. The code should give errors for each row, as described above.
> While fixing this bug, see SqlOperatorBaseTest.testArgumentBounds and remove
> restrictions related to /, MOD and OVERLAY, LIKE.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-1413) New CASE statement simplification
[
https://issues.apache.org/jira/browse/CALCITE-1413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648641#comment-16648641
]
Julian Hyde commented on CALCITE-1413:
--
I'll let [~jcamachorodriguez] review this. Two minor things:
* Be sure to use 2/4 indentation. You use 4/8 in some places.
* The fields in CaseBranch should be final. Or you can make it extend
Pair.
> New CASE statement simplification
> -
>
> Key: CALCITE-1413
> URL: https://issues.apache.org/jira/browse/CALCITE-1413
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.9.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Zoltan Haindrich
>Priority: Major
>
> Another possible simplification for CASE statements that return boolean
> values.
> {code:sql}
> CASE
> WHEN p1 THEN e1
> WHEN p2 THEN e2
> ELSE TRUE
> END
> {code}
> where e1...en are boolean expressions, can be rewritten into:
> {code:sql}
> (p1 and e1) or (p2 and e2 and not(p1)) or (not(p1) and not(p2))
> {code}
> if p1...pn cannot be nullable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-2617) FilterProjectTransposeRule should allow filter conditions with correlated variables to be pushed down
[ https://issues.apache.org/jira/browse/CALCITE-2617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648634#comment-16648634 ] Julian Hyde commented on CALCITE-2617: -- Reviewing the PR: * I don't see any need for a new constructor (especially one with no javadoc) comments. * I don't think see a need for HAS_CORRELATION to be a field (especially when its javadoc comment is the opposite of what it actually does). Just use a lambda expression. > FilterProjectTransposeRule should allow filter conditions with correlated > variables to be pushed down > - > > Key: CALCITE-2617 > URL: https://issues.apache.org/jira/browse/CALCITE-2617 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.17.0 >Reporter: Stamatis Zampetakis >Assignee: Julian Hyde >Priority: Major > Fix For: 1.18.0 > > > The rule always forbids conditions with correlated variables to be pushed > down (as of [CALCITE-769|https://issues.apache.org/jira/browse/CALCITE-769] > to avoid certain problems in the decorrelation of the query). However, in the > general context of query optimization, it is beneficial to push-down filters > and the fact that there is a correlated variable is not a reason to skip this > optimization. > In order to avoid regressions, and at the same time enable correlated > conditions to be pushed down we should make the pushing of correlated > variables configurable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CALCITE-2617) FilterProjectTransposeRule should allow filter conditions with correlated variables to be pushed down
[
https://issues.apache.org/jira/browse/CALCITE-2617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648631#comment-16648631
]
Julian Hyde commented on CALCITE-2617:
--
[~rubenql], Do not change the {{containsOver}} call. It is always wrong to push
a Filter through a Project that contains windowed-aggregates. It changes the
result. So it should not be a predicate.
I think that we should make it configurable via a Predicate in the constructor.
Not via a command-line {{-D}} argument.
If we make the new behavior more aggressive (i.e. push down even if there are
correlating variables), are there any regressions? If not, I think aggressive
should be the new default behavior. [~amansinha100], Since you were involved
in CALCITE-769, let me know if you disagree.
> FilterProjectTransposeRule should allow filter conditions with correlated
> variables to be pushed down
> -
>
> Key: CALCITE-2617
> URL: https://issues.apache.org/jira/browse/CALCITE-2617
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.17.0
>Reporter: Stamatis Zampetakis
>Assignee: Julian Hyde
>Priority: Major
> Fix For: 1.18.0
>
>
> The rule always forbids conditions with correlated variables to be pushed
> down (as of [CALCITE-769|https://issues.apache.org/jira/browse/CALCITE-769]
> to avoid certain problems in the decorrelation of the query). However, in the
> general context of query optimization, it is beneficial to push-down filters
> and the fact that there is a correlated variable is not a reason to skip this
> optimization.
> In order to avoid regressions, and at the same time enable correlated
> conditions to be pushed down we should make the pushing of correlated
> variables configurable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-2619) Reduce string literal creation cost by removing charset check
[ https://issues.apache.org/jira/browse/CALCITE-2619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648585#comment-16648585 ] Julian Hyde commented on CALCITE-2619: -- I think we should focus on the cost of validating whether a string is valid in UTF-8. (I probably have not seen this issue because I use Latin1 or something similar.) Is there a cheap general check? I did a Google search and the top hit was by my friend [~lemire]: [https://lemire.me/blog/2018/05/09/how-quickly-can-you-check-that-a-string-is-valid-unicode-utf-8/] > Reduce string literal creation cost by removing charset check > - > > Key: CALCITE-2619 > URL: https://issues.apache.org/jira/browse/CALCITE-2619 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Ted Xu >Assignee: Ted Xu >Priority: Major > > The cost of creating NlsString is very high, due to its charset check. In > some cases, e.g., expression evaluate because of Partition Prune, the > NlsString creation costs 40%+ of total executor's overhead. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CALCITE-2623) Updating unparseCall() in SqlDialect(s) for dialect specifc SqlNode unparsing
[
https://issues.apache.org/jira/browse/CALCITE-2623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648579#comment-16648579
]
Julian Hyde commented on CALCITE-2623:
--
Also note that an implementation of {{SqlNode.unparse(SqlWriter, int, int)}}
can call {{SqlWriter.getDialect()}}. This is how identifiers get quoted using
double-quote on Oracle, back-ticks on MySQL etc.
> Updating unparseCall() in SqlDialect(s) for dialect specifc SqlNode unparsing
> -
>
> Key: CALCITE-2623
> URL: https://issues.apache.org/jira/browse/CALCITE-2623
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: KrishnaKant Agrawal
>Assignee: Julian Hyde
>Priority: Minor
>
> Owing to the syntactical differences between Calcite SQL and certain other
> Dialects, emulation for SqlNode unparsing needs to be added based on the
> Dialect in RelToSqlConcverter.
> Saw some code in PostgreSqlDialect which is already doing this.
> Changes in other Dialects can follow suit.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-2623) Updating unparseCall() in SqlDialect(s) for dialect specifc SqlNode unparsing
[ https://issues.apache.org/jira/browse/CALCITE-2623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648575#comment-16648575 ] Julian Hyde commented on CALCITE-2623: -- Can you give an example where it currently fails? > Updating unparseCall() in SqlDialect(s) for dialect specifc SqlNode unparsing > - > > Key: CALCITE-2623 > URL: https://issues.apache.org/jira/browse/CALCITE-2623 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: KrishnaKant Agrawal >Assignee: Julian Hyde >Priority: Minor > > Owing to the syntactical differences between Calcite SQL and certain other > Dialects, emulation for SqlNode unparsing needs to be added based on the > Dialect in RelToSqlConcverter. > Saw some code in PostgreSqlDialect which is already doing this. > Changes in other Dialects can follow suit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CALCITE-2625) Function Specific Sql Window Unparsing where Framing Not Allowed
[ https://issues.apache.org/jira/browse/CALCITE-2625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Hyde updated CALCITE-2625: - Component/s: (was: core) jdbc-adapter > Function Specific Sql Window Unparsing where Framing Not Allowed > > > Key: CALCITE-2625 > URL: https://issues.apache.org/jira/browse/CALCITE-2625 > Project: Calcite > Issue Type: Bug > Components: jdbc-adapter >Reporter: KrishnaKant Agrawal >Assignee: Julian Hyde >Priority: Major > > The SQL standard says:- > If , , or > ROW_NUMBER is specified, then: … The window framing clause of WDX shall not > be present. > So, Calcite should not print the Window Frames when such functions are used. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CALCITE-2625) Function Specific Sql Window Unparsing where Framing Not Allowed
[
https://issues.apache.org/jira/browse/CALCITE-2625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648569#comment-16648569
]
Julian Hyde commented on CALCITE-2625:
--
I agree.
Note that we already have {{boolean SqlOperator.allowsFraming()}}. We just need
to use it when generating SQL.
Can you change the Jira case title to something more specific, e.g. about RANK
generating invalid SQL.
> Function Specific Sql Window Unparsing where Framing Not Allowed
>
>
> Key: CALCITE-2625
> URL: https://issues.apache.org/jira/browse/CALCITE-2625
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: KrishnaKant Agrawal
>Assignee: Julian Hyde
>Priority: Major
>
> The SQL standard says:-
> If , , or
> ROW_NUMBER is specified, then: … The window framing clause of WDX shall not
> be present.
> So, Calcite should not print the Window Frames when such functions are used.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CALCITE-2624) Add a rule to copy a sort below a binary operator over the left or right input
[ https://issues.apache.org/jira/browse/CALCITE-2624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648066#comment-16648066 ] Julian Hyde commented on CALCITE-2624: -- Slight difference in tactics: I’d focus on the concrete rules, then perhaps abstract the code into a base class if applicable. It’s better to write JIRA cases in terms of features that are useful to end users. It might apply to MInus, probably wouldn’t apply to Union > Add a rule to copy a sort below a binary operator over the left or right input > -- > > Key: CALCITE-2624 > URL: https://issues.apache.org/jira/browse/CALCITE-2624 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.17.0 >Reporter: Stamatis Zampetakis >Assignee: Julian Hyde >Priority: Major > Fix For: 1.18.0 > > > Currently, the only rule that allows a sort to traverse a binary operator is > the SortJoinTransposeRule. The rule was introduced mainly to push limits in > the case of left and right outer joins (see CALCITE-831). > I assume that the main reason that we don't have more rules is that sorts > with limits and offsets cannot be pushed safely below many binary operators. > However, in many cases, it is possible and beneficial for optimization > purposes to just push the sort (without the limit and offset) below the > binary operator. Since we do not know in advance if the binary operator > preserves the sort we cannot remove (that is why I am saying copy and not > transpose) the sort operator on top of the binary operator. The latter is not > really a problem since the SortRemoveRule can detect such cases and remove > the sort if it is redundant. > A few concrete examples where this optimization makes sense are outlined > below: > * allow the sort to be later absorbed by an index scan and disappear from > the plan (Sort + Tablescan => IndexScan with RelCollation); > * allow operators that require sorted inputs to be exploited more easily > (e.g., merge join); > * allow the sort to be performed on a possibly smaller result (assuming that > the physical binary operator that is going to be used preserves the order of > left/right input and the top sort operator can be removed entirely). > I propose to add a new rule (e.g., SortCopyBelowBiRelRule, > SortBiRelCopyBelowRule, SortBiRelTransposeRule) which allows a sort to be > copied to the left or right of a BiRel operator (excluding the limit and > offset attributes) if the respective input is not already sorted. For my use > case, it would suffice to apply the rule only on joins and correlates but I > don't see why not making it a bit more general from the beginning. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CALCITE-2626) RelBuilder generates wrong field list for join
[ https://issues.apache.org/jira/browse/CALCITE-2626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648059#comment-16648059 ] Julian Hyde commented on CALCITE-2626: -- Good one. Do you have an illustrative test case? > RelBuilder generates wrong field list for join > -- > > Key: CALCITE-2626 > URL: https://issues.apache.org/jira/browse/CALCITE-2626 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.17.0 >Reporter: Pavel Gubin >Assignee: Julian Hyde >Priority: Major > > When building OUTER JOIN RelBuilder doesn't change nullability of underlying > fields just copying them as is. But with outer join fields shall become > nullable at appropriate sides of join. > The error is at these lines: > https://github.com/apache/calcite/blob/fb7914ec61fc57c8348d9da144e2fbd608c00c61/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L1546 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2626) RelBuilder generates wrong field list for join
Pavel Gubin created CALCITE-2626: Summary: RelBuilder generates wrong field list for join Key: CALCITE-2626 URL: https://issues.apache.org/jira/browse/CALCITE-2626 Project: Calcite Issue Type: Bug Components: core Affects Versions: 1.17.0 Reporter: Pavel Gubin Assignee: Julian Hyde When building OUTER JOIN RelBuilder doesn't change nullability of underlying fields just copying them as is. But with outer join fields shall become nullable at appropriate sides of join. The error is at these lines: https://github.com/apache/calcite/blob/fb7914ec61fc57c8348d9da144e2fbd608c00c61/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L1546 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CALCITE-2621) New rule: SemiJoinToCorrelateRule (SemiJoin => LogicalCorrelate)
[
https://issues.apache.org/jira/browse/CALCITE-2621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647672#comment-16647672
]
Ruben Quesada Lopez commented on CALCITE-2621:
--
Modification pushed, test also included
> New rule: SemiJoinToCorrelateRule (SemiJoin => LogicalCorrelate)
>
>
> Key: CALCITE-2621
> URL: https://issues.apache.org/jira/browse/CALCITE-2621
> Project: Calcite
> Issue Type: Improvement
>Reporter: Ruben Quesada Lopez
>Assignee: Julian Hyde
>Priority: Major
> Fix For: 1.18.0
>
>
> Similarly to JoinToCorrelateRule, in some contexts it might be interesting to
> transform a SemiJoin into a LogicalCorrelate, for this purpose a
> SemiJoinToCorrelateRule will be needed.
> This feature requires the fix CALCITE-2605, otherwise we would get an
> exception in EnumerableCorrelate#implement when trying to generate the
> Expression selector:
> {code}
> Caused by: java.lang.IllegalStateException: Unable to convert SEMI to
> JoinRelType
> at org.apache.calcite.sql.SemiJoinType.toJoinType(SemiJoinType.java:83)
> at
> org.apache.calcite.adapter.enumerable.EnumerableCorrelate.implement(EnumerableCorrelate.java:103)
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CALCITE-2625) Function Specific Sql Window Unparsing where Framing Not Allowed
KrishnaKant Agrawal created CALCITE-2625: Summary: Function Specific Sql Window Unparsing where Framing Not Allowed Key: CALCITE-2625 URL: https://issues.apache.org/jira/browse/CALCITE-2625 Project: Calcite Issue Type: Bug Components: core Reporter: KrishnaKant Agrawal Assignee: Julian Hyde The SQL standard says:- If , , or ROW_NUMBER is specified, then: … The window framing clause of WDX shall not be present. So, Calcite should not print the Window Frames when such functions are used. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2624) Add a rule to copy a sort below a binary operator over the left or right input
Stamatis Zampetakis created CALCITE-2624: Summary: Add a rule to copy a sort below a binary operator over the left or right input Key: CALCITE-2624 URL: https://issues.apache.org/jira/browse/CALCITE-2624 Project: Calcite Issue Type: Improvement Components: core Affects Versions: 1.17.0 Reporter: Stamatis Zampetakis Assignee: Julian Hyde Fix For: 1.18.0 Currently, the only rule that allows a sort to traverse a binary operator is the SortJoinTransposeRule. The rule was introduced mainly to push limits in the case of left and right outer joins (see CALCITE-831). I assume that the main reason that we don't have more rules is that sorts with limits and offsets cannot be pushed safely below many binary operators. However, in many cases, it is possible and beneficial for optimization purposes to just push the sort (without the limit and offset) below the binary operator. Since we do not know in advance if the binary operator preserves the sort we cannot remove (that is why I am saying copy and not transpose) the sort operator on top of the binary operator. The latter is not really a problem since the SortRemoveRule can detect such cases and remove the sort if it is redundant. A few concrete examples where this optimization makes sense are outlined below: * allow the sort to be later absorbed by an index scan and disappear from the plan (Sort + Tablescan => IndexScan with RelCollation); * allow operators that require sorted inputs to be exploited more easily (e.g., merge join); * allow the sort to be performed on a possibly smaller result (assuming that the physical binary operator that is going to be used preserves the order of left/right input and the top sort operator can be removed entirely). I propose to add a new rule (e.g., SortCopyBelowBiRelRule, SortBiRelCopyBelowRule, SortBiRelTransposeRule) which allows a sort to be copied to the left or right of a BiRel operator (excluding the limit and offset attributes) if the respective input is not already sorted. For my use case, it would suffice to apply the rule only on joins and correlates but I don't see why not making it a bit more general from the beginning. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (CALCITE-2623) Updating unparseCall() in SqlDialect(s) for dialect specifc SqlNode unparsing
[ https://issues.apache.org/jira/browse/CALCITE-2623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647652#comment-16647652 ] KrishnaKant Agrawal edited comment on CALCITE-2623 at 10/12/18 8:34 AM: Will raise the first PR and then we can discuss if it is the right approach to approach this functionality. was (Author: krishnakant a): Will raise the first PR in a couple of Days. > Updating unparseCall() in SqlDialect(s) for dialect specifc SqlNode unparsing > - > > Key: CALCITE-2623 > URL: https://issues.apache.org/jira/browse/CALCITE-2623 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: KrishnaKant Agrawal >Assignee: Julian Hyde >Priority: Minor > > Owing to the syntactical differences between Calcite SQL and certain other > Dialects, emulation for SqlNode unparsing needs to be added based on the > Dialect in RelToSqlConcverter. > Saw some code in PostgreSqlDialect which is already doing this. > Changes in other Dialects can follow suit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CALCITE-2623) Updating unparseCall() in SqlDialect(s) for dialect specifc SqlNode unparsing
[ https://issues.apache.org/jira/browse/CALCITE-2623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647652#comment-16647652 ] KrishnaKant Agrawal commented on CALCITE-2623: -- Will raise the first PR in a couple of Days. > Updating unparseCall() in SqlDialect(s) for dialect specifc SqlNode unparsing > - > > Key: CALCITE-2623 > URL: https://issues.apache.org/jira/browse/CALCITE-2623 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: KrishnaKant Agrawal >Assignee: Julian Hyde >Priority: Minor > > Owing to the syntactical differences between Calcite SQL and certain other > Dialects, emulation for SqlNode unparsing needs to be added based on the > Dialect in RelToSqlConcverter. > Saw some code in PostgreSqlDialect which is already doing this. > Changes in other Dialects can follow suit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2623) Updating unparseCall() in SqlDialect(s) for dialect specifc SqlNode unparsing
KrishnaKant Agrawal created CALCITE-2623: Summary: Updating unparseCall() in SqlDialect(s) for dialect specifc SqlNode unparsing Key: CALCITE-2623 URL: https://issues.apache.org/jira/browse/CALCITE-2623 Project: Calcite Issue Type: Improvement Components: core Reporter: KrishnaKant Agrawal Assignee: Julian Hyde Owing to the syntactical differences between Calcite SQL and certain other Dialects, emulation for SqlNode unparsing needs to be added based on the Dialect in RelToSqlConcverter. Saw some code in PostgreSqlDialect which is already doing this. Changes in other Dialects can follow suit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CALCITE-2621) New rule: SemiJoinToCorrelateRule (SemiJoin => LogicalCorrelate)
[
https://issues.apache.org/jira/browse/CALCITE-2621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Quesada Lopez updated CALCITE-2621:
-
Description:
Similarly to JoinToCorrelateRule, in some contexts it might be interesting to
transform a SemiJoin into a LogicalCorrelate, for this purpose a
SemiJoinToCorrelateRule will be needed.
This feature requires the fix CALCITE-2605, otherwise we would get an exception
in EnumerableCorrelate#implement when trying to generate the Expression
selector:
{code}
Caused by: java.lang.IllegalStateException: Unable to convert SEMI to
JoinRelType
at org.apache.calcite.sql.SemiJoinType.toJoinType(SemiJoinType.java:83)
at
org.apache.calcite.adapter.enumerable.EnumerableCorrelate.implement(EnumerableCorrelate.java:103)
...
{code}
was:
Similarly to JoinToCorrelateRule, in some contexts it might be interesting to
transform a SemiJoin into a LogicalCorrelate, for this purpose a
SemiJoinToCorrelateRule will be needed.
This feature requires the fix CALCITE-2605, otherwise we would get an exception
in EnumerableCorrelate#implement when trying to generate the Expression
selector:
{{Caused by: java.lang.IllegalStateException: Unable to convert SEMI to
JoinRelType
at org.apache.calcite.sql.SemiJoinType.toJoinType(SemiJoinType.java:83)
at
org.apache.calcite.adapter.enumerable.EnumerableCorrelate.implement(EnumerableCorrelate.java:103)}}
> New rule: SemiJoinToCorrelateRule (SemiJoin => LogicalCorrelate)
>
>
> Key: CALCITE-2621
> URL: https://issues.apache.org/jira/browse/CALCITE-2621
> Project: Calcite
> Issue Type: Improvement
>Reporter: Ruben Quesada Lopez
>Assignee: Julian Hyde
>Priority: Major
> Fix For: 1.18.0
>
>
> Similarly to JoinToCorrelateRule, in some contexts it might be interesting to
> transform a SemiJoin into a LogicalCorrelate, for this purpose a
> SemiJoinToCorrelateRule will be needed.
> This feature requires the fix CALCITE-2605, otherwise we would get an
> exception in EnumerableCorrelate#implement when trying to generate the
> Expression selector:
> {code}
> Caused by: java.lang.IllegalStateException: Unable to convert SEMI to
> JoinRelType
> at org.apache.calcite.sql.SemiJoinType.toJoinType(SemiJoinType.java:83)
> at
> org.apache.calcite.adapter.enumerable.EnumerableCorrelate.implement(EnumerableCorrelate.java:103)
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (CALCITE-2621) New rule: SemiJoinToCorrelateRule (SemiJoin => LogicalCorrelate)
[
https://issues.apache.org/jira/browse/CALCITE-2621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruben Quesada Lopez updated CALCITE-2621:
-
Description:
Similarly to JoinToCorrelateRule, in some contexts it might be interesting to
transform a SemiJoin into a LogicalCorrelate, for this purpose a
SemiJoinToCorrelateRule will be needed.
This feature requires the fix CALCITE-2605, otherwise we would get an exception
in EnumerableCorrelate#implement when trying to generate the Expression
selector:
{{Caused by: java.lang.IllegalStateException: Unable to convert SEMI to
JoinRelType
at org.apache.calcite.sql.SemiJoinType.toJoinType(SemiJoinType.java:83)
at
org.apache.calcite.adapter.enumerable.EnumerableCorrelate.implement(EnumerableCorrelate.java:103)}}
was:
Similarly to JoinToCorrelateRule, in some contexts it might be interesting to
transform a SemiJoin into a LogicalCorrelate, for this purpose a
SemiJoinToCorrelateRule will be needed.
This feature requires the fix
> New rule: SemiJoinToCorrelateRule (SemiJoin => LogicalCorrelate)
>
>
> Key: CALCITE-2621
> URL: https://issues.apache.org/jira/browse/CALCITE-2621
> Project: Calcite
> Issue Type: Improvement
>Reporter: Ruben Quesada Lopez
>Assignee: Julian Hyde
>Priority: Major
> Fix For: 1.18.0
>
>
> Similarly to JoinToCorrelateRule, in some contexts it might be interesting to
> transform a SemiJoin into a LogicalCorrelate, for this purpose a
> SemiJoinToCorrelateRule will be needed.
> This feature requires the fix CALCITE-2605, otherwise we would get an
> exception in EnumerableCorrelate#implement when trying to generate the
> Expression selector:
> {{Caused by: java.lang.IllegalStateException: Unable to convert SEMI to
> JoinRelType
> at org.apache.calcite.sql.SemiJoinType.toJoinType(SemiJoinType.java:83)
> at
> org.apache.calcite.adapter.enumerable.EnumerableCorrelate.implement(EnumerableCorrelate.java:103)}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (CALCITE-2621) New rule: SemiJoinToCorrelateRule (SemiJoin => LogicalCorrelate)
[ https://issues.apache.org/jira/browse/CALCITE-2621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruben Quesada Lopez updated CALCITE-2621: - Description: Similarly to JoinToCorrelateRule, in some contexts it might be interesting to transform a SemiJoin into a LogicalCorrelate, for this purpose a SemiJoinToCorrelateRule will be needed. This feature requires the fix was:Similarly to JoinToCorrelateRule, in some contexts it might be interesting to transform a SemiJoin into a LogicalCorrelate, for this purpose a SemiJoinToCorrelateRule will be needed. > New rule: SemiJoinToCorrelateRule (SemiJoin => LogicalCorrelate) > > > Key: CALCITE-2621 > URL: https://issues.apache.org/jira/browse/CALCITE-2621 > Project: Calcite > Issue Type: Improvement >Reporter: Ruben Quesada Lopez >Assignee: Julian Hyde >Priority: Major > Fix For: 1.18.0 > > > Similarly to JoinToCorrelateRule, in some contexts it might be interesting to > transform a SemiJoin into a LogicalCorrelate, for this purpose a > SemiJoinToCorrelateRule will be needed. > This feature requires the fix -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CALCITE-2619) Reduce string literal creation cost by removing charset check
[ https://issues.apache.org/jira/browse/CALCITE-2619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647565#comment-16647565 ] Haisheng Yuan commented on CALCITE-2619: I am not worried about the memory overhead for caching charset object, since there are limited number of charsets. > Reduce string literal creation cost by removing charset check > - > > Key: CALCITE-2619 > URL: https://issues.apache.org/jira/browse/CALCITE-2619 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Ted Xu >Assignee: Ted Xu >Priority: Major > > The cost of creating NlsString is very high, due to its charset check. In > some cases, e.g., expression evaluate because of Partition Prune, the > NlsString creation costs 40%+ of total executor's overhead. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (CALCITE-2619) Reduce string literal creation cost by removing charset check
[ https://issues.apache.org/jira/browse/CALCITE-2619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647560#comment-16647560 ] Haisheng Yuan edited comment on CALCITE-2619 at 10/12/18 7:16 AM: -- [~julianhyde] Once the CharsetEncoder of the CharSet passes the encoding test, do we need to do the charset test every time when NlsString is created with the same charset? If not, we can definitely cache the Charset after it passes charset test, and skip the charset test if it can be found in the cache. But [~tedxu] said it is still expensive, how much overhead for doing this? was (Author: hyuan): [~julianhyde] Once the CharsetEncoder of the CharSet passes the encoding test, do we need to do the charset test every time when NlsString is created with the same charset? If not, we can definitely cache the Charset after it passes charset test, and skip the charset test if it can be found in the cache. > Reduce string literal creation cost by removing charset check > - > > Key: CALCITE-2619 > URL: https://issues.apache.org/jira/browse/CALCITE-2619 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Ted Xu >Assignee: Ted Xu >Priority: Major > > The cost of creating NlsString is very high, due to its charset check. In > some cases, e.g., expression evaluate because of Partition Prune, the > NlsString creation costs 40%+ of total executor's overhead. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CALCITE-2619) Reduce string literal creation cost by removing charset check
[ https://issues.apache.org/jira/browse/CALCITE-2619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647560#comment-16647560 ] Haisheng Yuan commented on CALCITE-2619: [~julianhyde] Once the CharsetEncoder of the CharSet passes the encoding test, do we need to do the charset test every time when NlsString is created with the same charset? If not, we can definitely cache the Charset after it passes charset test, and skip the charset test if it can be found in the cache. > Reduce string literal creation cost by removing charset check > - > > Key: CALCITE-2619 > URL: https://issues.apache.org/jira/browse/CALCITE-2619 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Ted Xu >Assignee: Ted Xu >Priority: Major > > The cost of creating NlsString is very high, due to its charset check. In > some cases, e.g., expression evaluate because of Partition Prune, the > NlsString creation costs 40%+ of total executor's overhead. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CALCITE-2621) New rule: SemiJoinToCorrelateRule (SemiJoin => LogicalCorrelate)
[
https://issues.apache.org/jira/browse/CALCITE-2621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647551#comment-16647551
]
Ruben Quesada Lopez commented on CALCITE-2621:
--
Ok, I can remove the function and extract the SemiJoin type for the
LogicalCorrelate programatically:
{code}
RelNode newRel =
LogicalCorrelate.create(left,
relBuilder.build(),
correlationId,
requiredColumns.build(),
join instanceof SemiJoin ? SemiJoinType.SEMI :
SemiJoinType.of(join.getJoinType()));
{code}
> New rule: SemiJoinToCorrelateRule (SemiJoin => LogicalCorrelate)
>
>
> Key: CALCITE-2621
> URL: https://issues.apache.org/jira/browse/CALCITE-2621
> Project: Calcite
> Issue Type: Improvement
>Reporter: Ruben Quesada Lopez
>Assignee: Julian Hyde
>Priority: Major
> Fix For: 1.18.0
>
>
> Similarly to JoinToCorrelateRule, in some contexts it might be interesting to
> transform a SemiJoin into a LogicalCorrelate, for this purpose a
> SemiJoinToCorrelateRule will be needed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (CALCITE-2621) New rule: SemiJoinToCorrelateRule (SemiJoin => LogicalCorrelate)
[
https://issues.apache.org/jira/browse/CALCITE-2621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647551#comment-16647551
]
Ruben Quesada Lopez edited comment on CALCITE-2621 at 10/12/18 7:00 AM:
Ok, I can remove the function and extract the SemiJoin type for the
LogicalCorrelate programatically:
{code:java}
RelNode newRel =
LogicalCorrelate.create(left,
relBuilder.build(),
correlationId,
requiredColumns.build(),
join instanceof SemiJoin ? SemiJoinType.SEMI :
SemiJoinType.of(join.getJoinType()));
{code}
was (Author: rubenql):
Ok, I can remove the function and extract the SemiJoin type for the
LogicalCorrelate programatically:
{code}
RelNode newRel =
LogicalCorrelate.create(left,
relBuilder.build(),
correlationId,
requiredColumns.build(),
join instanceof SemiJoin ? SemiJoinType.SEMI :
SemiJoinType.of(join.getJoinType()));
{code}
> New rule: SemiJoinToCorrelateRule (SemiJoin => LogicalCorrelate)
>
>
> Key: CALCITE-2621
> URL: https://issues.apache.org/jira/browse/CALCITE-2621
> Project: Calcite
> Issue Type: Improvement
>Reporter: Ruben Quesada Lopez
>Assignee: Julian Hyde
>Priority: Major
> Fix For: 1.18.0
>
>
> Similarly to JoinToCorrelateRule, in some contexts it might be interesting to
> transform a SemiJoin into a LogicalCorrelate, for this purpose a
> SemiJoinToCorrelateRule will be needed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
