[jira] [Commented] (CALCITE-2041) Adding the ability to turn off nullability matching for ReduceExpressionsRule
[ https://issues.apache.org/jira/browse/CALCITE-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261731#comment-16261731 ] Jesus Camacho Rodriguez commented on CALCITE-2041: -- [~julianhyde], I have added a unit test and additional comments in : https://github.com/apache/calcite/pull/570 > Adding the ability to turn off nullability matching for ReduceExpressionsRule > - > > Key: CALCITE-2041 > URL: https://issues.apache.org/jira/browse/CALCITE-2041 > Project: Calcite > Issue Type: Bug >Reporter: slim bouguerra >Assignee: Julian Hyde > Fix For: 1.15.0 > > > In some cases, the user needs to select whether or not to add casts that > match nullability. > One of the motivations behind this is to avoid unnecessary casts like the > following example. > original filter > {code} > OR(AND(>=($0, CAST(_UTF-16LE'2010-01-01 00:00:00 > UTC'):TIMESTAMP_WITH_LOCAL_TIME_ZONE(15)), <=($0, CAST(_UTF-16LE'2012-03-01 > 00:00:00 UTC'):TIMESTAMP_WITH_LOCAL_TIME_ZONE(15 > {code} > the optimized expression with matching nullability > {code} > OR(AND(CAST(>=($0, 2010-01-01 00:00:00)):BOOLEAN, CAST(<=($0, 2012-03-01 > 00:00:00)):BOOLEAN)) > {code} > As you can see this extra cast gets into the way of following plan > optimization steps. > The desired expression can be obtained by turning off the nullability > matching. > {code} > OR(AND(>=($0, 2010-01-01 00:00:00), <=($0, 2012-03-01 00:00:00))) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (CALCITE-2041) Adding the ability to turn off nullability matching for ReduceExpressionsRule
[ https://issues.apache.org/jira/browse/CALCITE-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261639#comment-16261639 ] Julian Hyde commented on CALCITE-2041: -- We would like this to go into 1.15 but the PR isn't ready. The PR needs tests, and does not have a clear explanation of the new functionality. [~jcamachorodriguez] and [~bslim], it's possible that you are looking at this from a Hive perspective, and see this as a just a small change in Calcite necessary to make Hive work. But the PR needs to be self-contained from Calcite's perspective too. Can you please revisit it? > Adding the ability to turn off nullability matching for ReduceExpressionsRule > - > > Key: CALCITE-2041 > URL: https://issues.apache.org/jira/browse/CALCITE-2041 > Project: Calcite > Issue Type: Bug >Reporter: slim bouguerra >Assignee: Julian Hyde > Fix For: 1.15.0 > > > In some cases, the user needs to select whether or not to add casts that > match nullability. > One of the motivations behind this is to avoid unnecessary casts like the > following example. > original filter > {code} > OR(AND(>=($0, CAST(_UTF-16LE'2010-01-01 00:00:00 > UTC'):TIMESTAMP_WITH_LOCAL_TIME_ZONE(15)), <=($0, CAST(_UTF-16LE'2012-03-01 > 00:00:00 UTC'):TIMESTAMP_WITH_LOCAL_TIME_ZONE(15 > {code} > the optimized expression with matching nullability > {code} > OR(AND(CAST(>=($0, 2010-01-01 00:00:00)):BOOLEAN, CAST(<=($0, 2012-03-01 > 00:00:00)):BOOLEAN)) > {code} > As you can see this extra cast gets into the way of following plan > optimization steps. > The desired expression can be obtained by turning off the nullability > matching. > {code} > OR(AND(>=($0, 2010-01-01 00:00:00), <=($0, 2012-03-01 00:00:00))) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (CALCITE-2041) Adding the ability to turn off nullability matching for ReduceExpressionsRule
[ https://issues.apache.org/jira/browse/CALCITE-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16254394#comment-16254394 ] Jesus Camacho Rodriguez commented on CALCITE-2041: -- bq. Potentially this could cause the whole expression to change type from nullable to not null. So if the type of the whole expression needs to be preserved (think project expressions) compensating casts will need to be added at the top. In fact this flag will only be 'false' (i.e. do not match nullability) on the Hive side for the Filter instance of ReduceExpressionsRule. Setting the flag to 'false' for the Project instance of the rule would potentially cause issues with the planner, as the nullability of an expression row type generated by a rule needs to match the one of the expression that was matched by the rule (we talked about this in another issue). > Adding the ability to turn off nullability matching for ReduceExpressionsRule > - > > Key: CALCITE-2041 > URL: https://issues.apache.org/jira/browse/CALCITE-2041 > Project: Calcite > Issue Type: Bug >Reporter: slim bouguerra >Assignee: Julian Hyde > Fix For: 1.15.0 > > > In some cases, the user needs to select whether or not to add casts that > match nullability. > One of the motivations behind this is to avoid unnecessary casts like the > following example. > original filter > {code} > OR(AND(>=($0, CAST(_UTF-16LE'2010-01-01 00:00:00 > UTC'):TIMESTAMP_WITH_LOCAL_TIME_ZONE(15)), <=($0, CAST(_UTF-16LE'2012-03-01 > 00:00:00 UTC'):TIMESTAMP_WITH_LOCAL_TIME_ZONE(15 > {code} > the optimized expression with matching nullability > {code} > OR(AND(CAST(>=($0, 2010-01-01 00:00:00)):BOOLEAN, CAST(<=($0, 2012-03-01 > 00:00:00)):BOOLEAN)) > {code} > As you can see this extra cast gets into the way of following plan > optimization steps. > The desired expression can be obtained by turning off the nullability > matching. > {code} > OR(AND(>=($0, 2010-01-01 00:00:00), <=($0, 2012-03-01 00:00:00))) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (CALCITE-2041) Adding the ability to turn off nullability matching for ReduceExpressionsRule
[ https://issues.apache.org/jira/browse/CALCITE-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16254378#comment-16254378 ] Julian Hyde commented on CALCITE-2041: -- Nullability can only change in one direction, so let's be specific: a1 has a nullable type, and a2 is a not-null value. Potentially this could cause the whole expression to change type from nullable to not null. So if the type of the whole expression needs to be preserved (think project expressions) compensating casts will need to be added at the top. > Adding the ability to turn off nullability matching for ReduceExpressionsRule > - > > Key: CALCITE-2041 > URL: https://issues.apache.org/jira/browse/CALCITE-2041 > Project: Calcite > Issue Type: Bug >Reporter: slim bouguerra >Assignee: Julian Hyde > Fix For: 1.15.0 > > > In some cases, the user needs to select whether or not to add casts that > match nullability. > One of the motivations behind this is to avoid unnecessary casts like the > following example. > original filter > {code} > OR(AND(>=($0, CAST(_UTF-16LE'2010-01-01 00:00:00 > UTC'):TIMESTAMP_WITH_LOCAL_TIME_ZONE(15)), <=($0, CAST(_UTF-16LE'2012-03-01 > 00:00:00 UTC'):TIMESTAMP_WITH_LOCAL_TIME_ZONE(15 > {code} > the optimized expression with matching nullability > {code} > OR(AND(CAST(>=($0, 2010-01-01 00:00:00)):BOOLEAN, CAST(<=($0, 2012-03-01 > 00:00:00)):BOOLEAN)) > {code} > As you can see this extra cast gets into the way of following plan > optimization steps. > The desired expression can be obtained by turning off the nullability > matching. > {code} > OR(AND(>=($0, 2010-01-01 00:00:00), <=($0, 2012-03-01 00:00:00))) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (CALCITE-2041) Adding the ability to turn off nullability matching for ReduceExpressionsRule
[ https://issues.apache.org/jira/browse/CALCITE-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16254358#comment-16254358 ] Jesus Camacho Rodriguez commented on CALCITE-2041: -- We have an expression {{a1}} that is simplified to {{a2}}. "Nullability matching" means that if {{a2}} is the same type as {{a1}} except for _nullability_ flag, we introduce a CAST. However, if we are not matching nullability, then we will not introduce the CAST, as they are already the same base type. > Adding the ability to turn off nullability matching for ReduceExpressionsRule > - > > Key: CALCITE-2041 > URL: https://issues.apache.org/jira/browse/CALCITE-2041 > Project: Calcite > Issue Type: Bug >Reporter: slim bouguerra >Assignee: Julian Hyde > Fix For: 1.15.0 > > > In some cases, the user needs to select whether or not to add casts that > match nullability. > One of the motivations behind this is to avoid unnecessary casts like the > following example. > original filter > {code} > OR(AND(>=($0, CAST(_UTF-16LE'2010-01-01 00:00:00 > UTC'):TIMESTAMP_WITH_LOCAL_TIME_ZONE(15)), <=($0, CAST(_UTF-16LE'2012-03-01 > 00:00:00 UTC'):TIMESTAMP_WITH_LOCAL_TIME_ZONE(15 > {code} > the optimized expression with matching nullability > {code} > OR(AND(CAST(>=($0, 2010-01-01 00:00:00)):BOOLEAN, CAST(<=($0, 2012-03-01 > 00:00:00)):BOOLEAN)) > {code} > As you can see this extra cast gets into the way of following plan > optimization steps. > The desired expression can be obtained by turning off the nullability > matching. > {code} > OR(AND(>=($0, 2010-01-01 00:00:00), <=($0, 2012-03-01 00:00:00))) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (CALCITE-2041) Adding the ability to turn off nullability matching for ReduceExpressionsRule
[ https://issues.apache.org/jira/browse/CALCITE-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16254345#comment-16254345 ] Julian Hyde commented on CALCITE-2041: -- Maybe I'm stupid, but can someone explain "nullability matching" in simple English? > Adding the ability to turn off nullability matching for ReduceExpressionsRule > - > > Key: CALCITE-2041 > URL: https://issues.apache.org/jira/browse/CALCITE-2041 > Project: Calcite > Issue Type: Bug >Reporter: slim bouguerra >Assignee: Julian Hyde > Fix For: 1.15.0 > > > In some cases, the user needs to select whether or not to add casts that > match nullability. > One of the motivations behind this is to avoid unnecessary casts like the > following example. > original filter > {code} > OR(AND(>=($0, CAST(_UTF-16LE'2010-01-01 00:00:00 > UTC'):TIMESTAMP_WITH_LOCAL_TIME_ZONE(15)), <=($0, CAST(_UTF-16LE'2012-03-01 > 00:00:00 UTC'):TIMESTAMP_WITH_LOCAL_TIME_ZONE(15 > {code} > the optimized expression with matching nullability > {code} > OR(AND(CAST(>=($0, 2010-01-01 00:00:00)):BOOLEAN, CAST(<=($0, 2012-03-01 > 00:00:00)):BOOLEAN)) > {code} > As you can see this extra cast gets into the way of following plan > optimization steps. > The desired expression can be obtained by turning off the nullability > matching. > {code} > OR(AND(>=($0, 2010-01-01 00:00:00), <=($0, 2012-03-01 00:00:00))) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (CALCITE-2041) Adding the ability to turn off nullability matching for ReduceExpressionsRule
[ https://issues.apache.org/jira/browse/CALCITE-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252187#comment-16252187 ] Jesus Camacho Rodriguez commented on CALCITE-2041: -- [~julianhyde], we have been trying to create tests for this issue (ReduceExpressionsRule#FILTER_INSTANCE is disabled by default so we tried to use RelOptRulesTest), but we could not come up with same plan as in Hive. Do you have any other suggestions? [~bslim] has been testing the rule in Hive with the change his proposed change (HIVE-17871) and everything is working as expected and without issues, so I would like to check in the patch if possible. To be clear, the proposed change is quite straightforward: it exposes a flag to match to indicate whether reduce expressions should match nullability or not when expressions are simplified (by default it is _true_, so current behavior is preserved). > Adding the ability to turn off nullability matching for ReduceExpressionsRule > - > > Key: CALCITE-2041 > URL: https://issues.apache.org/jira/browse/CALCITE-2041 > Project: Calcite > Issue Type: Bug >Reporter: slim bouguerra >Assignee: Julian Hyde > > In some cases, the user needs to select whether or not to add casts that > match nullability. > One of the motivations behind this is to avoid unnecessary casts like the > following example. > original filter > {code} > OR(AND(>=($0, CAST(_UTF-16LE'2010-01-01 00:00:00 > UTC'):TIMESTAMP_WITH_LOCAL_TIME_ZONE(15)), <=($0, CAST(_UTF-16LE'2012-03-01 > 00:00:00 UTC'):TIMESTAMP_WITH_LOCAL_TIME_ZONE(15 > {code} > the optimized expression with matching nullability > {code} > OR(AND(CAST(>=($0, 2010-01-01 00:00:00)):BOOLEAN, CAST(<=($0, 2012-03-01 > 00:00:00)):BOOLEAN)) > {code} > As you can see this extra cast gets into the way of following plan > optimization steps. > The desired expression can be obtained by turning off the nullability > matching. > {code} > OR(AND(>=($0, 2010-01-01 00:00:00), <=($0, 2012-03-01 00:00:00))) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (CALCITE-2041) Adding the ability to turn off nullability matching for ReduceExpressionsRule
[ https://issues.apache.org/jira/browse/CALCITE-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244315#comment-16244315 ] Julian Hyde commented on CALCITE-2041: -- [~bslim], Your communication style is difficult. What does it mean when you post a link to a PR, with no other text, as a comment to a JIRA case? (You did this in this case, CALCITE-2041, and also in CALCITE-2019.) Most people would assume this means that the PR is ready to review and commit. You seem to think this is personal. It is not. But I sure do get pissed off when people waste my time, as you did, by posting incomplete PRs without any indication that they were incomplete. I would have been a lot less pissed off if, after the checkstyle issue in CALCITE-2019, you had simply said "Sorry, I'll fix it." And now you are accusing me of making personal attacks. What you think that is going to do for our working relationship? > Adding the ability to turn off nullability matching for ReduceExpressionsRule > - > > Key: CALCITE-2041 > URL: https://issues.apache.org/jira/browse/CALCITE-2041 > Project: Calcite > Issue Type: Bug >Reporter: slim bouguerra >Assignee: Julian Hyde > > In some cases, the user needs to select whether or not to add casts that > match nullability. > One of the motivations behind this is to avoid unnecessary casts like the > following example. > original filter > {code} > OR(AND(>=($0, CAST(_UTF-16LE'2010-01-01 00:00:00 > UTC'):TIMESTAMP_WITH_LOCAL_TIME_ZONE(15)), <=($0, CAST(_UTF-16LE'2012-03-01 > 00:00:00 UTC'):TIMESTAMP_WITH_LOCAL_TIME_ZONE(15 > {code} > the optimized expression with matching nullability > {code} > OR(AND(CAST(>=($0, 2010-01-01 00:00:00)):BOOLEAN, CAST(<=($0, 2012-03-01 > 00:00:00)):BOOLEAN)) > {code} > As you can see this extra cast gets into the way of following plan > optimization steps. > The desired expression can be obtained by turning off the nullability > matching. > {code} > OR(AND(>=($0, 2010-01-01 00:00:00), <=($0, 2012-03-01 00:00:00))) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (CALCITE-2041) Adding the ability to turn off nullability matching for ReduceExpressionsRule
[ https://issues.apache.org/jira/browse/CALCITE-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244311#comment-16244311 ] slim bouguerra commented on CALCITE-2041: - Trying to add some testing to class {code} org.apache.calcite.rel.rules.ReduceExpressionsRule#FILTER_INSTANCE {code} part of {code} org.apache.calcite.prepare.CalcitePrepareImpl#CONSTANT_REDUCTION_RULES {code} but looking at the code base i can see that the only use of such rule is excluded via this block at org/apache/calcite/prepare/CalcitePrepareImpl.java:571 git hash 221739354b56e34e9f1d41b42a0e6881a8f5ddee {code} // Change the below to enable constant-reduction. if (false) { for (RelOptRule rule : CONSTANT_REDUCTION_RULES) { planner.addRule(rule); } } {code} while the comment says it is enabling constant reductions I can not see how this is done since the block is never executed? not sure what is the idea of keeping such code with {code} if (false) {code}. Not sure what is the best way to test such rule since it is excluded from planer anyway. > Adding the ability to turn off nullability matching for ReduceExpressionsRule > - > > Key: CALCITE-2041 > URL: https://issues.apache.org/jira/browse/CALCITE-2041 > Project: Calcite > Issue Type: Bug >Reporter: slim bouguerra >Assignee: Julian Hyde > > In some cases, the user needs to select whether or not to add casts that > match nullability. > One of the motivations behind this is to avoid unnecessary casts like the > following example. > original filter > {code} > OR(AND(>=($0, CAST(_UTF-16LE'2010-01-01 00:00:00 > UTC'):TIMESTAMP_WITH_LOCAL_TIME_ZONE(15)), <=($0, CAST(_UTF-16LE'2012-03-01 > 00:00:00 UTC'):TIMESTAMP_WITH_LOCAL_TIME_ZONE(15 > {code} > the optimized expression with matching nullability > {code} > OR(AND(CAST(>=($0, 2010-01-01 00:00:00)):BOOLEAN, CAST(<=($0, 2012-03-01 > 00:00:00)):BOOLEAN)) > {code} > As you can see this extra cast gets into the way of following plan > optimization steps. > The desired expression can be obtained by turning off the nullability > matching. > {code} > OR(AND(>=($0, 2010-01-01 00:00:00), <=($0, 2012-03-01 00:00:00))) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (CALCITE-2041) Adding the ability to turn off nullability matching for ReduceExpressionsRule
[ https://issues.apache.org/jira/browse/CALCITE-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244016#comment-16244016 ] slim bouguerra commented on CALCITE-2041: - [~julianhyde] why the discussions have to be always you reminding people how bad they are and how great you are? The fact that my previous PR failed the checkstyle has nothing to deal with this contribution and I haven't written test yet since am not even sure if this is the way to go. > Adding the ability to turn off nullability matching for ReduceExpressionsRule > - > > Key: CALCITE-2041 > URL: https://issues.apache.org/jira/browse/CALCITE-2041 > Project: Calcite > Issue Type: Bug >Reporter: slim bouguerra >Assignee: Julian Hyde > > In some cases, the user needs to select whether or not to add casts that > match nullability. > One of the motivations behind this is to avoid unnecessary casts like the > following example. > original filter > {code} > OR(AND(>=($0, CAST(_UTF-16LE'2010-01-01 00:00:00 > UTC'):TIMESTAMP_WITH_LOCAL_TIME_ZONE(15)), <=($0, CAST(_UTF-16LE'2012-03-01 > 00:00:00 UTC'):TIMESTAMP_WITH_LOCAL_TIME_ZONE(15 > {code} > the optimized expression with matching nullability > {code} > OR(AND(CAST(>=($0, 2010-01-01 00:00:00)):BOOLEAN, CAST(<=($0, 2012-03-01 > 00:00:00)):BOOLEAN)) > {code} > As you can see this extra cast gets into the way of following plan > optimization steps. > The desired expression can be obtained by turning off the nullability > matching. > {code} > OR(AND(>=($0, 2010-01-01 00:00:00), <=($0, 2012-03-01 00:00:00))) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (CALCITE-2041) Adding the ability to turn off nullability matching for ReduceExpressionsRule
[ https://issues.apache.org/jira/browse/CALCITE-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16243334#comment-16243334 ] Julian Hyde commented on CALCITE-2041: -- [~bslim], Please make it clear when you consider a pull request "ready", before we spend time fully testing it. This one doesn't have tests; your previous PR failed checkstyle. In this case, tests would help a lot. I don't understand from your example what the rule is supposed to do nor not do. Nor can I understand why everything is not NOT NULL in your example. > Adding the ability to turn off nullability matching for ReduceExpressionsRule > - > > Key: CALCITE-2041 > URL: https://issues.apache.org/jira/browse/CALCITE-2041 > Project: Calcite > Issue Type: Bug >Reporter: slim bouguerra >Assignee: Julian Hyde > > In some cases, the user needs to select whether or not to add casts that > match nullability. > One of the motivations behind this is to avoid unnecessary casts like the > following example. > original filter > {code} > OR(AND(>=($0, CAST(_UTF-16LE'2010-01-01 00:00:00 > UTC'):TIMESTAMP_WITH_LOCAL_TIME_ZONE(15)), <=($0, CAST(_UTF-16LE'2012-03-01 > 00:00:00 UTC'):TIMESTAMP_WITH_LOCAL_TIME_ZONE(15 > {code} > the optimized expression with matching nullability > {code} > OR(AND(CAST(>=($0, 2010-01-01 00:00:00)):BOOLEAN, CAST(<=($0, 2012-03-01 > 00:00:00)):BOOLEAN)) > {code} > As you can see this extra cast gets into the way of following plan > optimization steps. > The desired expression can be obtained by turning off the nullability > matching. > {code} > OR(AND(>=($0, 2010-01-01 00:00:00), <=($0, 2012-03-01 00:00:00))) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (CALCITE-2041) Adding the ability to turn off nullability matching for ReduceExpressionsRule
[ https://issues.apache.org/jira/browse/CALCITE-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16243157#comment-16243157 ] slim bouguerra commented on CALCITE-2041: - https://github.com/apache/calcite/pull/563 > Adding the ability to turn off nullability matching for ReduceExpressionsRule > - > > Key: CALCITE-2041 > URL: https://issues.apache.org/jira/browse/CALCITE-2041 > Project: Calcite > Issue Type: Bug >Reporter: slim bouguerra >Assignee: Julian Hyde > > In some cases, the user needs to select whether or not to add casts that > match nullability. > One of the motivations behind this is to avoid unnecessary casts like the > following example. > original filter > {code} > OR(AND(>=($0, CAST(_UTF-16LE'2010-01-01 00:00:00 > UTC'):TIMESTAMP_WITH_LOCAL_TIME_ZONE(15)), <=($0, CAST(_UTF-16LE'2012-03-01 > 00:00:00 UTC'):TIMESTAMP_WITH_LOCAL_TIME_ZONE(15 > {code} > the optimized expression with matching nullability > {code} > OR(AND(CAST(>=($0, 2010-01-01 00:00:00)):BOOLEAN, CAST(<=($0, 2012-03-01 > 00:00:00)):BOOLEAN)) > {code} > As you can see this extra cast gets into the way of following plan > optimization steps. > The desired expression can be obtained by turning off the nullability > matching. > {code} > OR(AND(>=($0, 2010-01-01 00:00:00), <=($0, 2012-03-01 00:00:00)),) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)