Hello Julian,
I added an extra check and prevent constant reduction in the HiveExecutor for
TIMESTAMPs that have precision higher than 3. Not based on type, since Hive has
always TIMESTAMP(9), but on actual values that contain non-zero sub-millisecond
nanos. This almost works, the problem is that the expression still gets reduced
at a later stage:
at org.apache.calcite.rex.RexBuilder.makeLiteral(RexBuilder.java:1239)
at org.apache.calcite.rex.RexBuilder.makeLiteral(RexBuilder.java:1236)
at org.apache.calcite.rex.RexExecutable.reduce(RexExecutable.java:86)
at
org.apache.calcite.rex.RexExecutorImpl.reduce(RexExecutorImpl.java:128)
at org.apache.calcite.rex.RexUtil.simplifyCast(RexUtil.java:2450)
at org.apache.calcite.rex.RexUtil.simplify(RexUtil.java:1633)
at org.apache.calcite.rex.RexUtil.simplify(RexUtil.java:1587)
at org.apache.calcite.rex.RexUtil.simplifyList(RexUtil.java:1747)
at org.apache.calcite.rex.RexUtil.simplifyComparison(RexUtil.java:1658)
at org.apache.calcite.rex.RexUtil.simplify(RexUtil.java:1648)
at
org.apache.calcite.rex.RexUtil$ExprSimplifier.visitCall(RexUtil.java:3051)
at
org.apache.calcite.rex.RexUtil$ExprSimplifier.visitCall(RexUtil.java:3016)
at org.apache.calcite.rex.RexCall.accept(RexCall.java:104)
at org.apache.calcite.rex.RexShuttle.apply(RexShuttle.java:279)
Here the issue is that the custom HiveExecutor gets lost on the stack and the
default EXECUTOR is used instead, resulting in incorrect behavior. I created
CALCITE-1695 Not all RexUtil.simplifyXxx code paths carry the provided
executor. This new CALCITE-1695 issue is fairly trivial to fix. I know Julian
talked about making the executor a property of the RexBuilder and that would be
better long term, but right now I tried just extending the signature of
simplifyComparison/SimplifyList to accept and pass on a RexExecutor and this
seems to solve the issues I was seeing in Hive.
Thanks,
~Remus
On 3/13/17, 4:32 PM, "Julian Hyde" <[email protected]> wrote:
1690 is a big one. For now, can you just disable constant reduction for
timestamp literals that have high precision?
> On Mar 13, 2017, at 4:30 PM, Remus Rusanu <[email protected]> wrote:
>
> CALCITE-1681 is not needed anymore
> The only pending issue for Hive is CALCITE-1690: Calcite timestamp
literals cannot express precision above millisecond.
>
> Thanks,
> ~Remus
>
> ________________________________________
> From: Julian Hyde <[email protected]>
> Sent: Tuesday, March 14, 2017 1:23 AM
> To: [email protected]
> Subject: Re: Towards Calcite release 1.12
>
> OK, what’s left before RC0?
>
> I am working on committing 1561, 1684, 1691, 1688. Also, I will review
1566 shortly. I may also get to 1689.
>
> Remus, Are all of your Hive issues resolved? In particular do you need
1681?
>
> Jesus, Where are you with 1421, 1456, 1413? Can you also commit Nishant’s
1683 since you +1’d it?
>
> Julian
>
>
>> On Mar 8, 2017, at 12:25 PM, Julian Hyde <[email protected]> wrote:
>>
>> We have that dependency already, albeit transitively: CALCITE-1681
blocks HIVE-15708, which is depended upon by CALCITE-1657 “Release Calcite
1.12.0”.
>>
>>> On Mar 8, 2017, at 12:12 PM, Remus Rusanu <[email protected]>
wrote:
>>>
>>> Looks like CALCITE-1681 ‘Provide a way to copy RelNode trees between
clusters’ is also needed. I hope I’ll have it ready this week.
>>>
>>> On 3/8/17, 10:49 AM, "Julian Hyde" <[email protected]> wrote:
>>>
>>> Same comments as for CALCITE-1615. The list contains commits up to
2/24. I will fix before the release.
>>>
>>>> On Mar 8, 2017, at 10:43 AM, Eli Levine <[email protected]> wrote:
>>>>
>>>> https://issues.apache.org/jira/browse/CALCITE-1598 (Pig adapter) does
not
>>>> seem to be on the list.
>>>>
>>>> Thanks,
>>>>
>>>> Eli
>>>>
>>>>
>>>> On Wed, Mar 8, 2017 at 10:40 AM, Julian Hyde <[email protected]> wrote:
>>>>
>>>>> Commit 49888a6c581f6243a5b3f1a48ea2c9edd6aac9d5, which fixed
>>>>> CALCITE-1615, had not made it to https://github.com/julianhyde/
>>>>> calcite/tree/branch-1.12 <https://github.com/julianhyde/calcite/tree/
>>>>> branch-1.12> at the time I wrote that draft of the release notes.
There
>>>>> are about a dozen other cases fixed since then. I’ll be sure to
include it.
>>>>>
>>>>> Julian
>>>>>
>>>>>
>>>>>> On Mar 8, 2017, at 10:29 AM, Haohui Mai <[email protected]> wrote:
>>>>>>
>>>>>> Hi Julian,
>>>>>>
>>>>>> I think CALCITE-1615 is not on the list. Can you please double check?
>>>>>>
>>>>>> Haohui
>>>>>> On Wed, Mar 8, 2017 at 9:49 AM Julian Hyde <[email protected]> wrote:
>>>>>>
>>>>>>> First draft of release notes:
>>>>>>> https://github.com/julianhyde/calcite/blob/branch-1.12/site/
>>>>> _docs/history.md
>>>>>>> <
>>>>>>> https://github.com/julianhyde/calcite/blob/branch-1.12/site/
>>>>> _docs/history.md>
>>>>>>> (contains commits up to 12 days ago).
>>>>>>>
>>>>>>>
>>>>>>>> On Mar 8, 2017, at 9:16 AM, Julian Hyde <[email protected]> wrote:
>>>>>>>>
>>>>>>>> Thanks. That last change should be fairly low-risk, so we should be
>>>>> able
>>>>>>> to move to an RC and vote soon afterwards.
>>>>>>>>
>>>>>>>> Committers, Let’s try to stabilize. I propose we get any “risky”
>>>>> changes
>>>>>>> in by the end of the weekend.
>>>>>>>>
>>>>>>>> By the way, I volunteer to be release manager.
>>>>>>>>
>>>>>>>> Julian
>>>>>>>>
>>>>>>>>> On Mar 8, 2017, at 2:53 AM, Remus Rusanu <[email protected]>
>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> We have only one issue left to resolve, the materialized views
RelNode
>>>>>>> cloning into a new planner. I expect to have a resolution by end of
this
>>>>>>> week, so the all-done ETA from me would be mid-next week, let say
March
>>>>> 15.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> ~Remus
>>>>>>>>>
>>>>>>>>> On 3/7/17, 5:08 PM, "Julian Hyde" <[email protected]> wrote:
>>>>>>>>>
>>>>>>>>> Remus,
>>>>>>>>>
>>>>>>>>> Do you have an ETA for HIVE-15708? Just a guess is fine, but it
helps
>>>>>>> the rest of us know when this release is likely to drop.
>>>>>>>>>
>>>>>>>>> Julian
>>>>>>>>>
>>>>>>>>>> On Feb 28, 2017, at 10:57 AM, Julian Hyde <[email protected]>
wrote:
>>>>>>>>>>
>>>>>>>>>> On Tue, Feb 28, 2017 at 1:40 AM, Remus Rusanu <
>>>>> [email protected]>
>>>>>>> wrote:
>>>>>>>>>>> Put on hold, for how long?
>>>>>>>>>>
>>>>>>>>>> Until someone chimed in with their time constraints, as you just
did.
>>>>>>> :)
>>>>>>>>>>
>>>>>>>>>> We make decisions on-list, but for that we need people to provide
>>>>>>>>>> information on-list.
>>>>>>>>>>
>>>>>>>>>>> [We] did not yet have a 100% successful run for HIVE-15708,
>>>>>>>>>>> but we're close and once we have the changes ready, we
>>>>>>>>>>> cannot do the move unless Calcite does the release.
>>>>>>>>>>
>>>>>>>>>> What's your best guess for when HIVE-15708 will be fixed? We will
>>>>>>>>>> start the RC vote after that.
>>>>>>>>>>
>>>>>>>>>> Julian
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>
>>>
>>>
>>>
>>
>
>
