[jira] [Comment Edited] (HIVE-16888) Upgrade Calcite to 1.13 and Avatica to 1.10
[
https://issues.apache.org/jira/browse/HIVE-16888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064966#comment-16064966
]
Jesus Camacho Rodriguez edited comment on HIVE-16888 at 6/27/17 3:33 PM:
-
.07 picks up the final Calcite 1.13 release.
A couple of notes on the patch:
- Fixed the issues with new version of timestamp. Internal representation in
Calcite has changed in Calcite 1.13 to support larger precision than millis,
thus we needed to adapt to it.
- Set VARCHAR max precision to Integer.MAX_VALUE. Otherwise, Calcite will
reduce the precision of a VARCHAR when we translate the STRING type, as we
represent it in Calcite as VARCHAR(Integer.MAX_VALUE). The change was
introduced as part of the work on timestamp representation with nano precision.
https://github.com/apache/calcite/blob/205af8134857ba312415b9a5b4a48020fe0ce888/core/src/main/java/org/apache/calcite/sql/type/SqlTypeFactoryImpl.java#L59
- Includes workaround for an issue in the {{validate}} method of the
VolcanoPlanner that causes a NPE (recall that VolcanoPlanner is used in MV
rewriting).
As part of CALCITE-1812, there were some changes in the planner and the
RelMetadataQuery object is taken from the cluster of the root in the
{{validate}} method:
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/plan/volcano/VolcanoPlanner.java#L891
However, when we register the first expression, the root of the VolcanoPlanner
is null. Thus, we get the NPE.
Instead, we should take the RelMetadataQuery from the cluster of the
subexpressions:
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/plan/volcano/VolcanoPlanner.java#L905
The workaround is setting the LOG level to INFO for the Calcite planner in the
tests, since the {{validate}} method is only called at DEBUG level. I created
CALCITE-1859 to fix this issue.
was (Author: jcamachorodriguez):
.07 picks up the final Calcite 1.13 release.
A couple of notes on the patch:
- Fixed the issues with new version of timestamp. Internal representation in
Calcite has changed in Calcite 1.13 to support larger precision than millis,
thus we needed to adapt to it.
- Set VARCHAR max precision to Integer.MAX_VALUE. Otherwise, Calcite will
reduce the precision of a VARCHAR when we translate the STRING type, as we
represent it in Calcite as VARCHAR(Integer.MAX_VALUE). The change was
introduced as part of the work on timestamp representation with nano precision.
https://github.com/apache/calcite/blob/205af8134857ba312415b9a5b4a48020fe0ce888/core/src/main/java/org/apache/calcite/sql/type/SqlTypeFactoryImpl.java#L59
- Includes workaround for an issue in the {{validate}} method of the
VolcanoPlanner that causes a NPE (recall that VolcanoPlanner is used in MV
rewriting).
As part of CALCITE-1812, there were some changes in the planner and the
RelMetadataQuery object is taken from the cluster of the root in the
{{validate}} method:
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/plan/volcano/VolcanoPlanner.java#L891
However, when we register the first expression, the root of the VolcanoPlanner
is null. Thus, we get the NPE.
Instead, we should take the RelMetadataQuery from the cluster of the
subexpressions:
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/plan/volcano/VolcanoPlanner.java#L905
The workaround is setting the LOG level to INFO for the Calcite planner in the
tests, since the {{validate}} method is only called at DEBUG level. I will
create a Calcite JIRA to fix this issue.
> Upgrade Calcite to 1.13 and Avatica to 1.10
> ---
>
> Key: HIVE-16888
> URL: https://issues.apache.org/jira/browse/HIVE-16888
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 3.0.0
>Reporter: Remus Rusanu
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-16888.01.patch, HIVE-16888.02.patch,
> HIVE-16888.03.patch, HIVE-16888.04.patch, HIVE-16888.05.patch,
> HIVE-16888.06.patch, HIVE-16888.07.patch
>
>
> I'm creating this early to be able to ptest the current Calcite
> 1.13.0-SNAPSHOT
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HIVE-16888) Upgrade Calcite to 1.13 and Avatica to 1.10
[
https://issues.apache.org/jira/browse/HIVE-16888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16060948#comment-16060948
]
Remus Rusanu edited comment on HIVE-16888 at 6/23/17 2:13 PM:
--
[~jcamachorodriguez] yes, with HIVE-16751 it passes. It also updates, similar
to other HIVE-16888 druid changes:
{noformat}
-druid.query.json
{"queryType":"groupBy","dataSource":"wikipedia","granularity":"all","dimensions":["robot"],"limitSpec":{"type":"default"},"filter":{"type":"selector","dimension":"language","value":"en"},"aggregations":[{"type":"longSum","name":"dummy_agg","fieldName":"dummy_agg"}],"intervals":["1900-01-01T00:00:00.000/3000-01-01T00:00:00.000"]}
+druid.query.json
{"queryType":"groupBy","dataSource":"wikipedia","granularity":"all","dimensions":[{"type":"default","dimension":"robot"}],"limitSpec":{"type":"default"},"filter":{"type":"selector","dimension":"language","value":"en"},"aggregations":[{"type":"longSum","name":"dummy_agg","fieldName":"dummy_agg"}],"intervals":["1900-01-01T00:00:00.000/3000-01-01T00:00:00.000"]}
{noformat}
was (Author: rusanu):
[~jcamachorodriguez] yes, with HIVE-16751 it passes. It also updates, not sure
if from HIVE-16751 or from HIVE-16888:
{noformat}
-druid.query.json
{"queryType":"groupBy","dataSource":"wikipedia","granularity":"all","dimensions":["robot"],"limitSpec":{"type":"default"},"filter":{"type":"selector","dimension":"language","value":"en"},"aggregations":[{"type":"longSum","name":"dummy_agg","fieldName":"dummy_agg"}],"intervals":["1900-01-01T00:00:00.000/3000-01-01T00:00:00.000"]}
+druid.query.json
{"queryType":"groupBy","dataSource":"wikipedia","granularity":"all","dimensions":[{"type":"default","dimension":"robot"}],"limitSpec":{"type":"default"},"filter":{"type":"selector","dimension":"language","value":"en"},"aggregations":[{"type":"longSum","name":"dummy_agg","fieldName":"dummy_agg"}],"intervals":["1900-01-01T00:00:00.000/3000-01-01T00:00:00.000"]}
{noformat}
> Upgrade Calcite to 1.13 and Avatica to 1.10
> ---
>
> Key: HIVE-16888
> URL: https://issues.apache.org/jira/browse/HIVE-16888
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 3.0.0
>Reporter: Remus Rusanu
>Assignee: Remus Rusanu
> Attachments: HIVE-16888.01.patch, HIVE-16888.02.patch,
> HIVE-16888.03.patch, HIVE-16888.04.patch, HIVE-16888.05.patch
>
>
> I'm creating this early to be able to ptest the current Calcite
> 1.13.0-SNAPSHOT
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HIVE-16888) Upgrade Calcite to 1.13 and Avatica to 1.10
[
https://issues.apache.org/jira/browse/HIVE-16888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16060804#comment-16060804
]
Remus Rusanu edited comment on HIVE-16888 at 6/23/17 12:22 PM:
---
{{druid_basic2}} falls with exception:
{noformat}
2017-06-23T05:18:31,795 DEBUG [ecae368c-2c5d-46af-b4b3-4da8b9c0d23b main]
parse.CalcitePlanner: Created Table Plan for druid_table_1 TS[0]
2017-06-23T05:18:31,795 DEBUG [ecae368c-2c5d-46af-b4b3-4da8b9c0d23b main]
exec.FunctionRegistry: Method didn't match: passed = [string] accepted =
[timestamp] method = public org.apache.hadoop.hive.serde2.io.TimestampWritable
org.apache.hadoop.hive.ql.udf.UDFDateFloor.evaluate(org.apache.hadoop.hive.serde2.io.TimestampWritable)
2017-06-23T05:18:31,796 ERROR [ecae368c-2c5d-46af-b4b3-4da8b9c0d23b main]
parse.CalcitePlanner: CBO failed, skipping CBO.
org.apache.hadoop.hive.ql.parse.SemanticException: Line 0:-1 Wrong arguments
'extract': No matching method for class
org.apache.hadoop.hive.ql.udf.UDFDateFloorDay with (string). Possible choices:
_FUNC_(timestamp)
at
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1363)
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.lib.ExpressionWalker.walk(ExpressionWalker.java:76)
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:229)
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:176)
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:11746)
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:11701)
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:11669)
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFilterPlan(SemanticAnalyzer.java:3325)
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFilterPlan(SemanticAnalyzer.java:3305)
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:9695)
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:10652)
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:10530)
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:433)
[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11269)
[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:294)
[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:261)
[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:169)
[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
{noformat}
on this plan:
{noformat}
2017-06-23T05:18:31,790 DEBUG [ecae368c-2c5d-46af-b4b3-4da8b9c0d23b main]
translator.PlanModifierForASTConv: Final plan after modifier
HiveSortLimit(sort0=[$0], dir0=[ASC-nulls-first], fetch=[10])
HiveProject(robot=[$1], __time=[$0])
HiveFilter(condition=[BETWEEN(false, FLOOR_DAY($0, FLAG(DAY)),
CAST(1999-11-01 08:00:00):TIMESTAMP(9), CAST(1999-11-10
08:00:00):TIMESTAMP(9))])
DruidQuery(table=[[default.druid_table_1]],
intervals=[[1900-01-01T00:00:00.000/3000-01-01T00:00:00.000]], groups=[{0, 1}],
aggs=[[]])
{noformat}
was (Author: rusanu):
{{druid_basic2}} falls with exception:
[jira] [Comment Edited] (HIVE-16888) Upgrade Calcite to 1.13 and Avatica to 1.10
[
https://issues.apache.org/jira/browse/HIVE-16888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16060527#comment-16060527
]
Remus Rusanu edited comment on HIVE-16888 at 6/23/17 7:46 AM:
--
patch 04 rebased to current master and updated ~50 GF with diffs only in more
aggressive predicate reduction ({{key > 10 and key > 15}} -> {{key > 15}})
was (Author: rusanu):
patch 04 rebased to current master and updated ~50 GF with diffs only in more
aggresiive predicate reduction ({{ key > 10 and key > 15}} -> {{key > 15}})
> Upgrade Calcite to 1.13 and Avatica to 1.10
> ---
>
> Key: HIVE-16888
> URL: https://issues.apache.org/jira/browse/HIVE-16888
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 3.0.0
>Reporter: Remus Rusanu
>Assignee: Remus Rusanu
> Attachments: HIVE-16888.01.patch, HIVE-16888.02.patch,
> HIVE-16888.03.patch, HIVE-16888.04.patch
>
>
> I'm creating this early to be able to ptest the current Calcite
> 1.13.0-SNAPSHOT
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
