[
https://issues.apache.org/jira/browse/HIVE-22126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17175452#comment-17175452
]
zhengchenyu edited comment on HIVE-22126 at 8/11/20, 11:02 AM:
---------------------------------------------------------------
When I run the program, I found an AbstractMethodError. I think
HiveAggregateFactoryImpl's createAggregate doesn't implement AggregateFactory,
I think com.google.common.collect.ImmutableList would be relocation to
org.apache.hive.com.google.common.collect.ImmutableList, so throw
AbstractMethodError.
Notes: for shade all guava jar, I remove guava lib from lib dir and use
maven-shade-plugin in main pom.xml to shade all guava.
Error stack are below:
{code:java}
2020-08-11T18:26:51,434 ERROR [0ae58217-4908-4697-a9e7-a57f279a22a0 main]
parse.CalcitePlanner: CBO failed, skipping CBO.
java.lang.RuntimeException: java.lang.AbstractMethodError:
org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories$HiveAggregateFactoryImpl.createAggregate(Lorg/apache/calcite/rel/RelNode;ZLorg/apache/calcite/util/ImmutableBitSet;Lcom/google/common/collect/ImmutableList;Ljava/util/List;)Lorg/apache/calcite/rel/RelNode;
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.rethrowCalciteException(CalcitePlanner.java:1539)
~[hive-exec-3.1.2.jar:3.1.2]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1417)
~[hive-exec-3.1.2.jar:3.1.2]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedAST(CalcitePlanner.java:1430)
~[hive-exec-3.1.2.jar:3.1.2]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:450)
~[hive-exec-3.1.2.jar:3.1.2]
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12164)
~[hive-exec-3.1.2.jar:3.1.2]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:330)
~[hive-exec-3.1.2.jar:3.1.2]
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:285)
~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:659)
~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1826)
~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1773)
~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1768)
~[hive-exec-3.1.2.jar:3.1.2]
at
org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
~[hive-exec-3.1.2.jar:3.1.2]
at
org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:214)
~[hive-exec-3.1.2.jar:3.1.2]
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
~[hive-cli-3.1.2.jar:3.1.2]
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
~[hive-cli-3.1.2.jar:3.1.2]
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
~[hive-cli-3.1.2.jar:3.1.2]
at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
~[hive-cli-3.1.2.jar:3.1.2]
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
~[hive-cli-3.1.2.jar:3.1.2]
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
~[hive-cli-3.1.2.jar:3.1.2]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_241]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_241]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_241]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_241]
at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
~[hadoop-common-3.2.1.jar:?]
at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
~[hadoop-common-3.2.1.jar:?]
Caused by: java.lang.AbstractMethodError:
org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories$HiveAggregateFactoryImpl.createAggregate(Lorg/apache/calcite/rel/RelNode;ZLorg/apache/calcite/util/ImmutableBitSet;Lcom/google/common/collect/ImmutableList;Ljava/util/List;)Lorg/apache/calcite/rel/RelNode;
at org.apache.calcite.tools.RelBuilder.aggregate(RelBuilder.java:1267)
~[calcite-core-1.16.0.jar:1.16.0]
at
org.apache.calcite.sql2rel.RelFieldTrimmer.trimFields(RelFieldTrimmer.java:886)
~[calcite-core-1.16.0.jar:1.16.0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_241]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_241]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_241]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_241]
at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:524)
~[calcite-core-1.16.0.jar:1.16.0]
at
org.apache.calcite.sql2rel.RelFieldTrimmer.dispatchTrimFields(RelFieldTrimmer.java:273)
~[calcite-core-1.16.0.jar:1.16.0]
at
org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelFieldTrimmer.trimChild(HiveRelFieldTrimmer.java:125)
~[hive-exec-3.1.2.jar:3.1.2]
at
org.apache.calcite.sql2rel.RelFieldTrimmer.trimFields(RelFieldTrimmer.java:377)
~[calcite-core-1.16.0.jar:1.16.0]
at
org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelFieldTrimmer.trimFields(HiveRelFieldTrimmer.java:315)
~[hive-exec-3.1.2.jar:3.1.2]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_241]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_241]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_241]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_241]
at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:524)
~[calcite-core-1.16.0.jar:1.16.0]
at
org.apache.calcite.sql2rel.RelFieldTrimmer.dispatchTrimFields(RelFieldTrimmer.java:273)
~[calcite-core-1.16.0.jar:1.16.0]
at
org.apache.calcite.sql2rel.RelFieldTrimmer.trim(RelFieldTrimmer.java:162)
~[calcite-core-1.16.0.jar:1.16.0]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1692)
~[hive-exec-3.1.2.jar:3.1.2]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1609)
~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.calcite.tools.Frameworks$1.apply(Frameworks.java:118)
~[calcite-core-1.16.0.jar:1.16.0]
at
org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:1052)
~[calcite-core-1.16.0.jar:1.16.0]
at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:154)
~[calcite-core-1.16.0.jar:1.16.0]
at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:111)
~[calcite-core-1.16.0.jar:1.16.0]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1414)
~[hive-exec-3.1.2.jar:3.1.2]
... 23 more
{code}
was (Author: zhengchenyu):
When I run the program, I found an AbstractMethodError. I think
HiveAggregateFactoryImpl's createAggregate doesn't implement AggregateFactory,
I think com.google.common.collect.ImmutableList would be relocation to
org.apache.hive.com.google.common.collect.ImmutableList, so throw
AbstractMethodError.
Notes: for shade all guava jar, I remove guava lib from lib dir and use
maven-shade-plugin in main pom.xml to shade all guava.
Error stack are below:
{code}
2020-08-11T18:26:51,434 ERROR [0ae58217-4908-4697-a9e7-a57f279a22a0 main]
parse.CalcitePlanner: CBO failed, skipping CBO.2020-08-11T18:26:51,434 ERROR
[0ae58217-4908-4697-a9e7-a57f279a22a0 main] parse.CalcitePlanner: CBO failed,
skipping CBO.java.lang.RuntimeException: java.lang.AbstractMethodError:
org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories$HiveAggregateFactoryImpl.createAggregate(Lorg/apache/calcite/rel/RelNode;ZLorg/apache/calcite/util/ImmutableBitSet;Lcom/google/common/collect/ImmutableList;Ljava/util/List;)Lorg/apache/calcite/rel/RelNode;
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.rethrowCalciteException(CalcitePlanner.java:1539)
~[hive-exec-3.1.2.jar:3.1.2] at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1417)
~[hive-exec-3.1.2.jar:3.1.2] at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedAST(CalcitePlanner.java:1430)
~[hive-exec-3.1.2.jar:3.1.2] at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:450)
~[hive-exec-3.1.2.jar:3.1.2] at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12164)
~[hive-exec-3.1.2.jar:3.1.2] at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:330)
~[hive-exec-3.1.2.jar:3.1.2] at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:285)
~[hive-exec-3.1.2.jar:3.1.2] at
org.apache.hadoop.hive.ql.Driver.compile(Driver.java:659)
~[hive-exec-3.1.2.jar:3.1.2] at
org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1826)
~[hive-exec-3.1.2.jar:3.1.2] at
org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1773)
~[hive-exec-3.1.2.jar:3.1.2] at
org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1768)
~[hive-exec-3.1.2.jar:3.1.2] at
org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
~[hive-exec-3.1.2.jar:3.1.2] at
org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:214)
~[hive-exec-3.1.2.jar:3.1.2] at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
~[hive-cli-3.1.2.jar:3.1.2] at
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
~[hive-cli-3.1.2.jar:3.1.2] at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
~[hive-cli-3.1.2.jar:3.1.2] at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
~[hive-cli-3.1.2.jar:3.1.2] at
org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
~[hive-cli-3.1.2.jar:3.1.2] at
org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
~[hive-cli-3.1.2.jar:3.1.2] at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_241] at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_241] at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_241] at java.lang.reflect.Method.invoke(Method.java:498)
~[?:1.8.0_241] at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
~[hadoop-common-3.2.1.jar:?] at
org.apache.hadoop.util.RunJar.main(RunJar.java:236)
~[hadoop-common-3.2.1.jar:?]Caused by: java.lang.AbstractMethodError:
org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories$HiveAggregateFactoryImpl.createAggregate(Lorg/apache/calcite/rel/RelNode;ZLorg/apache/calcite/util/ImmutableBitSet;Lcom/google/common/collect/ImmutableList;Ljava/util/List;)Lorg/apache/calcite/rel/RelNode;
at org.apache.calcite.tools.RelBuilder.aggregate(RelBuilder.java:1267)
~[calcite-core-1.16.0.jar:1.16.0] at
org.apache.calcite.sql2rel.RelFieldTrimmer.trimFields(RelFieldTrimmer.java:886)
~[calcite-core-1.16.0.jar:1.16.0] at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_241] at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_241] at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_241] at java.lang.reflect.Method.invoke(Method.java:498)
~[?:1.8.0_241] at
org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:524)
~[calcite-core-1.16.0.jar:1.16.0] at
org.apache.calcite.sql2rel.RelFieldTrimmer.dispatchTrimFields(RelFieldTrimmer.java:273)
~[calcite-core-1.16.0.jar:1.16.0] at
org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelFieldTrimmer.trimChild(HiveRelFieldTrimmer.java:125)
~[hive-exec-3.1.2.jar:3.1.2] at
org.apache.calcite.sql2rel.RelFieldTrimmer.trimFields(RelFieldTrimmer.java:377)
~[calcite-core-1.16.0.jar:1.16.0] at
org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelFieldTrimmer.trimFields(HiveRelFieldTrimmer.java:315)
~[hive-exec-3.1.2.jar:3.1.2] at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_241] at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_241] at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_241] at java.lang.reflect.Method.invoke(Method.java:498)
~[?:1.8.0_241] at
org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:524)
~[calcite-core-1.16.0.jar:1.16.0] at
org.apache.calcite.sql2rel.RelFieldTrimmer.dispatchTrimFields(RelFieldTrimmer.java:273)
~[calcite-core-1.16.0.jar:1.16.0] at
org.apache.calcite.sql2rel.RelFieldTrimmer.trim(RelFieldTrimmer.java:162)
~[calcite-core-1.16.0.jar:1.16.0] at
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1692)
~[hive-exec-3.1.2.jar:3.1.2] at
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1609)
~[hive-exec-3.1.2.jar:3.1.2] at
org.apache.calcite.tools.Frameworks$1.apply(Frameworks.java:118)
~[calcite-core-1.16.0.jar:1.16.0] at
org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:1052)
~[calcite-core-1.16.0.jar:1.16.0] at
org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:154)
~[calcite-core-1.16.0.jar:1.16.0] at
org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:111)
~[calcite-core-1.16.0.jar:1.16.0] at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1414)
~[hive-exec-3.1.2.jar:3.1.2] ... 23 more
{code}
> hive-exec packaging should shade guava
> --------------------------------------
>
> Key: HIVE-22126
> URL: https://issues.apache.org/jira/browse/HIVE-22126
> Project: Hive
> Issue Type: Bug
> Reporter: Vihang Karajgaonkar
> Assignee: Eugene Chung
> Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22126.01.patch, HIVE-22126.02.patch,
> HIVE-22126.03.patch, HIVE-22126.04.patch, HIVE-22126.05.patch,
> HIVE-22126.06.patch, HIVE-22126.07.patch, HIVE-22126.08.patch,
> HIVE-22126.09.patch, HIVE-22126.09.patch, HIVE-22126.09.patch,
> HIVE-22126.09.patch, HIVE-22126.09.patch
>
>
> The ql/pom.xml includes complete guava library into hive-exec.jar
> https://github.com/apache/hive/blob/master/ql/pom.xml#L990 This causes a
> problems for downstream clients of hive which have hive-exec.jar in their
> classpath since they are pinned to the same guava version as that of hive.
> We should shade guava classes so that other components which depend on
> hive-exec can independently use a different version of guava as needed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
