Alexander Bezzubov created ZEPPELIN-98:
------------------------------------------
Summary: Zeppelin does not work on YARN after deps moved from
pom.xml to spark/pom.xml
Key: ZEPPELIN-98
URL: https://issues.apache.org/jira/browse/ZEPPELIN-98
Project: Zeppelin
Issue Type: Bug
Reporter: Alexander Bezzubov
Priority: Critical
Zeppelin does not work on YARN after
https://github.com/apache/incubator-zeppelin/commit/109b0807fce6453b7543c68069f30b7cece0f20e
because of Hadoop version missmatch
in runtime exception looks like
{code}
Exception in thread "main" java.lang.AbstractMethodError:
org.apache.hadoop.yarn.api.records.ContainerId.setContainerId(J)V
at
org.apache.hadoop.yarn.api.records.ContainerId.newInstance(ContainerId.java:60)
at
org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:178)
at
org.apache.spark.deploy.yarn.YarnSparkHadoopUtil.getContainerId(YarnSparkHadoopUtil.scala:142)
at
org.apache.spark.deploy.yarn.YarnRMClient.getAttemptId(YarnRMClient.scala:92)
at
org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:82)
at
org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$main$1.apply$mcV$sp(ApplicationMaster.scala:587)
...
{code}
The reason looks like version Hadoop yarn artefacts version mismatch from below:
{code}
mvn dependency:tree -Pspark-1.4 -Dspark.version=1.4.0-SNAPSHOT -Phadoop-2.6
-Pyarn | grep "org.apache.hadoop:hadoop-yarn-*"
[INFO] | +- org.apache.hadoop:hadoop-yarn-common:jar:2.4.0:compile
[INFO] | +- org.apache.hadoop:hadoop-yarn-server-web-proxy:jar:2.4.0:compile
[INFO] | | +- org.apache.hadoop:hadoop-yarn-server-common:jar:2.4.0:compile
[INFO] | \- org.apache.hadoop:hadoop-yarn-client:jar:2.4.0:compile
[INFO] \- org.apache.hadoop:hadoop-yarn-api:jar:2.6.0:compile
{code}
{code}
mvn dependency:tree -Pyarn | grep "org.apache.hadoop:hadoop-yarn-*"
[INFO] | +- org.apache.hadoop:hadoop-yarn-common:jar:2.2.0:compile
[INFO] | +- org.apache.hadoop:hadoop-yarn-server-web-proxy:jar:2.2.0:compile
[INFO] | | \- org.apache.hadoop:hadoop-yarn-server-common:jar:2.2.0:compile
[INFO] | \- org.apache.hadoop:hadoop-yarn-client:jar:2.2.0:compile
[INFO] \- org.apache.hadoop:hadoop-yarn-api:jar:2.3.0:compile
{code}
{code}
mvn dependency:tree -Pspark-1.4 -Pyarn | grep "org.apache.hadoop:hadoop-yarn-*"
{code}
{code}
mvn dependency:tree -Pspark-1.2 -Pyarn | grep "org.apache.hadoop:hadoop-yarn-*"
[INFO] | +- org.apache.hadoop:hadoop-yarn-common:jar:2.2.0:compile
[INFO] | +- org.apache.hadoop:hadoop-yarn-server-web-proxy:jar:2.2.0:compile
[INFO] | | \- org.apache.hadoop:hadoop-yarn-server-common:jar:2.2.0:compile
[INFO] | \- org.apache.hadoop:hadoop-yarn-client:jar:2.2.0:compile
[INFO] \- org.apache.hadoop:hadoop-yarn-api:jar:2.3.0:compile
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
