I tried to build jar using sbt with below config it works on few use case. But
not sure it’s a full proof solution. Is anyone use yarn for mesos, I believe
this problem will arise. Is this common or something I am facing , I am using
EMR for spark jobs. (Earlier I used mesos I never had this kind of problem
which is understood.)
assemblyShadeRules in assembly := Seq(
ShadeRule.rename("com.google.common.**" -> "shaded.@1").inAll,
ShadeRule.zap("org.apache.hadoop.mapreduce.v2.app.**").inAll
)
assemblyMergeStrategy in assembly := {
case PathList("MRAppMaster.java") => MergeStrategy.discard
case PathList("org.apache.hadoop","MRAppMaster", xs @ _*) =>
MergeStrategy.discard
case PathList("org","aopalliance", xs @ _*) => MergeStrategy.last
case PathList("javax", "inject", xs @ _*) => MergeStrategy.last
case PathList("javax", "servlet", xs @ _*) => MergeStrategy.last
case PathList("javax", "activation", xs @ _*) => MergeStrategy.last
case PathList("org", "apache", xs @ _*) => MergeStrategy.last
case PathList("com", "google", xs @ _*) => MergeStrategy.last
-----Original Message-----
From: Andras Piros [mailto:[email protected]]
Sent: Friday, November 04, 2016 2:21 AM
To: [email protected]
Subject: Re: Spark oozie job failed on launching
Hi Saurabh,
apparently there is some classpath issue along w/ the runtimes used / the
common jar.
In order to track down class loading issues like this one the best you can do
is to give extra parameters like -verbose:class (*reference
<http://www.oracle.com/technetwork/java/javase/clopts-139448.html#gbmtm>*),
or just list the contents of each and every jar file loaded by the jvm trying
to run MRAppMaster.main().
Regards,
Andras
--
Andras PIROS
Software Engineer
<http://www.cloudera.com/>
On Fri, Nov 4, 2016 at 7:10 AM, Saurabh Malviya (samalviy) <
[email protected]> wrote:
> Hi,
>
> Spark oozie jobs keep failing on deployment, although it works fine
> for the first team. We are using common jar for all jobs
> Oozie.libpath= jar
>
> Log Type: syslog
> Log Upload Time: Fri Nov 04 05:24:17 +0000 2016 Log Length: 432
> 2016-11-04 05:24:15,845 ERROR [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster:
> Error starting MRAppMaster
> java.lang.NoSuchMethodError: org.apache.hadoop.yarn.api.
> records.ContainerId.fromString(Ljava/lang/String;)
> Lorg/apache/hadoop/yarn/api/records/ContainerId;
> at org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(
> ConverterUtils.java:179)
> at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(
> MRAppMaster.java:1343)
> ===========================
>
