[
https://issues.apache.org/jira/browse/OOZIE-2360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14903078#comment-14903078
]
Robert Kanter commented on OOZIE-2360:
--------------------------------------
[~pavan kumar], it's working for me (once I add the mesos jar). Though your
workflow looks fine as far as I can tell. I think we should keep the scope for
this JIRA to just the missing jar. Can you open a new JIRA for the issue you
described in SO? Can you also paste the stdout from the launcher log? The
bulk of the changes in OOZIE-2277 had to do with parsing the local and remote
classpath/distributed cache jars into various arguments for Spark, so there
could be an edge case there that I missed; we probably just need to exclude
something.
> Spark Action fails due to missing mesos jar
> -------------------------------------------
>
> Key: OOZIE-2360
> URL: https://issues.apache.org/jira/browse/OOZIE-2360
> Project: Oozie
> Issue Type: Bug
> Components: action
> Affects Versions: trunk
> Reporter: Robert Kanter
> Assignee: Robert Kanter
> Priority: Blocker
> Fix For: trunk
>
> Attachments: OOZIE-2360.001.patch
>
>
> When doing OOZIE-2277, I redid the Spark dependencies, and excluded a mesos
> jar (mesos-0.18.1-shaded-protobuf.jar) figuring we don't need that in a
> Hadoop/Yarn/MR environment. It turns out that Spark actually needs this.
> When testing this out for OOZIE-2277, I must have accidentally put the mesos
> jar in the sharelib or something, which is why I didn't see this before.
> {noformat}
> 2015-09-11 14:06:56,398 ERROR [sparkExecutor-akka.actor.default-dispatcher-5]
> actor.ActorSystemImpl (Slf4jLogger.scala:apply$mcV$sp(66)) - Uncaught fatal
> error from thread [sparkExecutor-akka.actor.default-dispatcher-3] shutting
> down ActorSystem [sparkExecutor]
> java.lang.NoClassDefFoundError: org/apache/mesos/Protos$TaskState
> at java.lang.Class.getDeclaredMethods0(Native Method)
> at java.lang.Class.privateGetDeclaredMethods(Class.java:2570)
> at java.lang.Class.getDeclaredMethod(Class.java:2002)
> at
> java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:1431)
> at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:72)
> at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:494)
> at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:468)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:468)
> at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:365)
> at
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1133)
> at
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
> at
> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
> at
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
> at
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
> at
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
> at
> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
> at
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
> at
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
> at
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
> at
> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
> at
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
> at
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
> at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
> at
> akka.serialization.JavaSerializer$$anonfun$toBinary$1.apply$mcV$sp(Serializer.scala:129)
> at
> akka.serialization.JavaSerializer$$anonfun$toBinary$1.apply(Serializer.scala:129)
> at
> akka.serialization.JavaSerializer$$anonfun$toBinary$1.apply(Serializer.scala:129)
> at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
> at akka.serialization.JavaSerializer.toBinary(Serializer.scala:129)
> at
> akka.remote.MessageSerializer$.serialize(MessageSerializer.scala:36)
> at
> akka.remote.EndpointWriter$$anonfun$akka$remote$EndpointWriter$$serializeMessage$1.apply(Endpoint.scala:672)
> at
> akka.remote.EndpointWriter$$anonfun$akka$remote$EndpointWriter$$serializeMessage$1.apply(Endpoint.scala:672)
> at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
> at
> akka.remote.EndpointWriter.akka$remote$EndpointWriter$$serializeMessage(Endpoint.scala:671)
> at
> akka.remote.EndpointWriter$$anonfun$7.applyOrElse(Endpoint.scala:559)
> at
> akka.remote.EndpointWriter$$anonfun$7.applyOrElse(Endpoint.scala:544)
> at
> scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)
> at akka.actor.FSM$class.processEvent(FSM.scala:595)
> at akka.remote.EndpointWriter.processEvent(Endpoint.scala:443)
> at akka.actor.FSM$class.akka$actor$FSM$$processMsg(FSM.scala:589)
> at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:583)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
> at akka.actor.ActorCell.invoke(ActorCell.scala:456)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
> at akka.dispatch.Mailbox.run(Mailbox.scala:219)
> at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
> at
> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> Caused by: java.lang.ClassNotFoundException: org.apache.mesos.Protos$TaskState
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> ... 50 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
