Re: java.lang.ClassNotFoundException on driver class in executor
Hi Andrew,
Thanks for the prompt response. I tried command line and it works fine. But, I
want to try from IDE for easier debugging and transparency into code execution.
I would try and see if there is any way to get the jar over to the executor
from within the IDE.
- Barrington
> On Sep 21, 2014, at 10:52 PM, Andrew Or wrote:
>
> Hi Barrington,
>
> Have you tried running it from the command line? (i.e. bin/spark-submit
> --master yarn-client --class YOUR_CLASS YOUR_JAR) Does it still fail? I am
> not super familiar with running Spark through intellij, but the AFAIK the
> classpaths are setup a little differently there. Also, Spark submit does this
> for you nicely, so if you go through this path you don't even have to call
> `setJars` as you did in your application.
>
> -Andrew
>
> 2014-09-21 12:52 GMT-07:00 Barrington Henry <mailto:barrington.he...@me.com>>:
> Hi,
>
> I am running spark from my IDE (InteliJ) using YARN as my cluster manager.
> However, the executor node is not able to find my main driver class
> “LascoScript”. I keep getting java.lang.ClassNotFoundException.
> I tried adding the jar of the main class by running the snippet below
>
>
>val conf = new SparkConf().set("spark.driver.host", "barrymac")
> .setMaster("yarn-client")
> .setAppName("Lasco Script”)
>
> .setJars(SparkContext.jarOfClass(this.getClass).toSeq)
>
> But the jarOfClass function returns nothing. See below for logs.
>
>
>
> 14/09/21 10:53:15 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0,
> barrymac): java.lang.ClassNotFoundException: LascoScript$$anonfun$1
> java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> java.security.AccessController.doPrivileged(Native Method)
> java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> java.lang.ClassLoader.loadClass(ClassLoader.java:423)
> java.lang.ClassLoader.loadClass(ClassLoader.java:356)
> java.lang.Class.forName0(Native Method)
> java.lang.Class.forName(Class.java:264)
>
> org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:59)
>
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1593)
> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1514)
>
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1750)
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
>
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1964)
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1888)
>
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
>
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1964)
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1888)
>
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
> java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
>
> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
>
> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:57)
> org.apache.spark.scheduler.Task.run(Task.scala:54)
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> java.lang.Thread.run(Thread.java:722)
> 14/09/21 10:53:15 INFO TaskSetManager: Lost task 1.0 in stage 0.0 (TID 1) on
> executor barrymac: java.lang.ClassNotFoundException (LascoScript$$anonfun$1)
> [duplicate 1]
> 14/09/21 10:53:15 INFO TaskSetManager: Starting task 1.1 in stage 0.0 (TID 4,
> barrymac, NODE_LOCAL, 1312 bytes)
> 14/09/21 10:53:15 INFO TaskSetManager: Lost task 2.0 in stage 0.0 (TID 2) on
> executor barrymac: java.lang.ClassNotFoundException (LascoScript$$anonfun$1)
> [duplicate 2]
> 14/09/21 10:53:15 INFO TaskSetManager: Starting task 2.1 in stage 0.0 (TID 5,
> barrymac, NODE_LOCAL, 1312 bytes)
> 14/09/21 10:53:15 INFO TaskSetManager: Lost task 3.0 in st
java.lang.ClassNotFoundException on driver class in executor
ka.actor.ActorCell.invoke(ActorCell.scala:456) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237) at akka.dispatch.Mailbox.run(Mailbox.scala:219) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 14/09/21 10:53:15 INFO TaskSetManager: Lost task 3.3 in stage 0.0 (TID 12) on executor barrymac: java.lang.ClassNotFoundException (LascoScript$$anonfun$1) [duplicate 12] 14/09/21 10:53:15 INFO YarnClientClusterScheduler: Removed TaskSet 0.0, whose tasks have all completed, from pool - Barrington - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
YARN mode not available error
Hi,
I am running Spark in cluster mode with Hadoop YARN as the underlying
cluster manager. I get this error when trying to initialize the
SparkContext.
Exception in thread "main" org.apache.spark.SparkException: YARN mode not
available ?
at
org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:1586)
at org.apache.spark.SparkContext.(SparkContext.scala:310)
at org.apache.spark.SparkContext.(SparkContext.scala:86)
at LascoScript$.main(LascoScript.scala:24)
at LascoScript.main(LascoScript.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
Caused by: java.lang.ClassNotFoundException:
org.apache.spark.scheduler.cluster.YarnClientClusterScheduler
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:190)
at
org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:1580)
My build.sbt file looks like this:
name := "LascoScript"
version := "1.0"
scalaVersion := "2.10.4"
val excludeJBossNetty = ExclusionRule(organization = "org.jboss.netty")
val excludeMortbayJetty = ExclusionRule(organization = "org.eclipse.jetty",
artifact = "jetty-server")
val excludeAsm = ExclusionRule(organization = "org.ow2.asm")
val excludeCommonsLogging = ExclusionRule(organization = "commons-logging")
val excludeSLF4J = ExclusionRule(organization = "org.slf4j")
val excludeOldAsm = ExclusionRule(organization = "asm")
val excludeServletApi = ExclusionRule(organization = "javax.servlet",
artifact = "servlet-api")
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.1.0"
excludeAll(
excludeServletApi, excludeMortbayJetty
)
libraryDependencies += "org.apache.hadoop" % "hadoop-client" % "2.5.1"
excludeAll(
excludeJBossNetty, excludeMortbayJetty, excludeAsm, excludeCommonsLogging,
excludeSLF4J, excludeOldAsm, excludeServletApi
)
libraryDependencies += "org.mortbay.jetty" % "servlet-api" % "3.0.20100224"
libraryDependencies += "org.eclipse.jetty" % "jetty-server" %
"8.1.16.v20140903"
unmanagedJars in Compile ++= {
val base = baseDirectory.value
val baseDirectories = (base / "lib") +++ (base)
val customJars = (baseDirectories ** "*.jar")
customJars.classpath
}
resolvers += "Akka Repository" at "http://repo.akka.io/releases/“
How can I fix this issue?
- Barrington
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/YARN-mode-not-available-error-tp14420.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
