Hi Andrew, Thanks for the prompt response. I tried command line and it works fine. But, I want to try from IDE for easier debugging and transparency into code execution. I would try and see if there is any way to get the jar over to the executor from within the IDE.
- Barrington > On Sep 21, 2014, at 10:52 PM, Andrew Or <and...@databricks.com> wrote: > > Hi Barrington, > > Have you tried running it from the command line? (i.e. bin/spark-submit > --master yarn-client --class YOUR_CLASS YOUR_JAR) Does it still fail? I am > not super familiar with running Spark through intellij, but the AFAIK the > classpaths are setup a little differently there. Also, Spark submit does this > for you nicely, so if you go through this path you don't even have to call > `setJars` as you did in your application. > > -Andrew > > 2014-09-21 12:52 GMT-07:00 Barrington Henry <barrington.he...@me.com > <mailto:barrington.he...@me.com>>: > Hi, > > I am running spark from my IDE (InteliJ) using YARN as my cluster manager. > However, the executor node is not able to find my main driver class > “LascoScript”. I keep getting java.lang.ClassNotFoundException. > I tried adding the jar of the main class by running the snippet below > > > val conf = new SparkConf().set("spark.driver.host", "barrymac") > .setMaster("yarn-client") > .setAppName("Lasco Script”) > > .setJars(SparkContext.jarOfClass(this.getClass).toSeq) > > But the jarOfClass function returns nothing. See below for logs. > > -------------------- > > 14/09/21 10:53:15 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, > barrymac): java.lang.ClassNotFoundException: LascoScript$$anonfun$1 > java.net.URLClassLoader$1.run(URLClassLoader.java:366) > java.net.URLClassLoader$1.run(URLClassLoader.java:355) > java.security.AccessController.doPrivileged(Native Method) > java.net.URLClassLoader.findClass(URLClassLoader.java:354) > java.lang.ClassLoader.loadClass(ClassLoader.java:423) > java.lang.ClassLoader.loadClass(ClassLoader.java:356) > java.lang.Class.forName0(Native Method) > java.lang.Class.forName(Class.java:264) > > org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:59) > > java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1593) > java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1514) > > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1750) > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347) > > java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1964) > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1888) > > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771) > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347) > > java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1964) > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1888) > > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771) > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347) > java.io.ObjectInputStream.readObject(ObjectInputStream.java:369) > > org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62) > > org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87) > org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:57) > org.apache.spark.scheduler.Task.run(Task.scala:54) > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > java.lang.Thread.run(Thread.java:722) > 14/09/21 10:53:15 INFO TaskSetManager: Lost task 1.0 in stage 0.0 (TID 1) on > executor barrymac: java.lang.ClassNotFoundException (LascoScript$$anonfun$1) > [duplicate 1] > 14/09/21 10:53:15 INFO TaskSetManager: Starting task 1.1 in stage 0.0 (TID 4, > barrymac, NODE_LOCAL, 1312 bytes) > 14/09/21 10:53:15 INFO TaskSetManager: Lost task 2.0 in stage 0.0 (TID 2) on > executor barrymac: java.lang.ClassNotFoundException (LascoScript$$anonfun$1) > [duplicate 2] > 14/09/21 10:53:15 INFO TaskSetManager: Starting task 2.1 in stage 0.0 (TID 5, > barrymac, NODE_LOCAL, 1312 bytes) > 14/09/21 10:53:15 INFO TaskSetManager: Lost task 3.0 in stage 0.0 (TID 3) on > executor barrymac: java.lang.ClassNotFoundException (LascoScript$$anonfun$1) > [duplicate 3] > 14/09/21 10:53:15 INFO TaskSetManager: Starting task 3.1 in stage 0.0 (TID 6, > barrymac, NODE_LOCAL, 1312 bytes) > 14/09/21 10:53:15 INFO TaskSetManager: Lost task 1.1 in stage 0.0 (TID 4) on > executor barrymac: java.lang.ClassNotFoundException (LascoScript$$anonfun$1) > [duplicate 4] > 14/09/21 10:53:15 INFO TaskSetManager: Starting task 1.2 in stage 0.0 (TID 7, > barrymac, NODE_LOCAL, 1312 bytes) > 14/09/21 10:53:15 INFO TaskSetManager: Lost task 2.1 in stage 0.0 (TID 5) on > executor barrymac: java.lang.ClassNotFoundException (LascoScript$$anonfun$1) > [duplicate 5] > 14/09/21 10:53:15 INFO TaskSetManager: Starting task 2.2 in stage 0.0 (TID 8, > barrymac, NODE_LOCAL, 1312 bytes) > 14/09/21 10:53:15 INFO TaskSetManager: Lost task 3.1 in stage 0.0 (TID 6) on > executor barrymac: java.lang.ClassNotFoundException (LascoScript$$anonfun$1) > [duplicate 6] > 14/09/21 10:53:15 INFO TaskSetManager: Starting task 3.2 in stage 0.0 (TID 9, > barrymac, NODE_LOCAL, 1312 bytes) > 14/09/21 10:53:15 INFO TaskSetManager: Lost task 1.2 in stage 0.0 (TID 7) on > executor barrymac: java.lang.ClassNotFoundException (LascoScript$$anonfun$1) > [duplicate 7] > 14/09/21 10:53:15 INFO TaskSetManager: Starting task 1.3 in stage 0.0 (TID > 10, barrymac, NODE_LOCAL, 1312 bytes) > 14/09/21 10:53:15 INFO TaskSetManager: Lost task 2.2 in stage 0.0 (TID 8) on > executor barrymac: java.lang.ClassNotFoundException (LascoScript$$anonfun$1) > [duplicate 8] > 14/09/21 10:53:15 INFO TaskSetManager: Starting task 2.3 in stage 0.0 (TID > 11, barrymac, NODE_LOCAL, 1312 bytes) > 14/09/21 10:53:15 INFO TaskSetManager: Lost task 3.2 in stage 0.0 (TID 9) on > executor barrymac: java.lang.ClassNotFoundException (LascoScript$$anonfun$1) > [duplicate 9] > 14/09/21 10:53:15 INFO TaskSetManager: Starting task 3.3 in stage 0.0 (TID > 12, barrymac, NODE_LOCAL, 1312 bytes) > 14/09/21 10:53:15 INFO TaskSetManager: Lost task 1.3 in stage 0.0 (TID 10) on > executor barrymac: java.lang.ClassNotFoundException (LascoScript$$anonfun$1) > [duplicate 10] > 14/09/21 10:53:15 ERROR TaskSetManager: Task 1 in stage 0.0 failed 4 times; > aborting job > 14/09/21 10:53:15 INFO TaskSetManager: Lost task 2.3 in stage 0.0 (TID 11) on > executor barrymac: java.lang.ClassNotFoundException (LascoScript$$anonfun$1) > [duplicate 11] > 14/09/21 10:53:15 INFO YarnClientClusterScheduler: Cancelling stage 0 > 14/09/21 10:53:15 INFO YarnClientClusterScheduler: Stage 0 was cancelled > 14/09/21 10:53:15 INFO DAGScheduler: Failed to run collect at > LascoScript.scala:69 > Exception in thread "main" org.apache.spark.SparkException: Job aborted due > to stage failure: Task 1 in stage 0.0 failed 4 times, most recent failure: > Lost task 1.3 in stage 0.0 (TID 10, barrymac): > java.lang.ClassNotFoundException: LascoScript$$anonfun$1 > java.net.URLClassLoader$1.run(URLClassLoader.java:366) > java.net.URLClassLoader$1.run(URLClassLoader.java:355) > java.security.AccessController.doPrivileged(Native Method) > java.net.URLClassLoader.findClass(URLClassLoader.java:354) > java.lang.ClassLoader.loadClass(ClassLoader.java:423) > java.lang.ClassLoader.loadClass(ClassLoader.java:356) > java.lang.Class.forName0(Native Method) > java.lang.Class.forName(Class.java:264) > > org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:59) > > java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1593) > java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1514) > > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1750) > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347) > > java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1964) > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1888) > > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771) > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347) > > java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1964) > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1888) > > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771) > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347) > java.io.ObjectInputStream.readObject(ObjectInputStream.java:369) > > org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62) > > org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87) > org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:57) > org.apache.spark.scheduler.Task.run(Task.scala:54) > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > java.lang.Thread.run(Thread.java:722) > Driver stacktrace: > at org.apache.spark.scheduler.DAGScheduler.org > <http://org.apache.spark.scheduler.dagscheduler.org/>$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1185) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1174) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1173) > at > scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) > at > org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1173) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688) > at scala.Option.foreach(Option.scala:236) > at > org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:688) > at > org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1391) > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498) > at akka.actor.ActorCell.invoke(ActorCell.scala:456) > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237) > at akka.dispatch.Mailbox.run(Mailbox.scala:219) > at > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386) > at > scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > at > scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > at > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > at > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > 14/09/21 10:53:15 INFO TaskSetManager: Lost task 3.3 in stage 0.0 (TID 12) on > executor barrymac: java.lang.ClassNotFoundException (LascoScript$$anonfun$1) > [duplicate 12] > 14/09/21 10:53:15 INFO YarnClientClusterScheduler: Removed TaskSet 0.0, whose > tasks have all completed, from pool > > > > > - Barrington > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > <mailto:user-unsubscr...@spark.apache.org> > For additional commands, e-mail: user-h...@spark.apache.org > <mailto:user-h...@spark.apache.org> > >
