Which version of Spark is this with? Did the logs print something about sending the JAR you added with ADD_JARS to the cluster?
Matei On Sep 8, 2013, at 8:56 AM, Gary Malouf <[email protected]> wrote: > I built a custom jar with among other things, nscalatime and joda time packed > inside of it. Using the ADD_JARS variable, I have added this super jar to my > classpath on the scheduler when running spark-shell. I wrote a function that > grabs protobuf data, filters and then maps each message to a (LocalDate, > Option[String]) format. Unfortunately, this does not run and I get the > following: > > 13/09/08 15:50:43 INFO cluster.TaskSetManager: Finished TID 6 in 348 ms > (progress: 7/576) > Exception in thread "Thread-159" java.lang.ClassNotFoundException: > org.joda.time.LocalDate > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at > scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.scala$tools$nsc$util$ScalaClassLoader$$super$findClass(ScalaClassLoader.scala:88) > at > scala.tools.nsc.util.ScalaClassLoader$class.findClass(ScalaClassLoader.scala:44) > at > scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.findClass(ScalaClassLoader.scala:88) > at java.lang.ClassLoader.loadClass(ClassLoader.java:423) > at > scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.scala$tools$nsc$util$ScalaClassLoader$$super$loadClass(ScalaClassLoader.scala:88) > at > scala.tools.nsc.util.ScalaClassLoader$class.loadClass(ScalaClassLoader.scala:50) > at > scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.loadClass(ScalaClassLoader.scala:88) > at java.lang.ClassLoader.loadClass(ClassLoader.java:356) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:266) > at > spark.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:20) > at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1610) > at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1515) > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1769) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348) > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) > at > it.unimi.dsi.fastutil.objects.Object2LongOpenHashMap.readObject(Object2LongOpenHashMap.java:757) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1004) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1891) > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348) > at > java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1989) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1913) > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348) > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) > at spark.scheduler.TaskResult.readExternal(TaskResult.scala:26) > at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1835) > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1794) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348) > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) > at spark.JavaDeserializationStream.readObject(JavaSerializer.scala:23) > at spark.JavaSerializerInstance.deserialize(JavaSerializer.scala:45) > at > spark.scheduler.cluster.TaskSetManager.taskFinished(TaskSetManager.scala:261) > at > spark.scheduler.cluster.TaskSetManager.statusUpdate(TaskSetManager.scala:236) > at > spark.scheduler.cluster.ClusterScheduler.statusUpdate(ClusterScheduler.scala:219) > at > spark.scheduler.mesos.MesosSchedulerBackend.statusUpdate(MesosSchedulerBackend.scala:264) > 13/09/08 15:50:43 INFO mesos.MesosSchedulerBackend: driver.run() returned > with code DRIVER_ABORTED > > > The code definitely compiles in the interpreter and the executors seem to > find the protobuf messages which are in the same jar - any idea what could be > causing the problem?
