Sorry Ted, that was me clumsily trying to redact my organization's name from the computer output (in my e-mail editor). I can assure you that basically defend7 and rick are the same thing in this case so the class is present in the jar.
On Mon Feb 23 2015 at 9:39:09 PM Ted Yu <yuzhih...@gmail.com> wrote: > The classname given in stack trace was com.rick.reports.Reports > > In the output from jar command the class is com.defend7.reports.Reports. > > FYI > > On Mon, Feb 23, 2015 at 9:33 PM, necro351 . <necro...@gmail.com> wrote: > >> Hi Ted, >> >> Yes it appears to be: >> rick@ubuntu:~/go/src/rick/sparksprint/containers/tests/StreamingReports$ >> jar tvf >> ../../../analyzer/spark/target/scala-2.10/rick-processors-assembly-1.0.jar|grep >> SensorReports >> 1128 Mon Feb 23 17:34:46 PST 2015 >> com/defend7/reports/Reports$SensorReports$1.class >> 13507 Mon Feb 23 17:34:46 PST 2015 >> com/defend7/reports/Reports$SensorReports$Builder.class >> 10640 Mon Feb 23 17:34:46 PST 2015 >> com/defend7/reports/Reports$SensorReports.class >> 815 Mon Feb 23 17:34:46 PST 2015 >> com/defend7/reports/Reports$SensorReportsOrBuilder.class >> >> >> On Mon Feb 23 2015 at 8:57:18 PM Ted Yu <yuzhih...@gmail.com> wrote: >> >>> bq. Caused by: java.lang.ClassNotFoundException: com.rick.reports. >>> Reports$SensorReports >>> >>> Is Reports$SensorReports class in rick-processors-assembly-1.0.jar ? >>> >>> Thanks >>> >>> On Mon, Feb 23, 2015 at 8:43 PM, necro351 . <necro...@gmail.com> wrote: >>> >>>> Hello, >>>> >>>> I am trying to deserialize some data encoded using proto buff from >>>> within Spark and am getting class-not-found exceptions. I have narrowed the >>>> program down to something very simple that shows the problem exactly (see >>>> 'The Program' below) and hopefully someone can tell me the easy fix :) >>>> >>>> So the situation is I have some proto buff reports in /tmp/reports. I >>>> also have a Spark project with the below Scala code (under The Program) as >>>> well as a Java file defining SensorReports all in the same src sub-tree in >>>> my Spark project. Its built using sbt in the standard way. The Spark job >>>> reads in the reports from /tmp/reports and then prints them to the console. >>>> When I build and run my spark job with spark-submit everything works as >>>> expected and the reports are printed out. When I uncomment the 'XXX' >>>> variant in the Scala spark program and try to print the reports from within >>>> a Spark Context I get the class-not-found exceptions. I don't understand >>>> why. If I get this working then I will want to do more than just print the >>>> reports from within the Spark Context. >>>> >>>> My read of the documentation tells me that my spark job should have >>>> access to everything in the submitted jar and that jar includes the Java >>>> code generated by the proto buff library which defines SensorReports. This >>>> is the spark-submit invocation I use after building my job as an assembly >>>> with the sbt-assembly plugin: >>>> >>>> spark-submit --class com.rick.processors.NewReportProcessor --master >>>> local[*] >>>> ../../../analyzer/spark/target/scala-2.10/rick-processors-assembly-1.0.jar >>>> >>>> I have also tried adding the jar programmatically using sc.addJar but >>>> that does not help. I found a bug from July ( >>>> https://github.com/apache/spark/pull/181) that seems related but it >>>> went into Spark 1.2.0 (which is what I am currently using) so I don't think >>>> that's it. >>>> >>>> Any ideas? Thanks! >>>> >>>> The Program: >>>> ========== >>>> package com.rick.processors >>>> >>>> >>>> >>>> import java.io.File >>>> >>>> import java.nio.file.{Path, Files, FileSystems} >>>> >>>> import org.apache.spark.{SparkContext, SparkConf} >>>> >>>> import com.rick.reports.Reports.SensorReports >>>> >>>> >>>> >>>> object NewReportProcessor { >>>> >>>> private val sparkConf = new SparkConf().setAppName("ReportProcessor") >>>> >>>> private val sc = new SparkContext(sparkConf) >>>> >>>> >>>> >>>> def main(args: Array[String]) = { >>>> >>>> val protoBuffsBinary = localFileReports() >>>> >>>> val sensorReportsBundles = protoBuffsBinary.map(bundle => >>>> SensorReports.parseFrom(bundle)) >>>> // XXX: Printing from within the SparkContext throws >>>> class-not-found >>>> // exceptions, why? >>>> >>>> // sc.makeRDD(sensorReportsBundles).foreach((x: SensorReports) => >>>> println(x.toString)) >>>> sensorReportsBundles.foreach((x: SensorReports) => >>>> println(x.toString)) >>>> } >>>> >>>> >>>> >>>> private def localFileReports() = { >>>> >>>> val reportDir = new File("/tmp/reports") >>>> >>>> val reportFiles = >>>> reportDir.listFiles.filter(_.getName.endsWith(".report")) >>>> >>>> reportFiles.map(file => { >>>> >>>> val path = FileSystems.getDefault().getPath("/tmp/reports", >>>> file.getName()) >>>> Files.readAllBytes(path) >>>> >>>> }) >>>> >>>> } >>>> >>>> } >>>> >>>> The Class-not-found exceptions: >>>> ========================= >>>> Spark assembly has been built with Hive, including Datanucleus jars on >>>> classpath >>>> Using Spark's default log4j profile: >>>> org/apache/spark/log4j-defaults.properties >>>> 15/02/23 17:35:03 WARN Utils: Your hostname, ubuntu resolves to a >>>> loopback address: 127.0.1.1; using 192.168.241.128 instead (on interface >>>> eth0) >>>> 15/02/23 17:35:03 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to >>>> another address >>>> 15/02/23 17:35:04 INFO SecurityManager: Changing view acls to: rick >>>> 15/02/23 17:35:04 INFO SecurityManager: Changing modify acls to: rick >>>> 15/02/23 17:35:04 INFO SecurityManager: SecurityManager: authentication >>>> disabled; ui acls disabled; users with view permissions: Set(rick); users >>>> with modify permissions: Set(rick) >>>> 15/02/23 17:35:04 INFO Slf4jLogger: Slf4jLogger started >>>> 15/02/23 17:35:04 INFO Remoting: Starting remoting >>>> 15/02/23 17:35:04 INFO Remoting: Remoting started; listening on >>>> addresses :[akka.tcp://sparkDriver@192.168.241.128:38110] >>>> 15/02/23 17:35:04 INFO Utils: Successfully started service >>>> 'sparkDriver' on port 38110. >>>> 15/02/23 17:35:04 INFO SparkEnv: Registering MapOutputTracker >>>> 15/02/23 17:35:04 INFO SparkEnv: Registering BlockManagerMaster >>>> 15/02/23 17:35:04 INFO DiskBlockManager: Created local directory at >>>> /tmp/spark-local-20150223173504-b26c >>>> 15/02/23 17:35:04 INFO MemoryStore: MemoryStore started with capacity >>>> 267.3 MB >>>> 15/02/23 17:35:05 WARN NativeCodeLoader: Unable to load native-hadoop >>>> library for your platform... using builtin-java classes where applicable >>>> 15/02/23 17:35:05 INFO HttpFileServer: HTTP File server directory is >>>> /tmp/spark-c77dbc9a-d626-4991-a9b7-f593acafbe64 >>>> 15/02/23 17:35:05 INFO HttpServer: Starting HTTP Server >>>> 15/02/23 17:35:05 INFO Utils: Successfully started service 'HTTP file >>>> server' on port 50950. >>>> 15/02/23 17:35:05 WARN Utils: Service 'SparkUI' could not bind on port >>>> 4040. Attempting port 4041. >>>> 15/02/23 17:35:05 WARN Utils: Service 'SparkUI' could not bind on port >>>> 4041. Attempting port 4042. >>>> 15/02/23 17:35:05 WARN Utils: Service 'SparkUI' could not bind on port >>>> 4042. Attempting port 4043. >>>> 15/02/23 17:35:06 WARN Utils: Service 'SparkUI' could not bind on port >>>> 4043. Attempting port 4044. >>>> 15/02/23 17:35:06 WARN Utils: Service 'SparkUI' could not bind on port >>>> 4044. Attempting port 4045. >>>> 15/02/23 17:35:06 WARN Utils: Service 'SparkUI' could not bind on port >>>> 4045. Attempting port 4046. >>>> 15/02/23 17:35:06 INFO Utils: Successfully started service 'SparkUI' on >>>> port 4046. >>>> 15/02/23 17:35:06 INFO SparkUI: Started SparkUI at >>>> http://192.168.241.128:4046 >>>> 15/02/23 17:35:06 INFO SparkContext: Added JAR >>>> file:/home/rick/go/src/rick/sparksprint/containers/tests/StreamingReports/../../../analyzer/spark/target/scala-2.10/rick-processors-assembly-1.0.jar >>>> at http://192.168.241.128:50950/jars/rick-processors-assembly-1.0.jar >>>> with timestamp 1424741706610 >>>> 15/02/23 17:35:06 INFO AkkaUtils: Connecting to HeartbeatReceiver: >>>> akka.tcp://sparkDriver@192.168.241.128:38110/user/HeartbeatReceiver >>>> 15/02/23 17:35:07 INFO NettyBlockTransferService: Server created on >>>> 57801 >>>> 15/02/23 17:35:07 INFO BlockManagerMaster: Trying to register >>>> BlockManager >>>> 15/02/23 17:35:07 INFO BlockManagerMasterActor: Registering block >>>> manager localhost:57801 with 267.3 MB RAM, BlockManagerId(<driver>, >>>> localhost, 57801) >>>> 15/02/23 17:35:07 INFO BlockManagerMaster: Registered BlockManager >>>> 15/02/23 17:35:07 INFO SparkContext: Starting job: foreach at >>>> NewReportProcessor.scala:17 >>>> 15/02/23 17:35:07 INFO DAGScheduler: Got job 0 (foreach at >>>> NewReportProcessor.scala:17) with 1 output partitions (allowLocal=false) >>>> 15/02/23 17:35:07 INFO DAGScheduler: Final stage: Stage 0(foreach at >>>> NewReportProcessor.scala:17) >>>> 15/02/23 17:35:07 INFO DAGScheduler: Parents of final stage: List() >>>> 15/02/23 17:35:07 INFO DAGScheduler: Missing parents: List() >>>> 15/02/23 17:35:07 INFO DAGScheduler: Submitting Stage 0 >>>> (ParallelCollectionRDD[0] at makeRDD at NewReportProcessor.scala:17), which >>>> has no missing parents >>>> 15/02/23 17:35:07 INFO MemoryStore: ensureFreeSpace(1360) called with >>>> curMem=0, maxMem=280248975 >>>> 15/02/23 17:35:07 INFO MemoryStore: Block broadcast_0 stored as values >>>> in memory (estimated size 1360.0 B, free 267.3 MB) >>>> 15/02/23 17:35:07 INFO MemoryStore: ensureFreeSpace(1071) called with >>>> curMem=1360, maxMem=280248975 >>>> 15/02/23 17:35:07 INFO MemoryStore: Block broadcast_0_piece0 stored as >>>> bytes in memory (estimated size 1071.0 B, free 267.3 MB) >>>> 15/02/23 17:35:07 INFO BlockManagerInfo: Added broadcast_0_piece0 in >>>> memory on localhost:57801 (size: 1071.0 B, free: 267.3 MB) >>>> 15/02/23 17:35:07 INFO BlockManagerMaster: Updated info of block >>>> broadcast_0_piece0 >>>> 15/02/23 17:35:07 INFO SparkContext: Created broadcast 0 from broadcast >>>> at DAGScheduler.scala:838 >>>> 15/02/23 17:35:07 INFO DAGScheduler: Submitting 1 missing tasks from >>>> Stage 0 (ParallelCollectionRDD[0] at makeRDD at >>>> NewReportProcessor.scala:17) >>>> 15/02/23 17:35:07 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 >>>> tasks >>>> 15/02/23 17:35:07 INFO TaskSetManager: Starting task 0.0 in stage 0.0 >>>> (TID 0, localhost, PROCESS_LOCAL, 5587 bytes) >>>> 15/02/23 17:35:07 INFO Executor: Running task 0.0 in stage 0.0 (TID 0) >>>> 15/02/23 17:35:07 INFO Executor: Fetching >>>> http://192.168.241.128:50950/jars/rick-processors-assembly-1.0.jar >>>> with timestamp 1424741706610 >>>> 15/02/23 17:35:08 INFO Utils: Fetching >>>> http://192.168.241.128:50950/jars/rick-processors-assembly-1.0.jar to >>>> /tmp/fetchFileTemp2793880583189398319.tmp >>>> 15/02/23 17:35:08 INFO Executor: Adding >>>> file:/tmp/spark-bdec3945-52d1-42bf-8b7a-30f14f492a42/rick-processors-assembly-1.0.jar >>>> to class loader >>>> 15/02/23 17:35:08 ERROR Executor: Exception in task 0.0 in stage 0.0 >>>> (TID 0) >>>> java.io.IOException: java.lang.RuntimeException: Unable to find proto >>>> buffer class >>>> at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:988) >>>> at >>>> org.apache.spark.rdd.ParallelCollectionPartition.readObject(ParallelCollectionRDD.scala:70) >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>> at >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>>> at >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>> at >>>> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) >>>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893) >>>> at >>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) >>>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) >>>> at >>>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) >>>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) >>>> at >>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) >>>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) >>>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) >>>> at >>>> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62) >>>> at >>>> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87) >>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:178) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>> at java.lang.Thread.run(Thread.java:745) >>>> Caused by: java.lang.RuntimeException: Unable to find proto buffer class >>>> at >>>> com.google.protobuf.GeneratedMessageLite$SerializedForm.readResolve(GeneratedMessageLite.java:775) >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>> at >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>>> at >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>> at >>>> java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1104) >>>> at >>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1807) >>>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) >>>> at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1706) >>>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1344) >>>> at >>>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) >>>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) >>>> at >>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) >>>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) >>>> at >>>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) >>>> at >>>> java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:500) >>>> at >>>> org.apache.spark.rdd.ParallelCollectionPartition$$anonfun$readObject$1.apply$mcV$sp(ParallelCollectionRDD.scala:74) >>>> at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:985) >>>> ... 20 more >>>> Caused by: java.lang.ClassNotFoundException: >>>> com.rick.reports.Reports$SensorReports >>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366) >>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355) >>>> at java.security.AccessController.doPrivileged(Native Method) >>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354) >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425) >>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358) >>>> at java.lang.Class.forName0(Native Method) >>>> at java.lang.Class.forName(Class.java:191) >>>> at >>>> com.google.protobuf.GeneratedMessageLite$SerializedForm.readResolve(GeneratedMessageLite.java:768) >>>> ... 37 more >>>> 15/02/23 17:35:08 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID >>>> 0, localhost): java.io.IOException: java.lang.RuntimeException: Unable to >>>> find proto buffer class >>>> at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:988) >>>> at >>>> org.apache.spark.rdd.ParallelCollectionPartition.readObject(ParallelCollectionRDD.scala:70) >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>> at >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>>> at >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>> at >>>> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) >>>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893) >>>> at >>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) >>>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) >>>> at >>>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) >>>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) >>>> at >>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) >>>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) >>>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) >>>> at >>>> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62) >>>> at >>>> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87) >>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:178) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>> at java.lang.Thread.run(Thread.java:745) >>>> Caused by: java.lang.RuntimeException: Unable to find proto buffer class >>>> at >>>> com.google.protobuf.GeneratedMessageLite$SerializedForm.readResolve(GeneratedMessageLite.java:775) >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>> at >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>>> at >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>> at >>>> java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1104) >>>> at >>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1807) >>>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) >>>> at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1706) >>>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1344) >>>> at >>>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) >>>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) >>>> at >>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) >>>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) >>>> at >>>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) >>>> at >>>> java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:500) >>>> at >>>> org.apache.spark.rdd.ParallelCollectionPartition$$anonfun$readObject$1.apply$mcV$sp(ParallelCollectionRDD.scala:74) >>>> at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:985) >>>> ... 20 more >>>> Caused by: java.lang.ClassNotFoundException: >>>> com.rick.reports.Reports$SensorReports >>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366) >>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355) >>>> at java.security.AccessController.doPrivileged(Native Method) >>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354) >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425) >>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358) >>>> at java.lang.Class.forName0(Native Method) >>>> at java.lang.Class.forName(Class.java:191) >>>> at >>>> com.google.protobuf.GeneratedMessageLite$SerializedForm.readResolve(GeneratedMessageLite.java:768) >>>> ... 37 more >>>> >>>> 15/02/23 17:35:08 ERROR TaskSetManager: Task 0 in stage 0.0 failed 1 >>>> times; aborting job >>>> 15/02/23 17:35:08 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose >>>> tasks have all completed, from pool >>>> 15/02/23 17:35:08 INFO TaskSchedulerImpl: Cancelling stage 0 >>>> 15/02/23 17:35:08 INFO DAGScheduler: Job 0 failed: foreach at >>>> NewReportProcessor.scala:17, took 0.644071 s >>>> Exception in thread "main" org.apache.spark.SparkException: Job aborted >>>> due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent >>>> failure: Lost task 0.0 in stage 0.0 (TID 0, localhost): >>>> java.io.IOException: java.lang.RuntimeException: Unable to find proto >>>> buffer class >>>> at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:988) >>>> at >>>> org.apache.spark.rdd.ParallelCollectionPartition.readObject(ParallelCollectionRDD.scala:70) >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>> at >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>>> at >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>> at >>>> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) >>>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893) >>>> at >>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) >>>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) >>>> at >>>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) >>>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) >>>> at >>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) >>>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) >>>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) >>>> at >>>> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62) >>>> at >>>> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87) >>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:178) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>> at java.lang.Thread.run(Thread.java:745) >>>> Caused by: java.lang.RuntimeException: Unable to find proto buffer class >>>> at >>>> com.google.protobuf.GeneratedMessageLite$SerializedForm.readResolve(GeneratedMessageLite.java:775) >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>> at >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>>> at >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>> at >>>> java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1104) >>>> at >>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1807) >>>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) >>>> at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1706) >>>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1344) >>>> at >>>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) >>>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) >>>> at >>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) >>>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) >>>> at >>>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) >>>> at >>>> java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:500) >>>> at >>>> org.apache.spark.rdd.ParallelCollectionPartition$$anonfun$readObject$1.apply$mcV$sp(ParallelCollectionRDD.scala:74) >>>> at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:985) >>>> ... 20 more >>>> Caused by: java.lang.ClassNotFoundException: >>>> com.rick.reports.Reports$SensorReports >>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366) >>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355) >>>> at java.security.AccessController.doPrivileged(Native Method) >>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354) >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425) >>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358) >>>> at java.lang.Class.forName0(Native Method) >>>> at java.lang.Class.forName(Class.java:191) >>>> at >>>> com.google.protobuf.GeneratedMessageLite$SerializedForm.readResolve(GeneratedMessageLite.java:768) >>>> ... 37 more >>>> >>>> Driver stacktrace: >>>> at org.apache.spark.scheduler.DAGScheduler.org >>>> $apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1214) >>>> at >>>> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1203) >>>> at >>>> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1202) >>>> at >>>> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) >>>> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) >>>> at >>>> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1202) >>>> at >>>> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:696) >>>> at >>>> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:696) >>>> at scala.Option.foreach(Option.scala:236) >>>> at >>>> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:696) >>>> at >>>> org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1420) >>>> at akka.actor.Actor$class.aroundReceive(Actor.scala:465) >>>> at >>>> org.apache.spark.scheduler.DAGSchedulerEventProcessActor.aroundReceive(DAGScheduler.scala:1375) >>>> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) >>>> at akka.actor.ActorCell.invoke(ActorCell.scala:487) >>>> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) >>>> at akka.dispatch.Mailbox.run(Mailbox.scala:220) >>>> at >>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) >>>> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) >>>> at >>>> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) >>>> at >>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) >>>> at >>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) >>>> >>>> The Proper Report Output: >>>> ==================== >>>> reports { >>>> network_report { >>>> begin_time: 1424380054789676056 >>>> end_time: 1424380054789740740 >>>> source: "<some-IP-address>:80" >>>> destination: "<some-IP-address>:46792" >>>> protocol: TCP >>>> stream_id: 3 >>>> stream_status: 8 >>>> } >>>> http_report { >>>> request { >>>> method: "GET" >>>> etc... >>>> >>>> >>> >
