OK it turned out setting -Dspark.serializer=org.apache.spark.serializer.KryoSerializer in SPARK_JAVA_OPTS on the workers/slaves caused all this. not sure why. this used to work fine in previous spark. but when i removed it the errors went away.
On Fri, Oct 18, 2013 at 2:59 PM, Koert Kuipers <[email protected]> wrote: > i installed the plain vanilla spark 0.8 on our cluster, downloaded from > here: > http://spark-project.org/download/spark-0.8.0-incubating-bin-hadoop1.tgz > after a restart of all spark daemons i still see the same issue for every > task: > > java.io.StreamCorruptedException: invalid type code: 00 > > so now i am guessing it must be something in my configuration. i guess > this is progress... > > looking at the logs of a worker, i see the task gets launched like this: > > Spark Executor Command: "java" "-cp" > ":/usr/local/lib/spark/conf:/usr/local/lib/spark/assembly/target/scala-2.9.3/spark-assembly_2.9.3-0.8.0-incubating-hadoop1.0.4.jar" > "-Dspar > k.worker.timeout=30000" "-Dspark.akka.timeout=30000" > "-Dspark.storage.blockManagerHeartBeatMs=120000" > "-Dspark.storage.blockManagerTimeoutIntervalMs=120000" "-Dspark.akka.retry > .wait=30000" "-Dspark.akka.frameSize=10000" > "-Dspark.akka.logLifecycleEvents=true" > "-Dspark.serializer=org.apache.spark.serializer.KryoSerializer" > "-Dspark.worker.timeout=30000 > " "-Dspark.akka.timeout=30000" > "-Dspark.storage.blockManagerHeartBeatMs=120000" > "-Dspark.storage.blockManagerTimeoutIntervalMs=120000" > "-Dspark.akka.retry.wait=30000" "-Dspark. > akka.frameSize=10000" "-Dspark.akka.logLifecycleEvents=true" > "-Dspark.serializer=org.apache.spark.serializer.KryoSerializer" "-Xms512M" > "-Xmx512M" "org.apache.spark.executor.St > andaloneExecutorBackend" "akka:// > [email protected]:38472/user/StandaloneScheduler" "1" "node02" "7" > > > and finally this is my spark-env.sh: > > export SCALA_HOME=/usr/local/lib/scala-2.9.3 > export SPARK_MASTER_IP=node01 > export SPARK_MASTER_PORT=7077 > export SPARK_MASTER_WEBUI_PORT=8080 > export SPARK_WORKER_CORES=7 > export SPARK_WORKER_MEMORY=14G > export SPARK_WORKER_PORT=7078 > export SPARK_WORKER_WEBUI_PORT=8081 > export SPARK_WORKER_DIR=/var/lib/spark > export SPARK_CLASSPATH=$SPARK_USER_CLASSPATH > export SPARK_JAVA_OPTS="-Dspark.worker.timeout=30000 > -Dspark.akka.timeout=30000 -Dspark.storage.blockManagerHeartBeatMs=120000 > -Dspark.storage.blockManagerTimeoutIntervalMs=120 > 000 -Dspark.akka.retry.wait=30000 -Dspark.akka.frameSize=10000 > -Dspark.akka.logLifecycleEvents=true > -Dspark.serializer=org.apache.spark.serializer.KryoSerializer $SPARK_JAVA_OP > TS" > export > SPARK_WORKER_OPTS="-Dspark.local.dir=/data/0/tmp,/data/1/tmp,/data/2/tmp,/data/3/tmp,/data/4/tmp,/data/5/tmp" > > > > > > On Fri, Oct 18, 2013 at 2:02 PM, Koert Kuipers <[email protected]> wrote: > >> i checked out the v0.8.0-incubating tag again, changed the settings to >> build against correct version of hadoop for our cluster, ran sbt-assembly, >> build tarball, installed it on cluster, restarted spark... same errors >> >> >> On Fri, Oct 18, 2013 at 12:49 PM, Koert Kuipers <[email protected]>wrote: >> >>> at this point i feel like it must be some sort of version mismatch? i am >>> gonna check the spark build that i deployed on the cluster >>> >>> >>> On Fri, Oct 18, 2013 at 12:46 PM, Koert Kuipers <[email protected]>wrote: >>> >>>> name := "Simple Project" >>>> >>>> version := "1.0" >>>> >>>> scalaVersion := "2.9.3" >>>> >>>> libraryDependencies += "org.apache.spark" %% "spark-core" % >>>> "0.8.0-incubating" >>>> >>>> resolvers += "Akka Repository" at "http://repo.akka.io/releases/"; >>>> >>>> resolvers += "Cloudera Repository" at " >>>> https://repository.cloudera.com/artifactory/cloudera-repos/"; >>>> >>>> libraryDependencies += "org.apache.hadoop" % "hadoop-client" % >>>> "2.0.0-mr1-cdh4.3.0" >>>> >>>> >>>> >>>> >>>> On Fri, Oct 18, 2013 at 12:34 PM, Matei Zaharia < >>>> [email protected]> wrote: >>>> >>>>> Can you post the build.sbt for your program? It needs to include >>>>> hadoop-client for CDH4.3, and that should *not* be listed as provided. >>>>> >>>>> Matei >>>>> >>>>> On Oct 18, 2013, at 8:23 AM, Koert Kuipers <[email protected]> wrote: >>>>> >>>>> ok this has nothing to do with hadoop access. even a simple program >>>>> that uses sc.parallelize blows up in this way. >>>>> >>>>> so spark-shell works well on the same machine i launch this from. >>>>> >>>>> if i launch a simple program without using kryo for serializer and >>>>> closure serialize i get a different error. see below. >>>>> at this point it seems to me i have some issue with task >>>>> serialization??? >>>>> >>>>> >>>>> >>>>> 13/10/18 11:20:37 INFO StandaloneExecutorBackend: Got assigned task 0 >>>>> 13/10/18 11:20:37 INFO StandaloneExecutorBackend: Got assigned task 1 >>>>> 13/10/18 11:20:37 INFO Executor: Running task ID 1 >>>>> 13/10/18 11:20:37 INFO Executor: Running task ID 0 >>>>> 13/10/18 11:20:37 INFO Executor: Fetching >>>>> http://192.168.3.171:41629/jars/simple-project_2.9.3-1.0.jar with >>>>> timestamp 1382109635095 >>>>> 13/10/18 11:20:37 INFO Utils: Fetching >>>>> http://192.168.3.171:41629/jars/simple-project_2.9.3-1.0.jar to >>>>> /tmp/fetchFileTemp378181753997570700.tmp >>>>> 13/10/18 11:20:37 INFO Executor: Adding >>>>> file:/var/lib/spark/app-20131018112035-0014/1/./simple-project_2.9.3-1.0.jar >>>>> to class loader >>>>> 13/10/18 11:20:37 INFO Executor: caught throwable with stacktrace >>>>> java.io.StreamCorruptedException: invalid type code: 00 >>>>> at >>>>> java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(ObjectInputStream.java:2467) >>>>> at >>>>> java.io.ObjectInputStream$BlockDataInputStream.refill(ObjectInputStream.java:2502) >>>>> at >>>>> java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2661) >>>>> at >>>>> java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2583) >>>>> at java.io.DataInputStream.readFully(DataInputStream.java:178) >>>>> at java.io.DataInputStream.readLong(DataInputStream.java:399) >>>>> at >>>>> java.io.ObjectInputStream$BlockDataInputStream.readLong(ObjectInputStream.java:2803) >>>>> at java.io.ObjectInputStream.readLong(ObjectInputStream.java:958) >>>>> at >>>>> org.apache.spark.rdd.ParallelCollectionPartition.readObject(ParallelCollectionRDD.scala:72) >>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>> at >>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >>>>> at >>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>>> at java.lang.reflect.Method.invoke(Method.java:597) >>>>> at >>>>> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:969) >>>>> at >>>>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1852) >>>>> at >>>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1756) >>>>> at >>>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1326) >>>>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:348) >>>>> at >>>>> org.apache.spark.scheduler.ResultTask.readExternal(ResultTask.scala:135) >>>>> at >>>>> java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1795) >>>>> at >>>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1754) >>>>> at >>>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1326) >>>>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:348) >>>>> at >>>>> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:39) >>>>> at >>>>> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:61) >>>>> at >>>>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:153) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> >>>>> >>>>> On Fri, Oct 18, 2013 at 10:59 AM, Koert Kuipers <[email protected]>wrote: >>>>> >>>>>> i created a tiny sbt project as described here: >>>>>> apache.org/docs/latest/quick-start.html#a-standalone-app-in-scala<http://spark.incubator.apache.org/docs/latest/quick-start.html#a-standalone-app-in-scala> >>>>>> >>>>>> it has the correct dependencies: spark-core and the correct >>>>>> hadoop-client for my platform. i tried both the generic spark-core >>>>>> dependency and spark-core dependency compiled against my platform. it >>>>>> runs >>>>>> fine in local mode, but when i switch to the cluster i still always get >>>>>> the >>>>>> following exceptions on tasks: >>>>>> >>>>>> 13/10/18 10:25:53 ERROR Executor: Uncaught exception in thread >>>>>> Thread[pool-5-thread-1,5,main] >>>>>> >>>>>> java.lang.NullPointerException >>>>>> at >>>>>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187) >>>>>> at >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) >>>>>> at >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) >>>>>> at java.lang.Thread.run(Thread.java:662) >>>>>> >>>>>> after adding some additional debugging to Executor i see the cause is >>>>>> this: >>>>>> 13/10/18 10:54:47 INFO Executor: caught throwable with stacktrace >>>>>> java.lang.NullPointerException >>>>>> at >>>>>> org.apache.spark.executor.Executor$TaskRunner$$anonfun$run$2.apply(Executor.scala:155) >>>>>> at >>>>>> org.apache.spark.executor.Executor$TaskRunner$$anonfun$run$2.apply(Executor.scala:155) >>>>>> at org.apache.spark.Logging$class.logInfo(Logging.scala:48) >>>>>> at org.apache.spark.executor.Executor.logInfo(Executor.scala:36) >>>>>> at >>>>>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:155) >>>>>> >>>>>> at >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) >>>>>> at >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) >>>>>> at java.lang.Thread.run(Thread.java:662) >>>>>> >>>>>> so it seems the offending line is: >>>>>> logInfo("Its epoch is " + task.epoch) >>>>>> >>>>>> i am guessing accessing epoch on the task is throwing the NPE. any >>>>>> ideas? >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Oct 17, 2013 at 8:12 PM, Koert Kuipers <[email protected]>wrote: >>>>>> >>>>>>> sorry one more related question: >>>>>>> i compile against a spark build for hadoop 1.0.4, but the actual >>>>>>> installed version of spark is build against cdh4.3.0-mr1. this also >>>>>>> used to >>>>>>> work, and i prefer to do this so i compile against a generic spark >>>>>>> build. >>>>>>> could this be the issue? >>>>>>> >>>>>>> >>>>>>> On Thu, Oct 17, 2013 at 8:06 PM, Koert Kuipers <[email protected]>wrote: >>>>>>> >>>>>>>> i have my spark and hadoop related dependencies as "provided" for >>>>>>>> my spark job. this used to work with previous versions. are these now >>>>>>>> supposed to be compile/runtime/default dependencies? >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Oct 17, 2013 at 8:04 PM, Koert Kuipers >>>>>>>> <[email protected]>wrote: >>>>>>>> >>>>>>>>> yes i did that and i can see the correct jars sitting in >>>>>>>>> lib_managed >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Oct 17, 2013 at 7:56 PM, Matei Zaharia < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Koert, did you link your Spark job to the right version of HDFS >>>>>>>>>> as well? In Spark 0.8, you have to add a Maven dependency on >>>>>>>>>> "hadoop-client" for your version of Hadoop. See >>>>>>>>>> http://spark.incubator.apache.org/docs/latest/quick-start.html#a-standalone-app-in-scala >>>>>>>>>> for >>>>>>>>>> example. >>>>>>>>>> >>>>>>>>>> Matei >>>>>>>>>> >>>>>>>>>> On Oct 17, 2013, at 4:38 PM, Koert Kuipers <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> i got the job a little further along by also setting this: >>>>>>>>>> System.setProperty("spark.closure.serializer", >>>>>>>>>> "org.apache.spark.serializer.KryoSerializer") >>>>>>>>>> >>>>>>>>>> not sure why i need to... but anyhow, now my workers start and >>>>>>>>>> then they blow up on this: >>>>>>>>>> >>>>>>>>>> 13/10/17 19:22:57 ERROR Executor: Uncaught exception in thread >>>>>>>>>> Thread[pool-5-thread-1,5,main] >>>>>>>>>> java.lang.NullPointerException >>>>>>>>>> at >>>>>>>>>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187) >>>>>>>>>> at >>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) >>>>>>>>>> at >>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) >>>>>>>>>> at java.lang.Thread.run(Thread.java:662) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> which is: >>>>>>>>>> val metrics = attemptedTask.flatMap(t => t.metrics) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, Oct 17, 2013 at 7:30 PM, dachuan <[email protected]>wrote: >>>>>>>>>> >>>>>>>>>>> thanks, Mark. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Thu, Oct 17, 2013 at 6:36 PM, Mark Hamstra < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> SNAPSHOTs are not fixed versions, but are floating names >>>>>>>>>>>> associated with whatever is the most recent code. So, Spark 0.8.0 >>>>>>>>>>>> is the >>>>>>>>>>>> current released version of Spark, which is exactly the same today >>>>>>>>>>>> as it >>>>>>>>>>>> was yesterday, and will be the same thing forever. Spark >>>>>>>>>>>> 0.8.1-SNAPSHOT is >>>>>>>>>>>> whatever is currently in branch-0.8. It changes every time new >>>>>>>>>>>> code is >>>>>>>>>>>> committed to that branch (which should be just bug fixes and the >>>>>>>>>>>> few >>>>>>>>>>>> additional features that we wanted to get into 0.8.0, but that >>>>>>>>>>>> didn't quite >>>>>>>>>>>> make it.) Not too long from now there will be a release of Spark >>>>>>>>>>>> 0.8.1, at >>>>>>>>>>>> which time the SNAPSHOT will got to 0.8.2 and 0.8.1 will be >>>>>>>>>>>> forever frozen. >>>>>>>>>>>> Meanwhile, the wild new development is taking place on the master >>>>>>>>>>>> branch, >>>>>>>>>>>> and whatever is currently in that branch becomes 0.9.0-SNAPSHOT. >>>>>>>>>>>> This >>>>>>>>>>>> could be quite different from day to day, and there are no >>>>>>>>>>>> guarantees that >>>>>>>>>>>> things won't be broken in 0.9.0-SNAPSHOT. Several months from now >>>>>>>>>>>> there >>>>>>>>>>>> will be a release of Spark 0.9.0 (unless the decision is made to >>>>>>>>>>>> bump the >>>>>>>>>>>> version to 1.0.0), at which point the SNAPSHOT goes to 0.9.1 and >>>>>>>>>>>> the whole >>>>>>>>>>>> process advances to the next phase of development. >>>>>>>>>>>> >>>>>>>>>>>> The short answer is that releases are stable, SNAPSHOTs are >>>>>>>>>>>> not, and SNAPSHOTs that aren't on maintenance branches can break >>>>>>>>>>>> things. >>>>>>>>>>>> You make your choice of which to use and pay the consequences. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Oct 17, 2013 at 3:18 PM, dachuan <[email protected]>wrote: >>>>>>>>>>>> >>>>>>>>>>>>> yeah, I mean 0.9.0-SNAPSHOT. I use git clone and that's what I >>>>>>>>>>>>> got.. what's the difference? I mean SNAPSHOT and non-SNAPSHOT. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, Oct 17, 2013 at 6:15 PM, Mark Hamstra < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Of course, you mean 0.9.0-SNAPSHOT. There is no Spark 0.9.0, >>>>>>>>>>>>>> and won't be for several months. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Thu, Oct 17, 2013 at 3:11 PM, dachuan >>>>>>>>>>>>>> <[email protected]>wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> I'm sorry if this doesn't answer your question directly, but >>>>>>>>>>>>>>> I have tried spark 0.9.0 and hdfs 1.0.4 just now, it works.. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Thu, Oct 17, 2013 at 6:05 PM, Koert Kuipers < >>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> after upgrading from spark 0.7 to spark 0.8 i can no longer >>>>>>>>>>>>>>>> access any files on HDFS. >>>>>>>>>>>>>>>> i see the error below. any ideas? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> i am running spark standalone on a cluster that also has >>>>>>>>>>>>>>>> CDH4.3.0 and rebuild spark accordingly. the jars in >>>>>>>>>>>>>>>> lib_managed look good >>>>>>>>>>>>>>>> to me. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> i noticed similar errors in the mailing list but found no >>>>>>>>>>>>>>>> suggested solutions. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> thanks! koert >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 13/10/17 17:43:23 ERROR Executor: Exception in task ID 0 >>>>>>>>>>>>>>>> java.io.EOFException >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> java.io.ObjectInputStream$BlockDataInputStream.readFully(ObjectInputStream.java:2703) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> java.io.ObjectInputStream.readFully(ObjectInputStream.java:1008) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:68) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:106) >>>>>>>>>>>>>>>> at org.apache.hadoop.io.UTF8.readChars(UTF8.java:258) >>>>>>>>>>>>>>>> at org.apache.hadoop.io.UTF8.readString(UTF8.java:250) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> org.apache.hadoop.mapred.FileSplit.readFields(FileSplit.java:87) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:280) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:75) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> org.apache.spark.SerializableWritable.readObject(SerializableWritable.scala:39) >>>>>>>>>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >>>>>>>>>>>>>>>> Method) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>>>>>>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:969) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1852) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1756) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1326) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1950) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1874) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1756) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1326) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> java.io.ObjectInputStream.readObject(ObjectInputStream.java:348) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> org.apache.spark.scheduler.ResultTask.readExternal(ResultTask.scala:135) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1795) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1754) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1326) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> java.io.ObjectInputStream.readObject(ObjectInputStream.java:348) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:39) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:61) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:153) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) >>>>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:662) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Dachuan Huang >>>>>>>>>>>>>>> Cellphone: 614-390-7234 >>>>>>>>>>>>>>> 2015 Neil Avenue >>>>>>>>>>>>>>> Ohio State University >>>>>>>>>>>>>>> Columbus, Ohio >>>>>>>>>>>>>>> U.S.A. >>>>>>>>>>>>>>> 43210 >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Dachuan Huang >>>>>>>>>>>>> Cellphone: 614-390-7234 >>>>>>>>>>>>> 2015 Neil Avenue >>>>>>>>>>>>> Ohio State University >>>>>>>>>>>>> Columbus, Ohio >>>>>>>>>>>>> U.S.A. >>>>>>>>>>>>> 43210 >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Dachuan Huang >>>>>>>>>>> Cellphone: 614-390-7234 >>>>>>>>>>> 2015 Neil Avenue >>>>>>>>>>> Ohio State University >>>>>>>>>>> Columbus, Ohio >>>>>>>>>>> U.S.A. >>>>>>>>>>> 43210 >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> >>>> >>> >> >
