UnresolvedAddressException in Kubernetes Cluster

[Suman Somasundar]

Hi,

I am trying to deploy a Spark app in a Kubernetes Cluster. The cluster consists 
of 2 machines - 1 master and 1 slave, each of them with the following config:
RHEL 7.2
Docker 17.03.1
K8S 1.7.

I am following the steps provided in 
https://apache-spark-on-k8s.github.io/userdocs/running-on-kubernetes.html 


When I submit an application (SparkPi), a driver pod is created on the slave 
machine of the cluster. But it exits with an exception:

2017-10-09 22:13:24 INFO  SecurityManager:54 - SecurityManager: authentication 
disabled; ui acls disabled; users  with view permissions: Set(root); groups 
with view permissions: Set(); users  with modify permissions: Set(root); groups 
with modify permissions: Set()
2017-10-09 22:13:30 ERROR SparkContext:91 - Error initializing SparkContext.
java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:101)
at 
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:218)
at 
io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:127)
at 
io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:501)
at 
io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1218)
at 
io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:496)
at 
io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:481)
at 
io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:965)
at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:210)
at 
io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:353)
at 
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:446)
at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
at 
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
at java.lang.Thread.run(Thread.java:748)
2017-10-09 22:13:30 INFO  SparkContext:54 - Successfully stopped SparkContext

Has anyone come across this problem or know why this might be happening?

Thanks,
Suman.

Log messages for shuffle phase

[Suman Somasundar]

Hi,

 

While going through the logs of an application, I noticed that I could not find 
any logs to dig deeper into any of the shuffle phases.

 

I am interested in finding out time taken by each shuffle phase, the size of 
data spilled to disk if any, among other things.

 

Does anyone know how I can do this?

 

Thanks,

Suman.

Running 2 spark application in parallel

[Suman Somasundar]

Hi all,

 

Is there a way to run 2 spark applications in parallel under Yarn in the same 
cluster?

 

Currently, if I submit 2 applications, one of them waits till the other one is 
completed.

 

I want both of them to start and run at the same time.

 

Thanks,
Suman.

Connection closed error while running Terasort

[Suman Somasundar]

Hi,

 

I am getting the following error while trying to run a 10GB terasort under Yarn 
with 8 nodes.

The command is:  

spark-submit --class com.github.ehiggs.spark.terasort.TeraSort --master 
yarn-cluster --num-executors 10 --executor-memory 32g 
spark-terasort-master/target/spark-terasort-1.0-SNAPSHOT-jar-with-dependencies.jar
 hdfs://hadoop-solaris-a:8020/user/hadoop/terasort/input-10 
hdfs://hadoop-solaris-a:8020/user/hadoop/terasort/output-10

 

What might be causing this error?

 

15/08/31 17:09:48 ERROR server.TransportRequestHandler: Error sending result 
ChunkFetchSuccess{streamChunkId=StreamChunkId{streamId=1867783019052, 
chunkIndex=0}, 
buffer=FileSegmentManagedBuffer{file=/tmp/hadoop/nm-local-dir/usercache/hadoop/appcache/application_1441064487503_0001/blockmgr-c3c8dbb3-9ae2-4e45-b537-fd0beeff98b5/3e/shuffle_1_9_0.data,
 offset=0, length=1059423784}} to /199.199.35.5:52486; closing connection

java.io.IOException: Broken pipe

at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)

at 
sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:443)

at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:575)

at 
org.apache.spark.network.buffer.LazyFileRegion.transferTo(LazyFileRegion.java:96)

at 
org.apache.spark.network.protocol.MessageWithHeader.transferTo(MessageWithHeader.java:89)

at 
io.netty.channel.socket.nio.NioSocketChannel.doWriteFileRegion(NioSocketChannel.java:237)

at 
io.netty.channel.nio.AbstractNioByteChannel.doWrite(AbstractNioByteChannel.java:233)

at 
io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:264)

at 
io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:707)

at 
io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.forceFlush(AbstractNioChannel.java:321)

at 
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:519)

at 
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)

at 
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)

at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)

at java.lang.Thread.run(Thread.java:745)

15/08/31 17:10:48 ERROR server.TransportChannelHandler: Connection to 
hadoop-solaris-c/199.199.35.4:48540 has been quiet for 12 ms while there 
are outstanding requests. Assuming connection is dead; please adjust 
spark.network.timeout if this is wrong.

15/08/31 17:10:48 ERROR client.TransportResponseHandler: Still have 1 requests 
outstanding when connection from hadoop-solaris-c/199.199.35.4:48540 is closed

15/08/31 17:10:48 INFO shuffle.RetryingBlockFetcher: Retrying fetch (3/3) for 1 
outstanding blocks after 5000 ms

15/08/31 17:10:49 ERROR server.TransportRequestHandler: Error sending result 
ChunkFetchSuccess{streamChunkId=StreamChunkId{streamId=1867783019053, 
chunkIndex=0}, 
buffer=FileSegmentManagedBuffer{file=/tmp/hadoop/nm-local-dir/usercache/hadoop/appcache/application_1441064487503_0001/blockmgr-c3c8dbb3-9ae2-4e45-b537-fd0beeff98b5/1b/shuffle_1_6_0.data,
 offset=0, length=1052128440}} to /199.199.35.6:45201; closing connection

java.nio.channels.ClosedChannelException

15/08/31 17:10:53 INFO client.TransportClientFactory: Found inactive connection 
to hadoop-solaris-c/199.199.35.4:48540, creating a new one.

15/08/31 17:11:31 ERROR server.TransportRequestHandler: Error sending result 
ChunkFetchSuccess{streamChunkId=StreamChunkId{streamId=1867783019054, 
chunkIndex=0}, 
buffer=FileSegmentManagedBuffer{file=/tmp/hadoop/nm-local-dir/usercache/hadoop/appcache/application_1441064487503_0001/blockmgr-c3c8dbb3-9ae2-4e45-b537-fd0beeff98b5/1b/shuffle_1_6_0.data,
 offset=0, length=1052128440}} to /199.199.35.10:55082; closing connection

java.nio.channels.ClosedChannelException

15/08/31 17:11:31 ERROR server.TransportRequestHandler: Error sending result 
ChunkFetchSuccess{streamChunkId=StreamChunkId{streamId=1867783019055, 
chunkIndex=0}, 
buffer=FileSegmentManagedBuffer{file=/tmp/hadoop/nm-local-dir/usercache/hadoop/appcache/application_1441064487503_0001/blockmgr-c3c8dbb3-9ae2-4e45-b537-fd0beeff98b5/3e/shuffle_1_9_0.data,
 offset=0, length=1059423784}} to /199.199.35.7:54328; closing connection

java.nio.channels.ClosedChannelException

15/08/31 17:11:53 ERROR server.TransportRequestHandler: Error sending result 
ChunkFetchSuccess{streamChunkId=StreamChunkId{streamId=1867783019056, 
chunkIndex=0}, 
buffer=FileSegmentManagedBuffer{file=/tmp/hadoop/nm-local-dir/usercache/hadoop/appcache/application_1441064487503_0001/blockmgr-c3c8dbb3-9ae2-4e45-b537-fd0beeff98b5/3e/shuffle_1_9_0.data,
 offset=0, length=1059423784}} to /199.199.35.5:50573; closing connection

java.nio.channels.ClosedChannelException

15/08/31 17:12:54 ERROR 

Restricting the number of iterations in Mllib Kmeans

[Suman Somasundar]

Hi,,

I want to run a definite number of iterations in Kmeans.  There is a command 
line argument to set maxIterations, but even if I set it to a number, Kmeans 
runs until the centroids converge. 

Is there a specific way to specify it in command line?


Also, I wanted to know if we can supply the initial set of centroids to the 
program instead of it choosing the centroids in random?

 

Thanks,
Suman.

How to increase parallelism in Yarn

[Suman Somasundar]

Hi,

 

I am using Spark 1.1.1 on Yarn. When I try to run K-Means, I see from the Yarn 
dashboard that only 3 containers are being used. How do I increase the number 
of containers used?

 

P.S: When I run K-Means on Mahout with the same settings, I see that there are 
25-30 containers being used.

 

Thanks,
Suman.

Re: Invalid Class Exception

[Suman Somasundar]

I am building Spark by myself and I am using Java 7 to both build and run.

I will try with Java 6.

Thanks,
Suman.

On 6/3/2014 7:18 PM, Matei Zaharia wrote:

What Java version do you have, and how did you get Spark (did you build it 
yourself by any chance or download a pre-built one)? If you build Spark 
yourself you need to do it with Java 6 — it’s a known issue because of the way 
Java 6 and 7 package JAR files. But I haven’t seen it result in this particular 
error.

Matei

On Jun 3, 2014, at 5:18 PM, Suman Somasundar suman.somasun...@oracle.com 
wrote:


Hi all,

I get the following exception when using Spark to run example k-means program.  
I am using Spark 1.0.0 and running the program locally.

java.io.InvalidClassException: scala.Tuple2; invalid descriptor for field _1
at java.io.ObjectStreamClass.readNonProxy(ObjectStreamClass.java:697)
at 
java.io.ObjectInputStream.readClassDescriptor(ObjectInputStream.java:827)
at 
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1583)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1514)
at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1750)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
at 
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:63)
at 
org.apache.spark.serializer.DeserializationStream$$anon$1.getNext(Serializer.scala:125)
at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:71)
at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
at 
org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:30)
at 
org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
at 
org.apache.spark.Aggregator.combineCombinersByKey(Aggregator.scala:87)
at 
org.apache.spark.rdd.PairRDDFunctions$$anonfun$combineByKey$3.apply(PairRDDFunctions.scala:101)
at 
org.apache.spark.rdd.PairRDDFunctions$$anonfun$combineByKey$3.apply(PairRDDFunctions.scala:100)
at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:582)
at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:582)
at 
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111)
at org.apache.spark.scheduler.Task.run(Task.scala:51)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.IllegalArgumentException: illegal signature
at java.io.ObjectStreamField.init(ObjectStreamField.java:119)
at java.io.ObjectStreamClass.readNonProxy(ObjectStreamClass.java:695)
... 26 more

Anyone know why this is happening?

Thanks,
Suman.

Re: Invalid Class Exception

[Suman Somasundar]

I tried building with Java 6 and also tried the pre-built packages. I am 
still getting the same error.


It works fine when I run it on a machine with Solaris OS and X-86 
architecture.


But, it does not work with Solaris OS and Sparc architecture.

Any ideas, why this would happen?

Thanks,
Suman.

On 6/4/2014 10:48 AM, Suman Somasundar wrote:
I am building Spark by myself and I am using Java 7 to both build and 
run.


I will try with Java 6.

Thanks,
Suman.

On 6/3/2014 7:18 PM, Matei Zaharia wrote:
What Java version do you have, and how did you get Spark (did you 
build it yourself by any chance or download a pre-built one)? If you 
build Spark yourself you need to do it with Java 6 — it’s a known 
issue because of the way Java 6 and 7 package JAR files. But I 
haven’t seen it result in this particular error.


Matei

On Jun 3, 2014, at 5:18 PM, Suman Somasundar 
suman.somasun...@oracle.com wrote:



Hi all,

I get the following exception when using Spark to run example 
k-means program.  I am using Spark 1.0.0 and running the program 
locally.


java.io.InvalidClassException: scala.Tuple2; invalid descriptor for 
field _1
at 
java.io.ObjectStreamClass.readNonProxy(ObjectStreamClass.java:697)
at 
java.io.ObjectInputStream.readClassDescriptor(ObjectInputStream.java:827)
at 
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1583)
at 
java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1514)
at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1750)
at 
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
at 
java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
at 
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:63)
at 
org.apache.spark.serializer.DeserializationStream$$anon$1.getNext(Serializer.scala:125)
at 
org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:71)
at 
scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
at 
org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:30)
at 
org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
at 
org.apache.spark.Aggregator.combineCombinersByKey(Aggregator.scala:87)
at 
org.apache.spark.rdd.PairRDDFunctions$$anonfun$combineByKey$3.apply(PairRDDFunctions.scala:101)
at 
org.apache.spark.rdd.PairRDDFunctions$$anonfun$combineByKey$3.apply(PairRDDFunctions.scala:100)

at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:582)
at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:582)
at 
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) 

at 
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)

at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
at 
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111)

at org.apache.spark.scheduler.Task.run(Task.scala:51)
at 
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.IllegalArgumentException: illegal signature
at java.io.ObjectStreamField.init(ObjectStreamField.java:119)
at 
java.io.ObjectStreamClass.readNonProxy(ObjectStreamClass.java:695)

... 26 more

Anyone know why this is happening?

Thanks,
Suman.

Invalid Class Exception

[Suman Somasundar]

Hi all,

I get the following exception when using Spark to run example k-means 
program.  I am using Spark 1.0.0 and running the program locally.


java.io.InvalidClassException: scala.Tuple2; invalid descriptor for field _1
at 
java.io.ObjectStreamClass.readNonProxy(ObjectStreamClass.java:697)
at 
java.io.ObjectInputStream.readClassDescriptor(ObjectInputStream.java:827)
at 
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1583)
at 
java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1514)
at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1750)
at 
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)

at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
at 
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:63)
at 
org.apache.spark.serializer.DeserializationStream$$anon$1.getNext(Serializer.scala:125)
at 
org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:71)

at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
at 
org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:30)
at 
org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
at 
org.apache.spark.Aggregator.combineCombinersByKey(Aggregator.scala:87)
at 
org.apache.spark.rdd.PairRDDFunctions$$anonfun$combineByKey$3.apply(PairRDDFunctions.scala:101)
at 
org.apache.spark.rdd.PairRDDFunctions$$anonfun$combineByKey$3.apply(PairRDDFunctions.scala:100)

at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:582)
at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:582)
at 
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)

at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
at 
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111)

at org.apache.spark.scheduler.Task.run(Task.scala:51)
at 
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.IllegalArgumentException: illegal signature
at java.io.ObjectStreamField.init(ObjectStreamField.java:119)
at 
java.io.ObjectStreamClass.readNonProxy(ObjectStreamClass.java:695)

... 26 more

Anyone know why this is happening?

Thanks,
Suman.

Re: Invalid Class Exception

[Suman Somasundar]

On 5/27/2014 1:28 PM, Marcelo Vanzin wrote:

On Tue, May 27, 2014 at 1:05 PM, Suman Somasundar
suman.somasun...@oracle.com wrote:

I am running this on a Solaris machine with logical partitions. All the
partitions (workers) access the same Spark folder.

Can you check whether you have multiple versions of the offending
class (org.apache.spark.SerializableWritable) in the classpath of your
apps? Maybe you do and different nodes are loading jars in different


I checked all the org.apache.spark.SerializableWritable classes and all 
of them have the same

serialVersionUID


On 5/23/2014 9:44 PM, Andrew Or wrote:

That means not all of your driver and executors have the same version of
Spark. Are you on a standalone EC2 cluster? If so, one way to fix this is to
run the following on the master node:

/root/spark-ec2/copy-dir --delete /root/spark

This syncs all of Spark across your cluster, configs, jars and everything.


2014-05-23 15:20 GMT-07:00 Suman Somasundar suman.somasun...@oracle.com:

Hi,

I get the following exception when using Spark to run various programs.

java.io.InvalidClassException: org.apache.spark.SerializableWritable;
local class incompatible: stream classdesc serialVersionUID =
6301214776158303468, local class serialVersionUID = -7785455416944904980
 at
java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:604)
 at
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1601)
 at
java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1514)
 at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1750)
 at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
 at
java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
 at
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:40)
 at
org.apache.spark.broadcast.HttpBroadcast$.read(HttpBroadcast.scala:165)
 at
org.apache.spark.broadcast.HttpBroadcast.readObject(HttpBroadcast.scala:56)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:601)
 at
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1004)
 at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1866)
 at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
 at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
 at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1964)
 at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1888)
 at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
 at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
 at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1964)
 at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1888)
 at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
 at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
 at
java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
 at
scala.collection.immutable.$colon$colon.readObject(List.scala:362)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:601)
 at
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1004)
 at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1866)
 at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
 at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
 at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1964)
 at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1888)
 at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
 at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
 at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1964)
 at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1888)
 at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
 at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
 at
java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
 at
scala.collection.immutable.$colon$colon.readObject(List.scala:362

Re: Invalid Class Exception

[Suman Somasundar]

I am running this on a Solaris machine with logical partitions. All the 
partitions (workers) access the same Spark folder.


Thanks,
Suman.

On 5/23/2014 9:44 PM, Andrew Or wrote:
That means not all of your driver and executors have the same version 
of Spark. Are you on a standalone EC2 cluster? If so, one way to fix 
this is to run the following on the master node:


/root/spark-ec2/copy-dir --delete /root/spark

This syncs all of Spark across your cluster, configs, jars and everything.


2014-05-23 15:20 GMT-07:00 Suman Somasundar 
suman.somasun...@oracle.com mailto:suman.somasun...@oracle.com:


Hi,

I get the following exception when using Spark to run various
programs.

java.io.InvalidClassException:
org.apache.spark.SerializableWritable; local class incompatible:
stream classdesc serialVersionUID = 6301214776158303468, local
class serialVersionUID = -7785455416944904980
at
java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:604)
at
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1601)
at
java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1514)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1750)
at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
at
java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
at

org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:40)
at
org.apache.spark.broadcast.HttpBroadcast$.read(HttpBroadcast.scala:165)
at
org.apache.spark.broadcast.HttpBroadcast.readObject(HttpBroadcast.scala:56)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at

sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1004)
at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1866)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1964)
at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1888)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1964)
at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1888)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
at
java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
at
scala.collection.immutable.$colon$colon.readObject(List.scala:362)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at

sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1004)
at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1866)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1964)
at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1888)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1964)
at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1888)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
at
java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
at
scala.collection.immutable.$colon$colon.readObject(List.scala:362)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native

Invalid Class Exception

[Suman Somasundar]

Hi,

I get the following exception when using Spark to run various programs.

java.io.InvalidClassException: org.apache.spark.SerializableWritable; 
local class incompatible: stream classdesc serialVersionUID = 
6301214776158303468, local class serialVersionUID = -7785455416944904980
at 
java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:604)
at 
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1601)
at 
java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1514)
at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1750)
at 
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)

at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
at 
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:40)
at 
org.apache.spark.broadcast.HttpBroadcast$.read(HttpBroadcast.scala:165)
at 
org.apache.spark.broadcast.HttpBroadcast.readObject(HttpBroadcast.scala:56)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:601)
at 
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1004)
at 
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1866)
at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
at 
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
at 
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1964)
at 
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1888)
at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
at 
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
at 
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1964)
at 
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1888)
at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
at 
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)

at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
at 
scala.collection.immutable.$colon$colon.readObject(List.scala:362)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:601)
at 
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1004)
at 
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1866)
at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
at 
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
at 
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1964)
at 
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1888)
at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
at 
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
at 
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1964)
at 
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1888)
at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
at 
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)

at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
at 
scala.collection.immutable.$colon$colon.readObject(List.scala:362)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:601)
at 
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1004)
at 
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1866)
at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
at 
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
at 
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1964)
at 
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1888)
at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
at 
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)

at