I'm wondering if it could be caused by the TDigest class from the artifact that I added to the dependency-reduced jar conflicting with the spark TDigest class which, as you pointed out the other day, is on the spark classpath. The exception is coming right when the summarizer is being used by the confusion matrix.
On 04/03/2015 06:22 PM, Dmitriy Lyubimov wrote:
saw a lot of these, some still bewildering, but they all related to non-local mode (different classpaths on backed and front end). On Fri, Apr 3, 2015 at 1:39 PM, Andrew Palumbo <[email protected]> wrote:Has anybody seen an exception like this when running a spark job? the job completes but this exception is reported in the middle. 15/04/02 12:43:54 ERROR Remoting: org.apache.spark.storage.BlockManagerId; local class incompatible: stream classdesc serialVersionUID = 2439208141545036836, local class serialVersionUID = -7366074099953117729 java.io.InvalidClassException: org.apache.spark.storage.BlockManagerId; local class incompatible: stream classdesc serialVersionUID = 2439208141545036836, local class serialVersionUID = -7366074099953117729 at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:617) at java.io.ObjectInputStream.readNonProxyDesc( ObjectInputStream.java:1622) at java.io.ObjectInputStream.readClassDesc( ObjectInputStream.java:1517) at java.io.ObjectInputStream.readOrdinaryObject( ObjectInputStream.java:1771) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) at java.io.ObjectInputStream.defaultReadFields( ObjectInputStream.java:1990) at java.io.ObjectInputStream.readSerialData( ObjectInputStream.java:1915) at java.io.ObjectInputStream.readOrdinaryObject( ObjectInputStream.java:1798) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) at akka.serialization.JavaSerializer$$anonfun$1. apply(Serializer.scala:136) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) at akka.serialization.JavaSerializer.fromBinary(Serializer.scala:136) at akka.serialization.Serialization$$anonfun$deserialize$1.apply( Serialization.scala:104) at scala.util.Try$.apply(Try.scala:161) at akka.serialization.Serialization.deserialize( Serialization.scala:98) at akka.remote.MessageSerializer$.deserialize( MessageSerializer.scala:23) at akka.remote.DefaultMessageDispatcher.payload$lzycompute$1(Endpoint. scala:55) at akka.remote.DefaultMessageDispatcher.payload$1(Endpoint.scala:55) at akka.remote.DefaultMessageDispatcher.dispatch(Endpoint.scala:73) at akka.remote.EndpointReader$$anonfun$receive$2.applyOrElse( Endpoint.scala:764) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498) at akka.actor.ActorCell.invoke(ActorCell.scala:456) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237) at akka.dispatch.Mailbox.run(Mailbox.scala:219) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec( AbstractDispatcher.scala:386) at scala.concurrent.forkjoin.ForkJoinTask.doExec( ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue. runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker( ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run( ForkJoinWorkerThread.java:107)
