Hey Soren, are you sure that the JAR you used on the executors is for the right version of Spark? Maybe they’re running an older version. The Kryo serializer should be initialized the same way on both.
Matei On May 12, 2014, at 10:39 AM, Soren Macbeth <so...@yieldbot.com> wrote: > I finally managed to track down the source of the kryo issues that I was > having under mesos. > > What happens is the for a reason that I haven't tracked down yet, a handful > of the scala collection classes from chill-scala down get registered by the > mesos executors, but they do all get registered in the driver process. > > This led to scala.Some classes which were serialized by the executors being > incorrectly deserialized as scala.collections.Wrappers$SeqWrapper in driver > during task deserialization, causing a KryoException. > > I resolved this issue in my spark job by explicitly registering the classes > in my Registrator like so: > > > kryo.register(scala.collection.convert.Wrappers.IteratorWrapper.class); > kryo.register(scala.collection.convert.Wrappers.SeqWrapper.class); > kryo.register(scala.collection.convert.Wrappers.MapWrapper.class); > kryo.register(scala.collection.convert.Wrappers.JListWrapper.class); > kryo.register(scala.collection.convert.Wrappers.JMapWrapper.class); > > Again, I'm not sure why they don't get registered in the mesos executors, > but I wanted to report wht I found as well as a workaround in case anyone > else hit this (extraordinarily frustrating) issue again. > > Some interactive debugging note are available in this gist: > > https://gist.github.com/sorenmacbeth/28707a7a973f7a1982dc > > Cheers, > Soren
