We've run into other problems caused by our old Kryo versions. I agree that the Chill dependency is one of the main blockers to upgrading Kryo, but I don't think that it's insurmountable: if necessary, we could just publish our own forked version of Chill under our own namespace, similar to what we used to do with Pyrolite.
A bigger concern, perhaps, is dependency conflicts with user-specified Kryo versions. See https://github.com/apache/spark/pull/6361 and https://issues.apache.org/jira/browse/SPARK-7708 for some more previous discussions RE: Kryo upgrade. Anyhow, I'm not sure what the right solution is yet, but just wanted to link to some previous context / discussions. - Josh On Thu, Jul 16, 2015 at 7:57 AM, Eugene Morozov <fathers...@list.ru> wrote: > Hi, some time ago we’ve found that it’s better use Kryo serializer instead > of Java one. > So, we turned it on and use it everywhere. > > I have pretty complex objects, which I can’t change. Previously my algo > was building such an objects and then storing them into external storage. > It was not required to reshuffle partitions. Now, it seems I have to > reshuffle them, but I’m stuck with ClassCastException. I investigated it a > little and it seems to me that KryoSerializer does not clear it’s state at > some point, so it tries to use StringSerializer for my non String object. > My objects are pretty complex, it’d be pretty hard to make them > serializable. > > Caused by: java.lang.ClassCastException: > com.company.metadata.model.cleanse.CleanseInfoSequence cannot be cast to > java.lang.String > at > com.esotericsoftware.kryo.serializers.DefaultSerializers$StringSerializer.write(DefaultSerializers.java:146) > at com.esotericsoftware.kryo.Kryo.writeObjectOrNull(Kryo.java:549) > at > com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:68) > at > com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:18) > at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501) > at > com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564) > ... 71 more > > I’ve found this state issue in Kryo jira and that it’s been fixed after > 2.21 (current kryo version in spark). But spark cannot update, because of > chill and chill cannot be updated because of some dependencies on their > side. So, spark sort of stuck with kryo version 2.21. > > My own thoughts how I could workaround this > 1. Rewrite algo, so that my objects shouldn’t be reshuffled. But at some > point it’be required. > 2. Make my objects implement Serializable and be stuck with java > serialization forever. > 3. My object inside of kryo looks like ArrayList with my object, so I’m > not sure it’s possible to register my class with custom serializer in kryo. > > Any advice would be highly appreciated. > Thanks. > -- > Eugene Morozov > fathers...@list.ru > > > > >
