Hi, some time ago we’ve found that it’s better use Kryo serializer instead of
Java one.
So, we turned it on and use it everywhere.
I have pretty complex objects, which I can’t change. Previously my algo was
building such an objects and then storing them into external storage. It was
not required to reshuffle partitions. Now, it seems I have to reshuffle them,
but I’m stuck with ClassCastException. I investigated it a little and it seems
to me that KryoSerializer does not clear it’s state at some point, so it tries
to use StringSerializer for my non String object. My objects are pretty
complex, it’d be pretty hard to make them serializable.
Caused by: java.lang.ClassCastException:
com.company.metadata.model.cleanse.CleanseInfoSequence cannot be cast to
java.lang.String
at
com.esotericsoftware.kryo.serializers.DefaultSerializers$StringSerializer.write(DefaultSerializers.java:146)
at com.esotericsoftware.kryo.Kryo.writeObjectOrNull(Kryo.java:549)
at
com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:68)
at
com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:18)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
at
com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
... 71 more
I’ve found this state issue in Kryo jira and that it’s been fixed after 2.21
(current kryo version in spark). But spark cannot update, because of chill and
chill cannot be updated because of some dependencies on their side. So, spark
sort of stuck with kryo version 2.21.
My own thoughts how I could workaround this
1. Rewrite algo, so that my objects shouldn’t be reshuffled. But at some point
it’be required.
2. Make my objects implement Serializable and be stuck with java serialization
forever.
3. My object inside of kryo looks like ArrayList with my object, so I’m not
sure it’s possible to register my class with custom serializer in kryo.
Any advice would be highly appreciated.
Thanks.
--
Eugene Morozov
[email protected]
