rdblue commented on issue #1438: URL: https://github.com/apache/iceberg/issues/1438#issuecomment-692879407
My understanding is that Kryo has the same general compatibility problems as Java serialization, it is just a bit faster. According to the Kryo docs, [compatibility is not guaranteed and varies across serializers](https://github.com/EsotericSoftware/kryo#compatibility). And the serializer that this would use, `FieldSerializer`, has guarantees similar to Java serialization: > [FieldSerializer] does not support adding, removing, or changing the type of fields without invalidating previously serialized bytes. That's why I think that anything that is serialized and later read by different versions of Iceberg should not use default Java or Kryo serialization. I think that means we need to have a format for Flink checkpoint and savepoint state. We should also have serialization tests. For the second case, I'm not sure I understand what the concern is. Are the tasks themselves serialized and not just the checkpoint state in this case? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
