stevenzwu commented on issue #1438: URL: https://github.com/apache/iceberg/issues/1438#issuecomment-693008830
> Are the tasks themselves serialized and not just the checkpoint state in this case? @rdblue Flink jobmanager (driver) serializes the tasks and ships them to taskmanagers (workers) to execute the tasks. Any final member of operator class needs to be serializable. It is actually unclear to me what kind of serializable guarantee that Flink provides. Does it guarantee compatibility with different binary versions in the same Flink cluster (like in rolling upgrade case)? @JingsongLi any comment? @chenjunjiedada were you doing a rolling upgrade of the flink/iceberg version when running into the serializable problem? We probably dodged the problem because we always do red/black deployment with standalone cluster mode. In another world, we never have different versions of binaries for a Flink cluster. I took a look at `RowDataTaskWriterFactory` class. - It contains a final member of `FileAppenderFactory` interface, which doesn't extend from `Serializable`. It seems that `GenericAppenderFactory` impl class is actually serializable. I am not sure about `SparkAppenderFactory`. @rdblue should `FileAppenderFactory` interface be serializable? - It also contains final members for many other serializable Iceberg classes, like PartitionSpec, Schema, FileIO, EncryptionManager. I thought the error msg was complaining about the serialVersionUID change for `org.apache.iceberg.PartitionSpec` class. @JingsongLi By custom serializer for `RowDataTaskWriterFactory`, you also mean custom serializer for all the Iceberg classes as the final members for `RowDataTaskWriterFactory`, right? @openinx @JingsongLi This does remind me the potential problem of using Iceberg DataFile interface for transmission between writer and committer. With unaligned checkpoints, those Kryo serialized bytes in network buffer can be included in the checkpoints. I think in this case it will fall back to Java serialization since we don't register any custom serializer to Kryo for `DataFile`. We may need a `FlinkDataFile` class that guarantees/controls the compatability. I believe this is the concern Ryan has. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
