JingsongLi commented on issue #1438: URL: https://github.com/apache/iceberg/issues/1438#issuecomment-694070682
> @stevenzwu ， The issue didn't happen when doing the rolling upgrade. As I mentioned at the beginning, the issue happens when I submit a new application with new Iceberg version to a Flink cluster which deployed an old version iceberg. After digging deeper, we found it is because the job manager and task manager still deserialize the old version iceberg classes from Flink dependency but Flink client uses the new iceberg class from the user application. This could be resolved by setting parameter `yarn.per-job-cluster.include-user-jar=FIRST` so that the job manager and task manager could load new version iceberg classes at first. > > For upgraded Flink cluster with existing application, @JingsongLi, Do we expect user upload application along with the iceberg-flink-runtime jar? Or the Flink cluster provides the iceberg-flink-runtime jar? In our case, the Flink cluster has the iceberg-flink-runtime jar. When the Flink cluster upgraded, the user job possibly needs to restart so it would not subject the compatibility issue. When it is a rolling upgrade, it probably still doesn't needs to transfer iceberg class between job manager and task managers during job running. Hi @chenjunjiedada , got it, I think this should be a problem of cluster deployment. This is because there are two versions of iceberg dependency in your cluster (`flink/lib/*` and user jar), and the class resolution order mechanism of Flink is inconsistent between client and runtime (of course, this can be solved by `yarn.per-job-cluster.include-user-jar`). - If your cluster is for DataStream users, I think Iceberg dependencies can be include in user jar. Because user program is strongly related to the Iceberg API. - If your cluster is for SQL users, I think Iceberg dependencies can be include in `flink/lib/*`. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
