Hi all! After updating the operator to version 1.6.0, suspended and resuming flink jobs stopped working. When job resumes, the high availability metadata is removed.
Suspend job: 2023-09-11 06:01:41,548 o.a.f.k.o.l.AuditUtils [INFO ][rec-job/rec-job] >>> Event | Info | SPECCHANGED | UPGRADE change(s) detected (Diff: FlinkDeploymentSpec[job.state : running -> suspended]), starting reconciliation. 2023-09-11 06:01:41,548 o.a.f.k.o.r.d.AbstractJobReconciler [INFO ][rec-job/rec-job] Job is in running state, ready for upgrade with LAST_STATE 2023-09-11 06:01:41,558 o.a.f.k.o.l.AuditUtils [INFO ][rec-job/rec-job] >>> Event | Info | SUSPENDED | Suspending existing deployment. 2023-09-11 06:01:41,558 o.a.f.k.o.s.AbstractFlinkService [INFO ][rec-job/rec-job] Deleting cluster with Foreground propagation 2023-09-11 06:01:41,558 o.a.f.k.o.s.NativeFlinkService [INFO ][rec-job/rec-job] Deleting JobManager deployment while preserving HA metadata. 2023-09-11 06:01:41,598 o.a.f.k.o.s.AbstractFlinkService [INFO ][rec-job/rec-job] Waiting for cluster shutdown... 2023-09-11 06:01:45,667 o.a.f.k.o.s.AbstractFlinkService [INFO ][rec-job/rec-job] Waiting for cluster shutdown... (5s) 2023-09-11 06:01:50,730 o.a.f.k.o.s.AbstractFlinkService [INFO ][rec-job/rec-job] Waiting for cluster shutdown... (10s) 2023-09-11 06:01:55,837 o.a.f.k.o.s.AbstractFlinkService [INFO ][rec-job/rec-job] Waiting for cluster shutdown... (15s) 2023-09-11 06:02:00,885 o.a.f.k.o.s.AbstractFlinkService [INFO ][rec-job/rec-job] Waiting for cluster shutdown... (20s) 2023-09-11 06:02:01,895 o.a.f.k.o.s.AbstractFlinkService [INFO ][rec-job/rec-job] Cluster shutdown completed. 2023-09-11 06:02:01,973 o.a.f.k.o.l.AuditUtils [INFO ][rec-job/rec-job] >>> Status | Info | SUSPENDED | The resource (job) has been suspended 2023-09-11 06:02:01,981 o.a.f.k.o.r.d.AbstractFlinkResourceReconciler [INFO ][rec-job/rec-job] Resource fully reconciled, nothing to do... Resume: 2023-09-11 06:02:07,481 o.a.f.k.o.o.d.ApplicationObserver [INFO ][rec-job/rec-job] Observing JobManager deployment. Previous status: MISSING 2023-09-11 06:02:07,488 o.a.f.k.o.o.d.ApplicationObserver [INFO ][rec-job/rec-job] JobManager is being deployed 2023-09-11 06:02:07,563 o.a.f.k.o.l.AuditUtils [INFO ][rec-job/rec-job] >>> Status | Info | SUSPENDED | The resource (job) has been suspended 2023-09-11 06:02:07,576 o.a.f.k.o.l.AuditUtils [INFO ][rec-job/rec-job] >>> Event | Info | SPECCHANGED | UPGRADE change(s) detected (Diff: FlinkDeploymentSpec[job.state : suspended -> running]), starting reconciliation. 2023-09-11 06:02:07,649 o.a.f.k.o.l.AuditUtils [INFO ][rec-job/rec-job] >>> Status | Info | UPGRADING | The resource is being upgraded 2023-09-11 06:02:07,649 o.a.f.k.o.r.d.ApplicationReconciler [INFO ][rec-job/rec-job] Deleting deployment with terminated application before new deployment 2023-09-11 06:02:07,649 o.a.f.k.o.s.AbstractFlinkService [INFO ][rec-job/rec-job] Deleting cluster with Foreground propagation 2023-09-11 06:02:07,649 o.a.f.k.o.s.NativeFlinkService [INFO ][rec-job/rec-job] Deleting JobManager deployment and HA metadata. 2023-09-11 06:02:07,691 o.a.f.k.o.s.AbstractFlinkService [INFO ][rec-job/rec-job] Waiting for cluster shutdown... 2023-09-11 06:02:07,763 o.a.f.k.o.s.AbstractFlinkService [INFO ][rec-job/rec-job] Cluster shutdown completed. 2023-09-11 06:02:07,763 o.a.f.k.o.s.AbstractFlinkService [INFO ][rec-job/rec-job] Deleting Kubernetes HA metadata 2023-09-11 06:02:07,820 o.a.f.k.o.s.AbstractFlinkService [INFO ][rec-job/rec-job] Waiting for cluster shutdown... 2023-09-11 06:02:07,831 o.a.f.k.o.s.AbstractFlinkService [INFO ][rec-job/rec-job] Cluster shutdown completed. 2023-09-11 06:02:07,975 o.a.f.k.o.l.AuditUtils [INFO ][rec-job/rec-job] >>> Status | Info | UPGRADING | The resource is being upgraded 2023-09-11 06:02:07,987 o.a.f.k.o.l.AuditUtils [INFO ][rec-job/rec-job] >>> Event | Info | SUBMIT | Starting deployment 2023-09-11 06:02:07,987 o.a.f.k.o.s.AbstractFlinkService [INFO ][rec-job/rec-job] Deploying application cluster requiring last-state from HA metadata 2023-09-11 06:02:07,999 o.a.f.k.o.c.FlinkDeploymentController [ERROR][rec-job/rec-job] Flink recovery failed 2023-09-11 06:02:08,012 o.a.f.k.o.l.AuditUtils [INFO ][rec-job/rec-job] >>> Event | Warning | RESTOREFAILED | HA metadata not available to restore from last state. It is possible that the job has finished or terminally failed, or the configmaps have been deleted. Manual restore required. 2023-09-11 06:02:08,099 o.a.f.k.o.l.AuditUtils [INFO ][rec-job/rec-job] >>> Status | Error | UPGRADING | {"type":"org.apache.flink.kubernetes.operator.exception.RecoveryFailureException","message":"HA metadata not available to restore from last state. It is possible that the job has finished or terminally failed, or the configmaps have been deleted. Manual restore required.","additionalMetadata":{},"throwableList":[]} 2023-09-11 06:02:08,193 o.a.f.k.o.l.AuditUtils [INFO ][rec-job/rec-job] >>> Status | Info | UPGRADING | The resource is being upgraded 2023-09-11 06:02:08,218 o.a.f.k.o.l.AuditUtils [INFO ][rec-job/rec-job] >>> Event | Info | SUBMIT | Starting deployment 2023-09-11 06:02:08,218 o.a.f.k.o.s.AbstractFlinkService [INFO ][rec-job/rec-job] Deploying application cluster requiring last-state from HA metadata 2023-09-11 06:02:08,228 o.a.f.k.o.c.FlinkDeploymentController [ERROR][rec-job/rec-job] Flink recovery failed ________________________________ "This message contains confidential information/commercial secret. If you are not the intended addressee of this message you may not copy, save, print or forward it to any third party and you are kindly requested to destroy this message and notify the sender thereof by email. Данное сообщение содержит конфиденциальную информацию/информацию, являющуюся коммерческой тайной. Если Вы не являетесь надлежащим адресатом данного сообщения, Вы не вправе копировать, сохранять, печатать или пересылать его каким либо иным лицам. Просьба уничтожить данное сообщение и уведомить об этом отправителя электронным письмом."