Gyula Fora created FLINK-29464:
----------------------------------
Summary: JobID generation logic could lead to state loss
Key: FLINK-29464
URL: https://issues.apache.org/jira/browse/FLINK-29464
Project: Flink
Issue Type: Improvement
Components: Kubernetes Operator
Affects Versions: kubernetes-operator-1.2.0
Reporter: Gyula Fora
Assignee: Gyula Fora
Fix For: kubernetes-operator-1.2.0
The recently added job id override logic
(https://issues.apache.org/jira/browse/FLINK-29109) can under certain cases
lead to state loss.
State loss scenario:
1. Either first deployment / Stateless upgrade mode used -> new jobId will be
generated and set in jobStatus
2. Operator/deployment fails during or directly after successful submission ->
status is not persisted with the generated jobId
3. User submits a spec update with last-state upgrade
4. If the job was never observed (due to a failure or early spec update) a
last-state upgrade would be performed, deleting the Deployment and simply
submitting the job.
5. The current logic would then generate a new jobid (because it's still empty)
leading to a failure to recover the state from HA -> data loss
There are multiple ways to solve this issue:
a ) Record status after generating a jobid
b ) Only ever set the status during stateless deployment
c ) Verify no HA data is present before setting the jobid when empty
Probably the most robust solution is a).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
