Santwana Verma created FLINK-41262:
--------------------------------------
Summary: Suspended job runs again from scratch if its JobManager
restarts (no HA)
Key: FLINK-41262
URL: https://issues.apache.org/jira/browse/FLINK-41262
Project: Flink
Issue Type: Bug
Components: Kubernetes Operator
Reporter: Santwana Verma
When you suspend an application-mode FlinkDeployment with `upgradeMode:
savepoint`, the operator stops the job with a savepoint but leaves the
JobManager running until jm-`deployment.shutdown-ttl `runs out, which is a day
by default.
If you don't have HA configured and that JobManager pod restarts in the
meantime (OOMKill, eviction, node drain, someone deleting the pod), the
entrypoint just submits the job again. It starts over from its initial state.
With a Kafka source that means it reads from the startup offsets again and
writes everything to the sink a second time.
The operator doesn't notice any of this. It stops observing the job once the
spec is suspended, so the resource still says SUSPENDED / FINISHED while the
job runs.
We hit this when a kept JobManager got OOMKilled overnight. To reproduce it on
demand:
# Deploy a streaming job in native application mode without
high-availability.type, and wait until it's RUNNING.
# Set job.state: suspended with upgradeMode: savepoint. The resource goes to
SUSPENDED, but the JobManager pod stays up.
# Delete the JobManager pod.
# The new JobManager logs "Submitting Job with JobId=..." and "No checkpoint
found during restore", and the job goes back to RUNNING. The resource still
shows SUSPENDED.
With Kubernetes HA turned on, the same steps don't resubmit the job. The
operator sets job-result-store.delete-on-commit: false, so the restarted
JobManager can see that the job already finished.
The CANCEL suspend mode has the same problem, and so does suspending a job that
has already finished. Standalone mode isn't affected, because it always deletes
the cluster on suspend.
I'd suggest deleting the JobManager Deployment as part of the suspend when HA
isn't enabled, like the stateless and last-state modes already do. With HA we
can keep the current behaviour. The downside is that without HA you lose the
Flink UI for a suspended job right away instead of a day later.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
