[
https://issues.apache.org/jira/browse/FLINK-41262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Santwana Verma updated FLINK-41262:
-----------------------------------
Priority: Minor (was: Major)
> Suspended job runs again from scratch if its JobManager restarts (no HA)
> ------------------------------------------------------------------------
>
> Key: FLINK-41262
> URL: https://issues.apache.org/jira/browse/FLINK-41262
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
> Reporter: Santwana Verma
> Priority: Minor
>
> When you suspend an application-mode FlinkDeployment with `upgradeMode:
> savepoint`, the operator stops the job with a savepoint but leaves the
> JobManager running until jm-`deployment.shutdown-ttl `runs out, which is a
> day by default.
> If you don't have HA configured and that JobManager pod restarts in the
> meantime (OOMKill, eviction, node drain, someone deleting the pod), the
> entrypoint just submits the job again. It starts over from its initial state.
> With a Kafka source that means it reads from the startup offsets again and
> writes everything to the sink a second time.
> The operator doesn't notice any of this. It stops observing the job once the
> spec is suspended, so the resource still says SUSPENDED / FINISHED while the
> job runs.
> We hit this when a kept JobManager got OOMKilled overnight. To reproduce it
> on demand:
> # Deploy a streaming job in native application mode without
> high-availability.type, and wait until it's RUNNING.
> # Set job.state: suspended with upgradeMode: savepoint. The resource goes to
> SUSPENDED, but the JobManager pod stays up.
> # Delete the JobManager pod.
> # The new JobManager logs "Submitting Job with JobId=..." and "No checkpoint
> found during restore", and the job goes back to RUNNING. The resource still
> shows SUSPENDED.
> With Kubernetes HA turned on, the same steps don't resubmit the job. The
> operator sets job-result-store.delete-on-commit: false, so the restarted
> JobManager can see that the job already finished.
> The CANCEL suspend mode has the same problem, and so does suspending a job
> that has already finished. Standalone mode isn't affected, because it always
> deletes the cluster on suspend.
> I'd suggest deleting the JobManager Deployment as part of the suspend when HA
> isn't enabled, like the stateless and last-state modes already do.
> +The downside is that without HA, the Flink UI and the pod logs of a
> suspended job go away right after the suspend instead of a day later.+ Why
> the job stopped is still on the resource: the suspend itself, its savepoint,
> and for a job that failed earlier, the error the operator already copied into
> the status and events.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
