[
https://issues.apache.org/jira/browse/FLINK-41262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-41262:
-----------------------------------
Labels: pull-request-available (was: )
> Suspended job runs again from scratch if its JobManager restarts (no HA)
> ------------------------------------------------------------------------
>
> Key: FLINK-41262
> URL: https://issues.apache.org/jira/browse/FLINK-41262
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
> Reporter: Santwana Verma
> Priority: Minor
> Labels: pull-request-available
>
> When you suspend an application-mode FlinkDeployment with {{{}upgradeMode:
> savepoint{}}}, the operator stops the job with a savepoint but leaves the
> JobManager running until \{{jm-deployment.shutdown-ttl }} runs out, which is
> a day by default.
> If you don't have HA configured and that JobManager pod restarts in the
> meantime (OOMKill, eviction, node drain, someone deleting the pod), the
> entrypoint just submits the job again. It starts over from its initial state.
> With a Kafka source that means it reads from the startup offsets again and
> writes everything to the sink a second time.
> The operator stops observing the job once the spec is suspended, so the
> resource still says SUSPENDED / FINISHED while the job runs.
> We hit this when a kept JobManager got OOMKilled overnight. To reproduce it:
> # Deploy a streaming job in native application mode without
> high-availability.type, and wait until it's RUNNING.
> # Set job.state: suspended with upgradeMode: savepoint. The resource goes to
> SUSPENDED, but the JobManager pod stays up.
> # Delete the JobManager pod.
> # The new JobManager logs "Submitting Job with JobId=..." and "No checkpoint
> found during restore", and the job goes back to RUNNING. The resource still
> shows SUSPENDED.
> With Kubernetes HA turned on, the same steps don't resubmit the job. The
> operator sets job-result-store.delete-on-commit: false, so the restarted
> JobManager can see that the job already finished.
> The CANCEL suspend mode has the same problem, and so does suspending a job
> that has already finished.
> Standalone mode isn't affected, because it always deletes the cluster on
> suspend.
> I'd suggest deleting the JobManager Deployment as part of the suspend when HA
> isn't enabled, like the stateless and last-state modes already do.
> +The downside is that without HA, the Flink UI and the pod logs of a
> suspended job go away right after the suspend instead of a day later.+ Why
> the job stopped is still on the resource: the suspend itself, its savepoint,
> and for a job that failed earlier, the error the operator already copied into
> the status and events.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
