[
https://issues.apache.org/jira/browse/FLINK-41262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Santwana Verma updated FLINK-41262:
-----------------------------------
Description:
When you suspend an application-mode FlinkDeployment with {{{}upgradeMode:
savepoint{}}}, the operator stops the job with a savepoint but leaves the
JobManager running until {{jm-deployment.shutdown-ttl }}runs out, which is a
day by default.
If you don't have HA configured and that JobManager pod restarts in the
meantime (OOMKill, eviction, node drain, someone deleting the pod), the
entrypoint just submits the job again. It starts over from its initial state.
With a Kafka source that means it reads from the startup offsets again and
writes everything to the sink a second time.
The operator doesn't notice any of this. It stops observing the job once the
spec is suspended, so the resource still says SUSPENDED / FINISHED while the
job runs.
We hit this when a kept JobManager got OOMKilled overnight. To reproduce it on
demand:
# Deploy a streaming job in native application mode without
high-availability.type, and wait until it's RUNNING.
# Set job.state: suspended with upgradeMode: savepoint. The resource goes to
SUSPENDED, but the JobManager pod stays up.
# Delete the JobManager pod.
# The new JobManager logs "Submitting Job with JobId=..." and "No checkpoint
found during restore", and the job goes back to RUNNING. The resource still
shows SUSPENDED.
With Kubernetes HA turned on, the same steps don't resubmit the job. The
operator sets job-result-store.delete-on-commit: false, so the restarted
JobManager can see that the job already finished.
The CANCEL suspend mode has the same problem, and so does suspending a job that
has already finished. Standalone mode isn't affected, because it always deletes
the cluster on suspend.
I'd suggest deleting the JobManager Deployment as part of the suspend when HA
isn't enabled, like the stateless and last-state modes already do.
+The downside is that without HA, the Flink UI and the pod logs of a suspended
job go away right after the suspend instead of a day later.+ Why the job
stopped is still on the resource: the suspend itself, its savepoint, and for a
job that failed earlier, the error the operator already copied into the status
and events.
was:
When you suspend an application-mode FlinkDeployment with {{{}upgradeMode:
savepoint{}}}, the operator stops the job with a savepoint but leaves the
JobManager running until jm-deployment.shutdown-ttl runs out, which is a day by
default.
If you don't have HA configured and that JobManager pod restarts in the
meantime (OOMKill, eviction, node drain, someone deleting the pod), the
entrypoint just submits the job again. It starts over from its initial state.
With a Kafka source that means it reads from the startup offsets again and
writes everything to the sink a second time.
The operator doesn't notice any of this. It stops observing the job once the
spec is suspended, so the resource still says SUSPENDED / FINISHED while the
job runs.
We hit this when a kept JobManager got OOMKilled overnight. To reproduce it on
demand:
# Deploy a streaming job in native application mode without
high-availability.type, and wait until it's RUNNING.
# Set job.state: suspended with upgradeMode: savepoint. The resource goes to
SUSPENDED, but the JobManager pod stays up.
# Delete the JobManager pod.
# The new JobManager logs "Submitting Job with JobId=..." and "No checkpoint
found during restore", and the job goes back to RUNNING. The resource still
shows SUSPENDED.
With Kubernetes HA turned on, the same steps don't resubmit the job. The
operator sets job-result-store.delete-on-commit: false, so the restarted
JobManager can see that the job already finished.
The CANCEL suspend mode has the same problem, and so does suspending a job that
has already finished. Standalone mode isn't affected, because it always deletes
the cluster on suspend.
I'd suggest deleting the JobManager Deployment as part of the suspend when HA
isn't enabled, like the stateless and last-state modes already do.
+The downside is that without HA, the Flink UI and the pod logs of a suspended
job go away right after the suspend instead of a day later.+ Why the job
stopped is still on the resource: the suspend itself, its savepoint, and for a
job that failed earlier, the error the operator already copied into the status
and events.
> Suspended job runs again from scratch if its JobManager restarts (no HA)
> ------------------------------------------------------------------------
>
> Key: FLINK-41262
> URL: https://issues.apache.org/jira/browse/FLINK-41262
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
> Reporter: Santwana Verma
> Priority: Minor
>
> When you suspend an application-mode FlinkDeployment with {{{}upgradeMode:
> savepoint{}}}, the operator stops the job with a savepoint but leaves the
> JobManager running until {{jm-deployment.shutdown-ttl }}runs out, which is a
> day by default.
> If you don't have HA configured and that JobManager pod restarts in the
> meantime (OOMKill, eviction, node drain, someone deleting the pod), the
> entrypoint just submits the job again. It starts over from its initial state.
> With a Kafka source that means it reads from the startup offsets again and
> writes everything to the sink a second time.
> The operator doesn't notice any of this. It stops observing the job once the
> spec is suspended, so the resource still says SUSPENDED / FINISHED while the
> job runs.
> We hit this when a kept JobManager got OOMKilled overnight. To reproduce it
> on demand:
> # Deploy a streaming job in native application mode without
> high-availability.type, and wait until it's RUNNING.
> # Set job.state: suspended with upgradeMode: savepoint. The resource goes to
> SUSPENDED, but the JobManager pod stays up.
> # Delete the JobManager pod.
> # The new JobManager logs "Submitting Job with JobId=..." and "No checkpoint
> found during restore", and the job goes back to RUNNING. The resource still
> shows SUSPENDED.
> With Kubernetes HA turned on, the same steps don't resubmit the job. The
> operator sets job-result-store.delete-on-commit: false, so the restarted
> JobManager can see that the job already finished.
> The CANCEL suspend mode has the same problem, and so does suspending a job
> that has already finished. Standalone mode isn't affected, because it always
> deletes the cluster on suspend.
> I'd suggest deleting the JobManager Deployment as part of the suspend when HA
> isn't enabled, like the stateless and last-state modes already do.
> +The downside is that without HA, the Flink UI and the pod logs of a
> suspended job go away right after the suspend instead of a day later.+ Why
> the job stopped is still on the resource: the suspend itself, its savepoint,
> and for a job that failed earlier, the error the operator already copied into
> the status and events.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
