Dennis-Mircea Ciupitu created FLINK-41375:
---------------------------------------------
Summary: JobManager crash loops are detected only in the back-off
phase and reported without the termination reason
Key: FLINK-41375
URL: https://issues.apache.org/jira/browse/FLINK-41375
Project: Flink
Issue Type: Bug
Components: Kubernetes Operator
Reporter: Dennis-Mircea Ciupitu
h1. Problem
When the JobManager container keeps dying (for example, OOMKilled reason), what
the operator reports depends on the moment it happens to look at the pod, and
it never says why the container died.
The observer only recognises a crash loop while the kubelet is backing off
({{{}state.waiting.reason=CrashLoopBackOff{}}}). Between 2 kills, while the
container is {{running}} again, it reports {{DEPLOYING}} as if everything were
fine, even though the pod already says
{{{}lastState.terminated.reason=OOMKilled{}}}, {{exitCode=137}} and counts the
restarts.
The actual problem is that the {{OOMKilled}} entry in
{{CONTAINER_ERROR_REASONS}} never matches on the current observation logic.
Kubernetes puts that reason under {{{}terminated{}}}, not under
{{{}waiting{}}}. The longer the JobManager runs before each kill, the more
often the operator says {{{}DEPLOYING{}}}.
When it does detect the loop, the error and the event are a copy of the
kubelet's back-off sentence with the pod name. The reason and the exit code are
nowhere in the resource. And because that sentence changes with every back-off
step and every pod, each detection produces a new event instead of counting up
the existing one.
On a deployment that was already stable, the {{DEPLOYING}} sample also wipes
{{{}status.error{}}}, so the resource alternates between an error and no error
at all.
h1. Reproduction
{{examples/basic.yaml}} with the below extra line + "1024m" as JM spec resource
memory:
{code:yaml}
flinkConfiguration:
env.java.opts.jobmanager: "-Xms1200m -Xmx1200m -XX:+AlwaysPreTouch"
{code}
{code:java}
kubectl get pods
NAME READY STATUS RESTARTS AGE
basic-example-65774988d7-tzv74 0/1 OOMKilled 0 3s
kubectl get pods
NAME READY STATUS
RESTARTS AGE
basic-example-65774988d7-tzv74 0/1 CrashLoopBackOff 2 (3s
ago) 20s
{code}
Operator log:
{code}
2026-11-07 09:59:52,330 INFO ecutor-flinkdeploymentcontroller-350
.r.d.AbstractFlinkResourceReconciler Deploying for the first time
2026-11-07 09:59:52,335 INFO ecutor-flinkdeploymentcontroller-350
o.a.f.k.operator.listener.AuditUtils >>> Status[Job] | Info | UPGRADING
| The resource is being upgraded
2026-11-07 09:59:52,336 INFO ecutor-flinkdeploymentcontroller-350
o.a.f.k.o.r.d.ApplicationReconciler Assigning JobId override to
5efdcfec84cb50f1fca3c5aecd224f0b
2026-11-07 09:59:52,343 INFO ecutor-flinkdeploymentcontroller-350
o.a.f.k.operator.listener.AuditUtils >>> Event[Job] | Info | SUBMIT
| Starting deployment
2026-11-07 09:59:52,343 INFO ecutor-flinkdeploymentcontroller-350
o.a.f.k.o.s.AbstractFlinkService Deploying application cluster
2026-11-07 09:59:52,343 INFO ecutor-flinkdeploymentcontroller-350
o.a.f.k.o.service.NativeFlinkService Deploying application cluster
2026-11-07 09:59:52,343 INFO ecutor-flinkdeploymentcontroller-350
f.c.d.a.c.ApplicationClusterDeployer Submitting application in 'Application
Mode'.
2026-11-07 09:59:52,346 INFO ecutor-flinkdeploymentcontroller-350
o.a.f.r.u.c.m.ProcessMemoryUtils The derived from fraction jvm overhead memory
(102.400mb (107374184 bytes)) is less than its min value 192.000mb (201326592
bytes), min value will be used instead
2026-11-07 09:59:52,392 INFO ecutor-flinkdeploymentcontroller-350
.a.DefaultKubernetesArtifactUploader Local artifact uploading is disabled. Set
'kubernetes.artifacts.local-upload-enabled' to enable.
2026-11-07 09:59:52,392 INFO ecutor-flinkdeploymentcontroller-350
o.a.f.k.utils.KubernetesUtils Kubernetes deployment requires a fixed port.
Configuration blob.server.port will be set to 6124
2026-11-07 09:59:52,393 INFO ecutor-flinkdeploymentcontroller-350
o.a.f.k.utils.KubernetesUtils Kubernetes deployment requires a fixed port.
Configuration taskmanager.rpc.port will be set to 6122
2026-11-07 09:59:52,401 INFO ecutor-flinkdeploymentcontroller-350
o.a.f.configuration.Configuration Config uses fallback configuration key
'kubernetes.service-account' instead of key
'kubernetes.jobmanager.service-account'
2026-11-07 09:59:52,479 WARN ecutor-flinkdeploymentcontroller-350
o.a.f.k.KubernetesClusterDescriptor Please note that Flink client
operations(e.g. cancel, list, stop, savepoint, etc.) won't work from outside
the Kubernetes cluster since 'kubernetes.rest-service.exposed.type' has been
set to ClusterIP.
2026-11-07 09:59:52,480 INFO ecutor-flinkdeploymentcontroller-350
o.a.f.k.KubernetesClusterDescriptor Create flink application cluster
basic-example successfully, JobManager Web Interface:
http://basic-example-rest.default:8081
2026-11-07 09:59:52,480 INFO ecutor-flinkdeploymentcontroller-350
o.a.f.k.o.service.NativeFlinkService Application cluster successfully deployed
2026-11-07 09:59:52,486 INFO ecutor-flinkdeploymentcontroller-350
o.a.f.k.operator.listener.AuditUtils >>> Status[Job] | Info | DEPLOYED
| The resource is deployed/submitted to Kubernetes, but it’s not yet
considered to be stable and might be rolled back in the future
2026-11-07 09:59:52,486 INFO ecutor-flinkdeploymentcontroller-350
o.a.f.k.o.o.d.ApplicationObserver Observing JobManager deployment. Previous
status: DEPLOYING
2026-11-07 09:59:52,489 INFO ecutor-flinkdeploymentcontroller-350
o.a.f.k.o.o.d.ApplicationObserver JobManager is being deployed
2026-11-07 09:59:52,491 INFO ecutor-flinkdeploymentcontroller-350
o.a.f.k.o.a.state.ConfigMapView Creating ConfigMap autoscaler-basic-example
2026-11-07 09:59:52,492 INFO ecutor-flinkdeploymentcontroller-350
.r.d.AbstractFlinkResourceReconciler Resource fully reconciled, nothing to do...
2026-11-07 10:00:02,495 INFO ecutor-flinkdeploymentcontroller-369
o.a.f.k.o.o.d.ApplicationObserver Observing JobManager deployment. Previous
status: DEPLOYING
2026-11-07 10:00:02,508 INFO ecutor-flinkdeploymentcontroller-369
o.a.f.k.operator.listener.AuditUtils >>> Event[Job] | Warning |
CRASHLOOPBACKOFF | [flink-main-container] back-off 10s restarting failed
container=flink-main-container
pod=basic-example-65774988d7-tzv74_default(b57eb99c-a3cf-4fad-8db0-d7b144669a0d)
{code}
This is the error the resource keeps reporting. Nothing in it says why the
container died:
{code:java}
kubectl get flinkdep basic-example -o jsonpath='{.status.error}'
{"type":"org.apache.flink.kubernetes.operator.exception.DeploymentFailedException","message":"[flink-main-container]
back-off 10s restarting failed container=flink-main-container
pod=basic-example-65774988d7-q8x7d_default(deccff18-576e-4a16-aeb4-6efaf894eb9a)","stackTrace":"org.apache.flink.kubernetes.operator.exception.DeploymentFailedException:
[flink-main-container] back-off 10s restarting failed
container=flink-main-container
pod=basic-example-65774988d7-q8x7d_default(deccff18-576e-4a16-aeb4-6efaf894eb9a)\n\tat
org.apache.flink.kubernetes.operator.exception.DeploymentFailedException.forContainerStatus(DeploymentFailedException.java:74)\n\tat
org.apache.flink.kubernetes.operator.observer.deployment.AbstractFlinkDeploymentObserver.checkContainerError(AbstractFlinkDeploymentObserver.java:251)\n\tat
java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown
Source)\n\tat
java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Unknown
Source)\n\tat java.base/java.util.stream.ReferencePipeline$Head.forEach(Unknown
Source)\n\tat
org.apache.flink.kubernetes.operator.observer.deployment.AbstractFlinkDeploymentObserver.checkContainerErrors(AbstractFlinkDeploymentObserver.java:238)\n\tat
org.apache.flink.kubernetes.operator.observer.deployment.AbstractFlinkDeploymentObserver.observeJmDeployment(AbstractFlinkDeploymentObserver.java:193)\n\tat
org.apache.flink.kubernetes.operator.observer.deployment.AbstractFlinkDeploymentObserver.observeInternal(AbstractFlinkDeploymentObserver.java:89)\n\tat
org.apache.flink.kubernetes.operator.observer.AbstractFlinkResourceObserver.observe(AbstractFlinkResourceObserver.java:49)\n\tat
org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:145)\n\tat
org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:59)\n\tat
io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:191)\n\tat
io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:147)\n\tat
org.apache.flink.kubernetes.operator.metrics.OperatorJosdkMetrics.timeControllerExecution(OperatorJosdkMetrics.java:82)\n\tat
io.javaoperatorsdk.operator.processing.Controller.reconcile(Controller.java:146)\n\tat
io.javaoperatorsdk.operator.processing.event.Rec","additionalMetadata":{"reason":"CrashLoopBackOff"},"throwableList":[]}
{code}
Sampled every 5 s over two runs (first deployment, and a stable deployment
whose JobManager starts crash-looping): every observe that lands in the
{{running}} window reports {{DEPLOYING}} with
{{{}lastState.terminated.reason=OOMKilled{}}}; on the stable deployment it also
wipes {{status.error}} for one progress-check interval; two
{{CrashLoopBackOff}} events in six minutes per run, one per back-off step.
h1. Expected
* A JobManager container that has already died with a non-zero exit code and
been restarted is reported as crash-looping whenever the operator looks at it,
whether the kubelet is currently backing off or the container is up again
between two kills.
* {{status.error}} and the event name the termination reason and exit code,
and the text does not change with the back-off timer or the pod name, so one
event is counted up instead of a new one per step.
* The error is not cleared until the JobManager is actually {{{}READY{}}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
