Dennis-Mircea opened a new pull request, #1250:
URL: https://github.com/apache/flink-kubernetes-operator/pull/1250
## What is the purpose of the change
This PR addresses a JobManager container that keeps dying was detected only
while the kubelet was backing off, reported as `DEPLOYING` in between, and
described with the kubelet's back-off sentence instead of the reason it died.
The observer now recognizes the crash from the container's termination record
in every phase, names the reason and exit code, and keeps the error until the
JobManager is `READY` again.
## Brief change log
- `DeploymentFailedException`: new `failedTermination` (non-zero exit in
`state.terminated` or `lastState.terminated`) and `fromContainerStatus`, which
describes a crash loop from that termination (`CrashLoopBackOff, last
termination OOMKilled (exit code 137)`), keeps the kubelet message for the
other waiting reasons, and is empty for a healthy container. `OOMKilled`
dropped from `CONTAINER_ERROR_REASONS`, it never matched there.
- `AbstractFlinkDeploymentObserver`: `checkContainerErrors` throws the first
such failure, so the running phase between two kills is reported like the
back-off phase, with identical text, and one event is counted up.
`clearErrorsIfDeploymentIsHealthy` clears the error only at `READY`.
## Verifying this change
This change can be verified by the newly added tests.
This change was also verified manually on minikube with
`examples/basic.yaml` with the configuration described within the JIRA.
Now the deployment gets stuck in the following state:
```
kubectl get flinkdep basic-example -w
NAME AGE JOBMANAGER STATUS JOB STATUS LIFECYCLE STATE
basic-example 58s ERROR RECONCILING DEPLOYED
```
The log sequencing is the following:
```
2026-11-07 20:52:20,764 INFO ecutor-flinkdeploymentcontroller-105
o.a.f.k.o.o.d.ApplicationObserver Observing JobManager deployment. Previous
status: DEPLOYING
2026-11-07 20:52:20,783 INFO ecutor-flinkdeploymentcontroller-105
o.a.f.k.o.o.d.ApplicationObserver JobManager is being deployed
2026-11-07 20:52:20,788 INFO ecutor-flinkdeploymentcontroller-105
o.a.f.k.o.a.state.ConfigMapView Creating ConfigMap autoscaler-basic-example
2026-11-07 20:52:20,796 INFO ecutor-flinkdeploymentcontroller-105
.r.d.AbstractFlinkResourceReconciler Resource fully reconciled, nothing to do...
2026-11-07 20:52:30,816 INFO ecutor-flinkdeploymentcontroller-109
o.a.f.k.o.o.d.ApplicationObserver Observing JobManager deployment. Previous
status: DEPLOYING
2026-11-07 20:52:30,836 INFO ecutor-flinkdeploymentcontroller-109
o.a.f.k.operator.listener.AuditUtils >>> Event[Job] | Warning |
CRASHLOOPBACKOFF | [flink-main-container] CrashLoopBackOff, last termination
OOMKilled (exit code 137)
```
The `status.error` mentions explicitly the termination reason:
```
kubectl get flinkdep basic-example -o jsonpath='{.status.error}'
{"type":"org.apache.flink.kubernetes.operator.exception.DeploymentFailedException","message":"[flink-main-container]
CrashLoopBackOff, last termination OOMKilled (exit code
137)","stackTrace":"org.apache.flink.kubernetes.operator.exception.DeploymentFailedException:
[flink-main-container] CrashLoopBackOff, last termination OOMKilled (exit code
137)\n\tat
org.apache.flink.kubernetes.operator.exception.DeploymentFailedException.fromContainerStatus(DeploymentFailedException.java:100)\n\tat
java.base/java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)\n\tat
java.base/java.util.ArrayList$ArrayListSpliterator.tryAdvance(Unknown
Source)\n\tat
java.base/java.util.stream.Streams$ConcatSpliterator.tryAdvance(Unknown
Source)\n\tat
java.base/java.util.stream.ReferencePipeline.forEachWithCancel(Unknown
Source)\n\tat
java.base/java.util.stream.AbstractPipeline.copyIntoWithCancel(Unknown
Source)\n\tat java.base/java.util.stream.AbstractPipeline.copyInto(Unknown
Source)\n\tat java.base/
java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)\n\tat
java.base/java.util.stream.FindOps$FindOp.evaluateSequential(Unknown
Source)\n\tat java.base/java.util.stream.AbstractPipeline.evaluate(Unknown
Source)\n\tat java.base/java.util.stream.ReferencePipeline.findFirst(Unknown
Source)\n\tat
org.apache.flink.kubernetes.operator.observer.deployment.AbstractFlinkDeploymentObserver.checkContainerErrors(AbstractFlinkDeploymentObserver.java:193)\n\tat
org.apache.flink.kubernetes.operator.observer.deployment.AbstractFlinkDeploymentObserver.observeJmDeployment(AbstractFlinkDeploymentObserver.java:144)\n\tat
org.apache.flink.kubernetes.operator.observer.deployment.AbstractFlinkDeploymentObserver.observeInternal(AbstractFlinkDeploymentObserver.java:65)\n\tat
org.apache.flink.kubernetes.operator.observer.AbstractFlinkResourceObserver.observe(AbstractFlinkResourceObserver.java:49)\n\tat
org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeplo
ymentController.java:145)\n\tat
org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:59)\n\tat
io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:191)\n\tat
io.javaoperatorsdk.operator.processing.Cont","additionalMetadata":{"reason":"CrashLoopBackOff"},"throwableList":[]}
```
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changes to the `CustomResourceDescriptors`:
no
- Core observer or reconciler logic that is regularly executed: yes
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR,
please
change the checkbox below to `[X]` followed by the name of the tool, and
uncomment the
"Generated-by" line. See the ASF Generative Tooling Guidance for details:
https://www.apache.org/legal/generative-tooling.html
You are responsible for the quality and correctness of every change in this
PR
regardless of the tooling used. Low-effort AI-generated PRs will be closed.
See
AGENTS.md for the full guidance.
-->
- [X] Yes (please specify the tool below)
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
