Dennis-Mircea Ciupitu created FLINK-41311:
---------------------------------------------
Summary: Unresponsive JobManager is never reported because READY
is set without a REST check
Key: FLINK-41311
URL: https://issues.apache.org/jira/browse/FLINK-41311
Project: Flink
Issue Type: Bug
Components: Kubernetes Operator
Reporter: Dennis-Mircea Ciupitu
h1. Problem
{{observeJmDeployment}} moves a {{FlinkDeployment}} from {{DEPLOYED_NOT_READY}}
to {{READY}} without any REST call, 10 s
({{kubernetes.operator.observer.rest-ready.delay}}) after the port check passed:
* {{DEPLOYED_NOT_READY}} only means that the Deployment has an available
replica and that the REST port accepts a TCP connection. Flink binds that port
before the Dispatcher holds leadership and before an application {{main()}}
submits the job.
* {{READY}} is documented as "ready to receive REST API calls", but nothing
checks that a call succeeds.
A JobManager that is alive and listening but never answers (frozen or
GC-stalled JVM, deadlocked Dispatcher, leader election that never completes) is
therefore never reported. Every reconcile:
* flips the status to {{READY}},
* times out after 10 s on {{/jobs/overview}},
* demotes it back to {{DEPLOYED_NOT_READY}} through the {{onTimeout}} hook and
reschedules in 10 s.
The status is patched once per reconcile, after the demotion, so for as long as
the JobManager stays stuck the resource shows a steady picture:
* {{jobManagerDeploymentStatus=DEPLOYED_NOT_READY}},
{{jobStatus.state=RECONCILING}}, {{lifecycleState=STABLE}}
* no {{status.error}}, no event, only a timeout warning in the operator log
every 20 s
* {{kubernetes.operator.deployment.readiness.timeout}} is never consulted
A stuck JobManager is indistinguishable from a slow start.
h1. Reproduction
The reproduction can be done within local minikube on
{{{}examples/basic.yaml{}}}. Once {{READY}} / {{RUNNING}} / {{{}STABLE{}}},
freeze the JobManager JVM from the node (inside the pod the signal is dropped,
the JVM is PID 1 of its namespace):
{code:java}
minikube ssh -- "sudo kill -STOP \$(pgrep -f
'KubernetesApplicationCluster[E]ntrypoint')"
{code}
The pod stays {{Running}} / {{Ready}} (no probes on the JobManager), the port
still accepts connections, HTTP hangs. Operator log, identical every 20 s for
the 10 minutes of the run (29 cycles):
{code:java}
2026-11-02 18:39:05,680 WARN ecutor-flinkdeploymentcontroller-395
o.a.f.k.o.observer.JobStatusObserver Exception while getting job status
java.util.concurrent.TimeoutException
at java.base/java.util.concurrent.CompletableFuture.timedGet(Unknown
Source)
at java.base/java.util.concurrent.CompletableFuture.get(Unknown Source)
at
org.apache.flink.kubernetes.operator.service.AbstractFlinkService.getJobStatus(AbstractFlinkService.java:314)
at
org.apache.flink.kubernetes.operator.observer.JobStatusObserver.observe(JobStatusObserver.java:89)
at
org.apache.flink.kubernetes.operator.observer.deployment.ApplicationObserver.observeFlinkCluster(ApplicationObserver.java:49)
at
org.apache.flink.kubernetes.operator.observer.deployment.AbstractFlinkDeploymentObserver.observeInternal(AbstractFlinkDeploymentObserver.java:71)
at
org.apache.flink.kubernetes.operator.observer.AbstractFlinkResourceObserver.observe(AbstractFlinkResourceObserver.java:49)
at
org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:145)
at
org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:59)
at
io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:191)
at
io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:147)
at
org.apache.flink.kubernetes.operator.metrics.OperatorJosdkMetrics.timeControllerExecution(OperatorJosdkMetrics.java:82)
at
io.javaoperatorsdk.operator.processing.Controller.reconcile(Controller.java:146)
at
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.reconcileExecution(ReconciliationDispatcher.java:164)
at
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleReconcile(ReconciliationDispatcher.java:145)
at
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleDispatch(ReconciliationDispatcher.java:111)
at
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:76)
at
io.javaoperatorsdk.operator.processing.event.EventProcessor$ReconcilerExecutor.run(EventProcessor.java:578)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.base/java.lang.Thread.run(Unknown Source)
2026-11-02 18:39:05,681 INFO ecutor-flinkdeploymentcontroller-395
o.a.f.k.o.o.d.ApplicationObserver Observing JobManager deployment. Previous
status: READY
2026-11-02 18:39:05,685 INFO ecutor-flinkdeploymentcontroller-395
o.a.f.k.o.o.d.ApplicationObserver JobManager deployment port is ready, waiting
for the Flink REST API...
2026-11-02 18:39:05,688 INFO ecutor-flinkdeploymentcontroller-395
.r.d.AbstractFlinkResourceReconciler Resource fully reconciled, nothing to do...
2026-11-02 18:39:10,729 INFO pool-10-thread-1
o.a.f.k.o.h.CanaryResourceManager Canary deployment healthy
2026-11-02 18:39:10,744 INFO ecutor-flinkdeploymentcontroller-401
o.a.f.k.o.h.CanaryResourceManager Reconciling canary resource
2026-11-02 18:39:10,744 INFO pool-10-thread-1
o.a.f.k.o.h.CanaryResourceManager Scheduling canary check for
ResourceID{name='canary', namespace='default'} in 60s
2026-11-02 18:39:15,696 INFO ecutor-flinkdeploymentcontroller-409
o.a.f.k.o.o.d.ApplicationObserver Observing JobManager deployment. Previous
status: DEPLOYED_NOT_READY
2026-11-02 18:39:15,696 INFO ecutor-flinkdeploymentcontroller-409
o.a.f.k.o.o.d.ApplicationObserver JobManager deployment is ready
2026-11-02 18:39:25,703 WARN ecutor-flinkdeploymentcontroller-409
o.a.f.k.o.observer.JobStatusObserver Exception while getting job status
...
{code}
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
