[
https://issues.apache.org/jira/browse/FLINK-41311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-41311:
-----------------------------------
Labels: pull-request-available (was: )
> Unresponsive JobManager is never reported
> -----------------------------------------
>
> Key: FLINK-41311
> URL: https://issues.apache.org/jira/browse/FLINK-41311
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
> Reporter: Dennis-Mircea Ciupitu
> Priority: Major
> Labels: pull-request-available
>
> h1. Problem
> {{observeJmDeployment}} moves a {{FlinkDeployment}} from
> {{DEPLOYED_NOT_READY}} to {{READY}} without any REST call, 10 s
> ({{kubernetes.operator.observer.rest-ready.delay}}) after the port check
> passed:
> * {{DEPLOYED_NOT_READY}} only means that the Deployment has an available
> replica and that the REST port accepts a TCP connection. Flink binds that
> port before the Dispatcher holds leadership and before an application
> {{main()}} submits the job.
> * {{READY}} is documented as "ready to receive REST API calls", but nothing
> checks that a call succeeds.
> A JobManager that is alive and listening but never answers (frozen or
> GC-stalled JVM, deadlocked Dispatcher, leader election that never completes)
> is therefore never reported. Every reconcile:
> * flips the status to {{READY}},
> * times out after 10 s on {{/jobs/overview}},
> * demotes it back to {{DEPLOYED_NOT_READY}} through the {{onTimeout}} hook
> and reschedules in 10 s.
> The status is patched once per reconcile, after the demotion, so for as long
> as the JobManager stays stuck the resource shows a steady picture:
> * {{jobManagerDeploymentStatus=DEPLOYED_NOT_READY}},
> {{jobStatus.state=RECONCILING}}, {{lifecycleState=STABLE}}
> * no {{status.error}}, no event, only a timeout warning in the operator log
> every 20 s
> * {{kubernetes.operator.deployment.readiness.timeout}} is never consulted
> A stuck JobManager is indistinguishable from a slow start.
> h1. Reproduction
> The reproduction can be done within local minikube on
> {{{}examples/basic.yaml{}}}. Once {{READY}} / {{RUNNING}} / {{{}STABLE{}}},
> freeze the JobManager JVM from the node (inside the pod the signal is
> dropped, the JVM is PID 1 of its namespace):
> {code:java}
> minikube ssh -- "sudo kill -STOP \$(pgrep -f
> 'KubernetesApplicationCluster[E]ntrypoint')"
> {code}
> The pod stays {{Running}} / {{Ready}} (no probes on the JobManager), the port
> still accepts connections, HTTP hangs. Operator log, identical every 20 s for
> the 10 minutes of the run (29 cycles):
> {code:java}
> 2026-11-02 18:39:05,680 WARN ecutor-flinkdeploymentcontroller-395
> o.a.f.k.o.observer.JobStatusObserver Exception while getting job status
> java.util.concurrent.TimeoutException
> at java.base/java.util.concurrent.CompletableFuture.timedGet(Unknown
> Source)
> at java.base/java.util.concurrent.CompletableFuture.get(Unknown
> Source)
> at
> org.apache.flink.kubernetes.operator.service.AbstractFlinkService.getJobStatus(AbstractFlinkService.java:314)
> at
> org.apache.flink.kubernetes.operator.observer.JobStatusObserver.observe(JobStatusObserver.java:89)
> at
> org.apache.flink.kubernetes.operator.observer.deployment.ApplicationObserver.observeFlinkCluster(ApplicationObserver.java:49)
> at
> org.apache.flink.kubernetes.operator.observer.deployment.AbstractFlinkDeploymentObserver.observeInternal(AbstractFlinkDeploymentObserver.java:71)
> at
> org.apache.flink.kubernetes.operator.observer.AbstractFlinkResourceObserver.observe(AbstractFlinkResourceObserver.java:49)
> at
> org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:145)
> at
> org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:59)
> at
> io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:191)
> at
> io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:147)
> at
> org.apache.flink.kubernetes.operator.metrics.OperatorJosdkMetrics.timeControllerExecution(OperatorJosdkMetrics.java:82)
> at
> io.javaoperatorsdk.operator.processing.Controller.reconcile(Controller.java:146)
> at
> io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.reconcileExecution(ReconciliationDispatcher.java:164)
> at
> io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleReconcile(ReconciliationDispatcher.java:145)
> at
> io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleDispatch(ReconciliationDispatcher.java:111)
> at
> io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:76)
> at
> io.javaoperatorsdk.operator.processing.event.EventProcessor$ReconcilerExecutor.run(EventProcessor.java:578)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.base/java.lang.Thread.run(Unknown Source)
> 2026-11-02 18:39:05,681 INFO ecutor-flinkdeploymentcontroller-395
> o.a.f.k.o.o.d.ApplicationObserver Observing JobManager deployment. Previous
> status: READY
> 2026-11-02 18:39:05,685 INFO ecutor-flinkdeploymentcontroller-395
> o.a.f.k.o.o.d.ApplicationObserver JobManager deployment port is ready,
> waiting for the Flink REST API...
> 2026-11-02 18:39:05,688 INFO ecutor-flinkdeploymentcontroller-395
> .r.d.AbstractFlinkResourceReconciler Resource fully reconciled, nothing to
> do...
> 2026-11-02 18:39:10,729 INFO pool-10-thread-1
> o.a.f.k.o.h.CanaryResourceManager Canary deployment healthy
> 2026-11-02 18:39:10,744 INFO ecutor-flinkdeploymentcontroller-401
> o.a.f.k.o.h.CanaryResourceManager Reconciling canary resource
> 2026-11-02 18:39:10,744 INFO pool-10-thread-1
> o.a.f.k.o.h.CanaryResourceManager Scheduling canary check for
> ResourceID{name='canary', namespace='default'} in 60s
> 2026-11-02 18:39:15,696 INFO ecutor-flinkdeploymentcontroller-409
> o.a.f.k.o.o.d.ApplicationObserver Observing JobManager deployment. Previous
> status: DEPLOYED_NOT_READY
> 2026-11-02 18:39:15,696 INFO ecutor-flinkdeploymentcontroller-409
> o.a.f.k.o.o.d.ApplicationObserver JobManager deployment is ready
> 2026-11-02 18:39:25,703 WARN ecutor-flinkdeploymentcontroller-409
> o.a.f.k.o.observer.JobStatusObserver Exception while getting job status
> ...
> {code}
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
