DanielLeens opened a new issue, #11587: URL: https://github.com/apache/seatunnel/issues/11587
## Background In separated Zeta clusters, Worker nodes run as Hazelcast Lite Members and should not own IMap partitions. However, a production Kubernetes case showed that when a Worker becomes unhealthy because of memory pressure, long GC pauses, or OOM risk, metrics-related REST/control-plane requests can still be amplified by synchronous RPC waits before Hazelcast removes the stale member. The root cause of that production incident was the Worker JVM/container memory layout, not Separated Cluster Mode itself. After reducing the Worker heap and leaving enough native-memory headroom, the cluster recovered. However, the case exposed a hardening gap in the metrics retrieval path. ## Current behavior Several metrics retrieval paths synchronously wait for Worker-side operations: - `JobMaster#getCurrJobMetrics()` sends `GetTaskGroupMetricsOperation` to Worker nodes and waits on the invocation future. - `CoordinatorService#getRunningJobMetrics()` sends `GetMetricsOperation` to Worker nodes and waits on the invocation future. - REST job-info / running metrics requests can therefore wait behind an unhealthy or slow Worker while the member is still visible in Hazelcast membership. When one Worker is experiencing long GC pauses, native-memory pressure, or delayed response, metrics and job-info requests can become slow or blocked, even though the control plane should ideally remain available with degraded metrics. ## Expected behavior Metrics retrieval should be best-effort and bounded: 1. Use an explicit request deadline for metrics collection. 2. Query Workers concurrently rather than serially where possible. 3. Return partial metrics when some Workers are unavailable or stale. 4. Mark missing Worker metrics as unavailable/stale instead of blocking the whole response. 5. Keep job status and control-plane metadata available even if live Worker metrics cannot be collected. 6. Avoid relying only on global Hazelcast operation timeouts, because those timeouts also affect other engine operations. ## Suggested scope - Add bounded timeout handling around Worker metrics RPCs. - Avoid serial unbounded waits across Workers. - Preserve existing metrics JSON compatibility as much as possible. - Add tests for one unreachable/slow Worker not blocking the whole metrics response. - Document the degraded/partial metrics behavior if the response shape changes. ## Related but separate work This is different from: - #10997 / #11564, which focus on offloading `ReportMetricsOperation` from Hazelcast generic-operation threads. - #11472, which focuses on inaccurate finished job metrics after worker failover. - #10808, which focuses on routing control-plane operations to coordinator-capable nodes in separated mode. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
