DanielLeens opened a new issue, #11846: URL: https://github.com/apache/seatunnel/issues/11846
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue) and found no similar issues. ### What happened The Zeta REST `/metrics` endpoint can answer `500 Internal Server Error` to a scraper that hits a member in the window between the member reaching STARTED (Jetty listener already accepting) and `CoordinatorService` finishing its wiring. Mechanism, from source on current `dev`: - `SeaTunnelServer.init()` starts Jetty synchronously, while `CoordinatorService` becomes active asynchronously. - `MetricsServlet.doGet()` iterates `collectorRegistry.metricFamilySamples()` in a single pass with no per-collector isolation. - Collectors such as `JobMetricExports.collect()` guard with check-then-act (`isMaster() && isCoordinatorReady()`) and then dereference coordinator-owned state; `SeaTunnelServer.getCoordinatorService()` throws `SeaTunnelEngineRetryableException` while the coordinator is still wiring up, and readiness can also flip between the check and the access. - `ExceptionHandlingFilter.doFilter()` converts any escaping collector exception into a 500 whose JSON body embeds the full Java stack trace. So one not-yet-ready collector aborts the whole scrape, and the scraper receives a 500 with an internal stack trace. This was discovered through `MetricsApiTest#metricsApiTest` flaking in CI with `Expected status code <200> but was <500>` on at least five unrelated PRs (#10874, #10958, #10973, #11060, #11382). PR #11641 stabilizes that test by polling until the endpoint is serviceable — which is correct for the test, but it also means CI no longer surfaces this window. This issue exists so the production-side gap is tracked rather than silently absorbed by the test change (raised by @SEZ9 in the #11641 reviews). ### What you expected to happen A freshly started (or re-electing) member should not answer `/metrics` with a 500 + stack trace during normal startup. Options worth evaluating: 1. Have each coordinator-dependent collector return an empty/partial metric family while its backing service is unavailable (the pattern `EngineStateStoreLogicalMetricExports` already follows), instead of throwing. 2. Isolate per-collector failures in `MetricsServlet` so one throwing collector degrades that collector's families instead of aborting the entire scrape. 3. Answer `503 Service Unavailable` (optionally with `Retry-After`) until `CoordinatorService` is ready — Prometheus records that as an explicit scrape failure with a clear cause rather than a server error. Any of these also stops the endpoint from leaking internal stack traces to whatever can reach the scrape port. ### How to reproduce Start a Zeta member with telemetry metrics and the HTTP listener enabled, and issue `GET /metrics` immediately after `createHazelcastInstance(...)` returns — before the coordinator finishes wiring. Intermittently returns 500 with a stack-trace body. The pre-#11641 version of `MetricsApiTest#metricsApiTest` is a ready-made reproducer (single un-retried GET right after startup). ### SeaTunnel Version dev ### SeaTunnel Config ```conf Not job-config related; engine REST/telemetry startup ordering. ``` ### Running Command ```shell See "How to reproduce". ``` ### Error Exception ```log Expected status code <200> but was <500>. (500 body carries the collector's stack trace via ExceptionHandlingFilter) ``` ### Zeta or Flink or Spark Version Zeta ### Java or Scala Version _No response_ ### Screenshots _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
