singer-bin opened a new pull request, #8909: URL: https://github.com/apache/hadoop/pull/8909
### Description of PR JIRA: [HADOOP-20054](https://issues.apache.org/jira/browse/HADOOP-20054) HADOOP-17133 added `HttpServer2Metrics`, which exposes Jetty's `StatisticsHandler` (request counts, response codes, request/dispatch latency, etc.) as Hadoop Metrics2. However, it does not expose anything about the Jetty HTTP worker thread pool (`QueuedThreadPool`). When a daemon's web endpoint (e.g. the RM at `:8088`, or the NN/DN/NM web UIs) becomes slow or unresponsive, there is currently no metric telling whether the HTTP thread pool is exhausted or whether requests are piling up in the queue. Operators can only guess, or take a `jstack` and count `qtp*` threads manually after the fact. A common failure mode: web worker threads block (for example while acquiring a scheduler lock), the pool saturates at its max (250 by default), new requests queue up, clients time out and drop the connection -- surfacing as Jetty `EofException` on the server side -- while the daemon process itself is otherwise healthy. Without thread-pool metrics this is hard to confirm. This PR extends `HttpServer2Metrics` to also expose the `QueuedThreadPool` state: | Metric | Source | Meaning | | --- | --- | --- | | `HttpThreadsMax` | `getMaxThreads()` | configured max threads | | `HttpThreadsMin` | `getMinThreads()` | configured min threads | | `HttpThreads` | `getThreads()` | current live threads | | `HttpThreadsIdle` | `getIdleThreads()` | idle threads | | `HttpThreadsBusy` | `getBusyThreads()` | busy threads | | `HttpThreadsQueueSize` | `getQueueSize()` | jobs queued waiting for a thread | | `HttpThreadsLowOnThreads` | `isLowOnThreads()` | pool low-on-threads flag (0/1) | These are published under the existing `HttpServer2-<port>` metrics source and remain gated by the existing `hadoop.http.metrics.enabled` switch, so there is no behavior change when metrics are disabled. ### How was this patch tested? Added `TestHttpServer2Metrics` which verifies the thread-pool gauges are passed through from a live `QueuedThreadPool` and that the metrics source registers/unregisters correctly. ``` [INFO] Running org.apache.hadoop.http.TestHttpServer2Metrics [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0 [INFO] BUILD SUCCESS ``` ### For code changes: - [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-20054. Your PR title ...')? - [ ] Does this PR introduce any user-facing change? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
