[
https://issues.apache.org/jira/browse/HADOOP-20054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HADOOP-20054:
------------------------------------
Labels: pull-request-available (was: )
> Expose Jetty QueuedThreadPool metrics in HttpServer2Metrics
> -----------------------------------------------------------
>
> Key: HADOOP-20054
> URL: https://issues.apache.org/jira/browse/HADOOP-20054
> Project: Hadoop Common
> Issue Type: Improvement
> Components: metrics
> Affects Versions: 3.5.0
> Reporter: yanbin.zhang
> Priority: Major
> Labels: pull-request-available
>
> HADOOP-17133 added HttpServer2Metrics, which exposes Jetty's
> StatisticsHandler (request counts, response codes, request/dispatch
> latency, etc.) as Hadoop Metrics2.
> However, it does not expose anything about the Jetty HTTP worker thread
> pool (QueuedThreadPool). When a daemon's web endpoint (e.g. the RM at
> :8088, or NN/DN/NM web UIs) becomes slow or unresponsive, there is
> currently no metric to tell whether the HTTP thread pool is exhausted or
> whether requests are piling up in the queue. Operators can only guess, or
> take a jstack and count "qtp*" threads manually after the fact.
> A common failure mode is that web worker threads block (for example while
> acquiring a scheduler lock), the pool saturates at its max (250 by
> default), new requests queue up, clients time out and drop the connection
> -- surfacing as jetty EofException on the server side -- while the daemon
> process itself is otherwise healthy. Without thread-pool metrics this is
> hard to confirm.
> This improvement extends HttpServer2Metrics to also expose the
> QueuedThreadPool state:
> - HttpThreadsMax (getMaxThreads)
> - HttpThreadsMin (getMinThreads)
> - HttpThreads (getThreads)
> - HttpThreadsIdle (getIdleThreads)
> - HttpThreadsBusy (getBusyThreads)
> - HttpThreadsQueueSize (getQueueSize)
> - HttpThreadsLowOnThreads(isLowOnThreads, as 0/1)
> These are published under the existing "HttpServer2-<port>" metrics
> source and remain gated by the existing hadoop.http.metrics.enabled
> switch, so there is no behavior change when metrics are disabled.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
