[
https://issues.apache.org/jira/browse/HBASE-28293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18109693#comment-18109693
]
mazhengxuan commented on HBASE-28293:
-------------------------------------
I traced this on current master. The generic RPC server metrics already record
aggregate response size and processing time, but they do not provide per-method
attribution, so they cannot tell us whether {{GetClusterStatus}} is responsible
for the load.
I would like to take this issue. My proposed scope is to instrument only
{{{}MasterRpcServices#getClusterStatus{}}}, following the existing RegionServer
operation metrics pattern:
* add a {{getClusterStatusTime}} histogram; its {{_num_ops}} metric will
provide the total invocation count, including failed calls when updated in
{{{}finally{}}};
* add a {{getClusterStatusResponseSize}} histogram, updated with
{{GetClusterStatusResponse#getSerializedSize()}} for successfully built
responses.
I plan to expose these under the existing Master Server metrics source and add
assertions to {{{}TestMasterMetrics{}}}.
I would avoid adding a fixed 5MB threshold or a generic per-RPC metrics
framework in this Jira. Operators can alert on the response-size
max/percentiles, while {{hbase.ipc.warn.response.size}} already provides
configurable threshold-based logging. If a separately named request counter, or
an exact count of responses above a threshold, is preferred, I can include that
instead.
Please let me know if this scope sounds good. I am happy to work on it.
> Add metric for GetClusterStatus request count.
> ----------------------------------------------
>
> Key: HBASE-28293
> URL: https://issues.apache.org/jira/browse/HBASE-28293
> Project: HBase
> Issue Type: Bug
> Reporter: Rushabh Shah
> Priority: Major
>
> We have been bitten multiple times by GetClusterStatus request overwhelming
> HMaster's memory usage. It would be good to add a metric for the total
> GetClusterStatus requests count.
> In almost all of our production incidents involving GetClusterStatus request,
> HMaster were running out of memory with many clients call this RPC in
> parallel and the response size is very big.
> In hbase2 we have
> [ClusterMetrics.Option|https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/ClusterMetrics.java#L164-L224]
> which can reduce the size of the response.
> It would be nice to add another metric to indicate if the response size of
> GetClusterStatus is greater than some threshold (like 5MB)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
