virajjasani commented on a change in pull request #754: HBASE-22978 : Online
slow response log
URL: https://github.com/apache/hbase/pull/754#discussion_r369326567
##########
File path: src/main/asciidoc/_chapters/ops_mgt.adoc
##########
@@ -1914,6 +1914,85 @@ In the case that the call is not a client operation,
that detailed fingerprint i
This particular example, for example, would indicate that the likely cause of
slowness is simply a very large (on the order of 100MB) multiput, as we can
tell by the "vlen," or value length, fields of each put in the multiPut.
+==== Get Slow Response Log from shell
+When an individual RPC exceeds a configurable time bound we log a complaint
+by way of the logging subsystem
+
+e.g.
+
+----
+2019-10-02 10:10:22,195 WARN [,queue=15,port=60020] ipc.RpcServer -
(responseTooSlow):
+{"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)",
+"starttimems":1567203007549,
+"responsesize":6819737,
+"method":"Scan",
+"param":"region { type: REGION_NAME value:
\"t1,\\000\\000\\215\\f)o\\\\\\024\\302\\220\\000\\000\\000\\000\\000\\001\\000\\000\\000\\000\\000\\006\\000\\000\\000\\000\\000\\005\\000\\000<TRUNCATED>",
+"processingtimems":28646,
+"client":"10.253.196.215:41116",
+"queuetimems":22453,
+"class":"HRegionServer"}
+----
+
+Unfortunately often the request parameters are truncated as per above Example.
+The truncation is unfortunate because it eliminates much of the utility of
+the warnings. For example, the region name, the start and end keys, and the
+filter hierarchy are all important clues for debugging performance problems
+caused by moderate to low selectivity queries or queries made at a high rate.
+
+HBASE-22978 introduces maintaining an in-memory ring buffer of requests that
were judged to
+be too slow in addition to the responseTooSlow logging. The in-memory
representation can be
+complete. There is some chance a high rate of requests will cause information
on other
+interesting requests to be overwritten before it can be read. This is an
acceptable trade off.
+
+shell commands to retrieve slowlog responses from RegionServers:
+
+----
+Retrieve latest SlowLog Responses maintained by each or specific RegionServers.
+Specify 'all_rs' to include all RS otherwise array of 'server_name' for
specific
+RS. 'server_name' is the host, port plus startcode of a RegionServer.
+e.g.: host187.example.com,60020,1289493121758 (find servername in
+master ui or when you do detailed status in shell)
+
+Examples:
+
+ hbase> get_slowlog_responses 'all_rs' => get
slowlog responses from all RS
+ hbase> get_slowlog_responses ['server_name1', 'server_name2'] => get
slowlog responses from server_name1,
+
server_name2
+ hbase> get_slowlog_responses 'all_rs', {'REGION_NAME' => 'hbase:meta,,1'}
+ => get
slowlog responses only related to meta
+ region
+ hbase> get_slowlog_responses 'all_rs', {'TABLE_NAME' => 't1'} => get
slowlog responses only related to t1 table
+ hbase> get_slowlog_responses 'all_rs', {'CLIENT_IP' => '192.162.1.40:60225'}
+ => get
slowlog responses with given client
+ IP
address
+ hbase> get_slowlog_responses 'all_rs', {'REGION_NAME' => 'hbase:meta,,1',
'TABLE_NAME' => 't1'}
+ => get
slowlog responses with given region name
+ or table
name
+ hbase> get_slowlog_responses 'all_rs', {'USER' => 'user_name', 'CLIENT_IP'
=> '192.162.1.40:60225'}
+ => get
slowlog responses that match either
+ provided
client IP address or user name
+
+
+----
+
+shell command to clear slowlog responses from RegionServer:
+
+----
+Clears SlowLog Responses maintained by each or specific RegionServers.
+Specify array of 'server_name' for specific RS. 'server_name' is
+the host, port plus startcode of a RegionServer.
+e.g.: host187.example.com,60020,1289493121758 (find servername in
+master ui or when you do detailed status in shell)
+
+Examples:
+
+ hbase> clear_slowlog_responses => clears
slowlog responses from all RS
+ hbase> clear_slowlog_responses ['server_name1', 'server_name2'] => clears
slowlog responses from server_name1,
+
server_name2
+
+
+----
+
Review comment:
Sure let me include `*` then
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
