[
https://issues.apache.org/jira/browse/HBASE-16577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15475115#comment-15475115
]
Andrew Purtell commented on HBASE-16577:
----------------------------------------
bq. So IMO we'd better just keep the real slow request (Get, mutation) as
'WARN' Level, ignore the uncertain ones (If we can't distinguish them, just
remove them from 'WARN' level).
The answer to the question 'What is a "real" slow request?' is "It depends",
though.
bq. Maybe have different limits to scan vs get vs multi?
This is a generalization of the above suggestion. So we would have new
configuration that sets a threshold for every op type, and one as a catch-all
for op types that don't have a specific threshold set? Only warn if above the
threshold? Default for < branch-2 is 1, a super small threshold leading to
warns on anything like today. Default for >= branch-2 is 0, meaning no warn.
> On reducing the log level of *TooSlow log lines
> -----------------------------------------------
>
> Key: HBASE-16577
> URL: https://issues.apache.org/jira/browse/HBASE-16577
> Project: HBase
> Issue Type: Brainstorming
> Reporter: Andrew Purtell
>
> I was looking at the nature and distribution of responseTooSlow messages on
> our clusters. The majority of responseTooSlow warnings are for Scan or Multi.
> Scans may take a long time to return. It will totally depend on how much data
> is in the table, how the data is distributed, and the range and selectivity
> of the query. We are not measuring response time in a way to know what is
> proportionate to the work requested. Another problematic example is Multi. We
> don't get valid results considering a multi with 1 op and a multi with 100
> ops to be equivalent as far as being "too slow".
> If we aggressively filter responseTooSlow messages to just include the ops we
> can expect to be small, constant, request-independent units of work, this
> leaves us with Get and Mutate. This gives us no more information then we get
> from the Canary with read and write checks turned on. The Canary issues Get
> and Mutate ops and, better, measures availability and latency from the client
> perspective.
>
> Where I end up is I think my shop should ignore responseTooSlow as signal as
> being far too noisy. The trouble then is it is logged at WARN level. This
> implies there is something wrong that needs to be fixed. That may not be the
> case. It's going to require some analysis of the application and the request
> particulars extracted from the log line. WARN seems inappropriate for this
> type of indication. There's nothing (necessarily) wrong with HBase, or the
> app. Should be logged at more like INFO.
> Furthermore the response might not be too slow, so calling it
> "responseTooSlow" isn't quite right. More like "responseMaybeSlow" :-)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
