[
https://issues.apache.org/jira/browse/HBASE-21991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16783976#comment-16783976
]
Sakthi edited comment on HBASE-21991 at 3/5/19 2:25 AM:
--------------------------------------------------------
Regarding the faulty remove logic:
* According to the lossy counting algorithm, non-eligible meters are swept off
from on every (1/e)th access [e = error rate. Default = 0.02]. Hence under
default settings, in a stream of accesses, at every 50th access, the
non-eligible meters are pruned off.
* But with current implementation(with a bug), if every 50th (i.e. every
(1/e)th) access is of a already existing clientRequestMeter then non-eligible
meters might never be pruned off and we might end up storing/exposing all the
meters rather than top-k-ish
{code:java}
private void registerLossyCountingMeterIfNotPresent(...) {
...
Set<String> metersToBeRemoved = lossyCounting.addByOne(requestMeter);
if(!requestsMap.containsKey(requestMeter) &&
metersToBeRemoved.contains(requestMeter)){
// <----Here, a remove happens only if the current metric doesn't already exist
in the requestsMap.
for(String meter: metersToBeRemoved)
{
requestsMap.remove(meter);
...
}
...{code}
Have verified with a unit test.
was (Author: jatsakthi):
Regarding the faulty remove logic:
* According to the lossy counting algorithm, non-eligible meters are swept off
from on every (1/e)th access [e = error rate. Default = 0.02]. Hence under
default settings, in a stream of accesses, at every 50th access, the
non-eligible meters are pruned off.
* But with current implementation(with a bug), if every 50th (i.e. every
(1/e)th) access is of a already existing clientRequestMeter then non-eligible
meters might never be pruned off and we might end up storing/exposing all the
meters rather than top-k-ish
Have verified with a unit test.
> Fix MetaMetrics issues - [Race condition, Faulty remove logic], few
> improvements
> --------------------------------------------------------------------------------
>
> Key: HBASE-21991
> URL: https://issues.apache.org/jira/browse/HBASE-21991
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors, metrics
> Reporter: Sakthi
> Assignee: Sakthi
> Priority: Major
>
> Here is a list of the issues related to the MetaMetrics implementation:
> +*Bugs*+:
> # [_Lossy counting for top-k_] *Faulty remove logic of non-eligible meters*:
> Under certain conditions, we might end up storing/exposing all the meters
> rather than top-k-ish
> # MetaMetrics can throw NPE resulting in aborting of the RS because of a
> *Race Condition*.
> +*Improvements*+:
> # With high number of regions in the cluster, exposure of metrics for each
> region blows up the JMX from ~140 Kbs to 100+ Mbs depending on the number of
> regions. It's better to use *lossy counting to maintain top-k for region
> metrics* as well.
> # As the lossy meters do not represent actual counts, I think, it'll be
> better to *rename the meters to include "lossy" in the name*. It would be
> more informative while monitoring the metrics and there would be less
> confusion regarding actual counts to lossy counts.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
