Gary Helmling commented on HBASE-16146:

We've seen Counter come up as a source of high CPU utilization in 1.3, 
especially since the switch of metrics to use FastLongHistogram (each instance 
of which uses 260 Counter instances internally) from HBASE-15222.  I think this 
is due to the use of the instance-level ThreadLocal in Counter to track the 
per-thread cell index, as perf output on hot nodes shows a huge amount of time 
in ThreadLocalMap.getEntryAfterMiss().  As the number of Counter instances (and 
ThreadLocal instances) we're retaining in memory grows, performance seems to 

This is all moot for master, since we've already deprecated Counter and 
replaced its usage with LongAdder.  But we still need a solution for Counter in 
branch-1.  I'm testing a patch which removes the ThreadLocal usage, which I'll 
attach here.  Benchmarking shows a small reduction in Counter performance, but 
a big improvement in FastLongHistogram performance when many histograms are 
retained in memory.

> Counters are expensive...
> -------------------------
>                 Key: HBASE-16146
>                 URL: https://issues.apache.org/jira/browse/HBASE-16146
>             Project: HBase
>          Issue Type: Sub-task
>            Reporter: stack
>         Attachments: counters.patch, less_and_less_counters.png
> Doing workloadc, perf shows 10%+ of CPU being spent on counter#add. If I 
> disable some of the hot ones -- see patch -- I can get 10% more throughput 
> (390k to 440k). Figure something better.

This message was sent by Atlassian JIRA

Reply via email to