[
https://issues.apache.org/jira/browse/HBASE-16146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15514741#comment-15514741
]
Gary Helmling commented on HBASE-16146:
---------------------------------------
We've seen Counter come up as a source of high CPU utilization in 1.3,
especially since the switch of metrics to use FastLongHistogram (each instance
of which uses 260 Counter instances internally) from HBASE-15222. I think this
is due to the use of the instance-level ThreadLocal in Counter to track the
per-thread cell index, as perf output on hot nodes shows a huge amount of time
in ThreadLocalMap.getEntryAfterMiss(). As the number of Counter instances (and
ThreadLocal instances) we're retaining in memory grows, performance seems to
degrade.
This is all moot for master, since we've already deprecated Counter and
replaced its usage with LongAdder. But we still need a solution for Counter in
branch-1. I'm testing a patch which removes the ThreadLocal usage, which I'll
attach here. Benchmarking shows a small reduction in Counter performance, but
a big improvement in FastLongHistogram performance when many histograms are
retained in memory.
> Counters are expensive...
> -------------------------
>
> Key: HBASE-16146
> URL: https://issues.apache.org/jira/browse/HBASE-16146
> Project: HBase
> Issue Type: Sub-task
> Reporter: stack
> Attachments: counters.patch, less_and_less_counters.png
>
>
> Doing workloadc, perf shows 10%+ of CPU being spent on counter#add. If I
> disable some of the hot ones -- see patch -- I can get 10% more throughput
> (390k to 440k). Figure something better.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
