[
https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-13142:
--------------------------
Release Note:
Adds buffer reuse sending Cell results. It is on by default and should not need
configuration. Improves GC profile and ups throughput. The benefit gets better
the larger the row size returned.
The buffer reservoir is bounded at a maximum count after which we will start
logging at WARN level that the reservoir is running at capacity (returned
buffers will be discarded and not added back to the reservoir pool). Default
maximum is twice the handler count: i.e. 2 * hbase.regionserver.handler.count.
This should be more than enough. Set the maximum with the new configuration:
hbase.ipc.server.reservoir.max
The reservoir will not cache buffers in excess of
hbase.ipc.server.reservoir.max.buffer.size The default is 10MB. This means
that if a row is very large, then we will allocate a buffer of the average size
that is currently in the pool and we will then resize it till we can
accommodate the return. These resizes are expensive. The resultant buffer will
be used and then discarded.
To check how the reservoir is doing, enable trace level logging for a few
seconds on a regionserver. You can do this from the regionserver UI. See 'Log
Level'. Set org.apache.hadoop.hbase.io.BoundedByteBufferPool to TRACE. The
BoundedByteBufferPool will spew report to the log. Disable the TRACE level and
then check the log. You'll see allocation rate, size of pool, size of buffers
in pool, etc.
> [PERF] Reuse the IPCUtil#buildCellBlock buffer
> ----------------------------------------------
>
> Key: HBASE-13142
> URL: https://issues.apache.org/jira/browse/HBASE-13142
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Reporter: stack
> Assignee: stack
> Labels: beginner
> Fix For: 2.0.0, 1.1.0
>
> Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, 13142v5.txt,
> traces.2.svg, traces.svg
>
>
> Running some scan profiling, flight recorder was mildly fingering resize of
> the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It
> was half-hearted blaming it for a few hundreds of ms over a five minute
> sampling with a few tens of instances showing.
> I tried then w/ flamegraph/lightweight profiler and this reported the buffer
> allocations taking 22% of our total CPU. See attachment trace.svg.
> I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and
> indeed every allocation was doing a resize from initial allocation of 16k --
> the default up to 220k (this test returns ten randomly sized rows zipfian
> sized between 0 and 8k).
> Upping the allocation to 220k meant we now avoided the resize but the initial
> allocation was now blamed for 10% of allocations (see trace.2.svg attached).
> Lets do buffer reuse. Will save a bunch of allocation and CPU.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
