[
https://issues.apache.org/jira/browse/HBASE-13876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhilash updated HBASE-13876:
-----------------------------
Description:
I am trying to improve the performance of DefaultHeapMemoryTuner by introducing
some more checks. The current checks under which the DefaultHeapMemoryTuner
works are very rare so I am trying to weaken these checks to improve its
performance.
Check current memstore size and current block cache size. For say if we are
using less than 50% of currently available block cache size we say block cache
is sufficient and same for memstore. This check will be very effective when
server is either load heavy or write heavy. Earlier version just waited for
number of evictions / number of flushes to be zero which are very rare.
Otherwise based on percent change in number of cache misses and number of
flushes we increase / decrease memory provided for caching / memstore. After
doing so, on next call of HeapMemoryTuner we verify that last change has indeed
decreased number of evictions / flush either of which it was expected to do. We
also check that it does not make the other (evictions / flush) increase much. I
am doing this analysis by comparing percent change (which is basically nothing
but normalized derivative) of number of evictions and number of flushes during
last two periods. The main motive for doing this was that if we have random
reads then we will be having a lot of cache misses. But even after increasing
block cache we wont be able to decrease number of cache misses and we will
revert back and eventually we will not waste memory on block caches. This will
also help us ignore random short term spikes in reads / writes. I have also
tried to take care not to tune memory if do do not have enough hints as
unnecessary tuning my slow down the system.
was:
I am trying to improve the performance of DefaultHeapMemoryTuner by introducing
some more checks. The current checks under which the DefaultHeapMemoryTuner
works are very rare so I am trying to weaken these checks to improve its
performance.
Check current memstore size and current block cache size. If we are using less
than 50% of currently available block cache size we say block cache is
sufficient and same for memstore. This check will be very effective when server
is either load heavy or write heavy. Earlier version just waited for number of
evictions / number of flushes to be zero which are very rare.
Otherwise based on percent change in number of cache misses and number of
flushes we increase / decrease memory provided for caching / memstore. After
doing so, on next call of HeapMemoryTuner we verify that last change has indeed
decreased number of evictions / flush ( combined). I am doing this analysis by
comparing percent change (which is basically nothing but normalized derivative)
of number of evictions and number of flushes during last two periods. The main
motive for doing this was that if we have random reads then we will be having a
lot of cache misses. But even after increasing block cache we wont be able to
decrease number of cache misses and we will revert back and eventually we will
not waste memory on block caches. This will also help us ignore random short
term spikes in reads / writes.
> Improving performance of HeapMemoryManager
> ------------------------------------------
>
> Key: HBASE-13876
> URL: https://issues.apache.org/jira/browse/HBASE-13876
> Project: HBase
> Issue Type: Improvement
> Components: hbase, regionserver
> Affects Versions: 2.0.0, 1.0.1, 1.1.0, 1.1.1
> Reporter: Abhilash
> Assignee: Abhilash
> Priority: Minor
> Attachments: HBASE-13876-v2.patch, HBASE-13876-v3.patch,
> HBASE-13876-v4.patch, HBASE-13876-v5.patch, HBASE-13876-v5.patch,
> HBASE-13876.patch
>
>
> I am trying to improve the performance of DefaultHeapMemoryTuner by
> introducing some more checks. The current checks under which the
> DefaultHeapMemoryTuner works are very rare so I am trying to weaken these
> checks to improve its performance.
> Check current memstore size and current block cache size. For say if we are
> using less than 50% of currently available block cache size we say block
> cache is sufficient and same for memstore. This check will be very effective
> when server is either load heavy or write heavy. Earlier version just waited
> for number of evictions / number of flushes to be zero which are very rare.
> Otherwise based on percent change in number of cache misses and number of
> flushes we increase / decrease memory provided for caching / memstore. After
> doing so, on next call of HeapMemoryTuner we verify that last change has
> indeed decreased number of evictions / flush either of which it was expected
> to do. We also check that it does not make the other (evictions / flush)
> increase much. I am doing this analysis by comparing percent change (which is
> basically nothing but normalized derivative) of number of evictions and
> number of flushes during last two periods. The main motive for doing this was
> that if we have random reads then we will be having a lot of cache misses.
> But even after increasing block cache we wont be able to decrease number of
> cache misses and we will revert back and eventually we will not waste memory
> on block caches. This will also help us ignore random short term spikes in
> reads / writes. I have also tried to take care not to tune memory if do do
> not have enough hints as unnecessary tuning my slow down the system.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
