[
https://issues.apache.org/jira/browse/HBASE-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13744064#comment-13744064
]
Anoop Sam John commented on HBASE-5349:
---------------------------------------
Configurations
--------------
For on heap block cache there is a config value. ie. what is the max heap space
and defaults to 40%. Now we need to have a config item to read a min heap space
for block cache. In fact this is not the min heap space which at least will be
always reserved for the block cache. This is the min value the auto tuner can
turn down the block cache size up to.
hfile.block.cache.size -> Current config to specify the Percentage of maximum
heap (-Xmx setting) to allocate to block cache
Now we need 2 more configs to specify a range for this above value. And the
above config item will continue to be there.
hfile.block.cache.size.max.range
hfile.block.cache.size.min.range
By default when HBase starts, the block cache will be allocated a size of
hfile.block.cache.size (As current way). Later the auto tuner can change this
btw the max range and min range values.
If
hfile.block.cache.size.max.range=hfile.block.cache.size.min.range=hfile.block.cache.size
auto tuning for block cache will be turned off. (And thus for memstore also)
Memstore having a higher and lower water mark now. Higher water mark is a max
heap size now itself. But lower water mark is not. This is the heap size when
it reached, we will start flushes to prevent the memstore reaching the higher
water mark.
Once the memstore size reaches higher water mark, we will block all updates.
Now we need a min heap size also
The current configs are hbase.regionserver.global.memstore.upperLimit and
hbase.regionserver.global.memstore.lowerLimit
We need to introduce 2 new configs
hbase.regionserver.global.memstore.size.max.range and
hbase.regionserver.global.memstore.size.min.range Also we can rename
hbase.regionserver.global.memstore.upperLimit to
hbase.regionserver.global.memstore.size (?)
By default when HBase starts it will allocate a global memstore size =
hbase.regionserver.global.memstore.size as now (BC with old config name also
will be provided). The auto tuner can change this btw the max and min range.
And the lower watermark can no longer be a % of the total heap size. This needs
to be a % of the max heap size(Higher water mark ie.
hbase.regionserver.global.memstore.size). At any point in time the actual heap
space will be having some value btw max and min range. The lower water mark %
should result to a heap space lesser than the actual max heap space for global
memstore and we should start the flushes at this point of time (as we do today).
So we should rename this config name also. ->
hbase.regionserver.global.memstore.size.lower.limit -> % of the
hbase.regionserver.global.memstore.size at which we will start flushes to avoid
stop the world flushes. For BC we will continue to support
hbase.regionserver.global.memstore.lowerLimit which is a % of the total JVM
heap space. On init we can convert this value to a % of the
hbase.regionserver.global.memstore.size. The default value for this lower limit
can be 95% of the global memstore size. (Default values for
hbase.regionserver.global.memstore.upperLimit and
hbase.regionserver.global.memstore.lowerLimit are 40% and 38% respectively)
Init check now is block cache size + higher water mark <=80%
This should become
hbase.regionserver.global.memstore.size + hfile.block.cache.size <= 80%
AND
hbase.regionserver.global.memstore.size.max.range +
hfile.block.cache.size.min.range <= 80%
AND
hbase.regionserver.global.memstore.size.min.range +
hfile.block.cache.size.max.range <= 80%
At any time the heap size of the block cache + that of memstore <=80%
We can have a HeapMemoryManager
On HRS start, this also can be started. This Manager can track the
block cache evictions (Can get from CacheStats). Also the memstore flushes can
be tracked.
MemstoreFlusher should support changing the global memstore size. Also
the BlockCache can be resizable. This manager should support a pluggable
HeapMemoryBalancer using which the new value for memstore and block cache sizes
can be determined.
When there is a change in the heap size, those can be set on
MemstoreFlusher/BlockCache
HeapMemoryBalancer
A custom implementation for this can be plugged using config param.
"hbase.regionserver.memory.balancer.class"
The default impl can do the checks for the memory adjustment by
comparing the block cache evictions against the flushes for memstores due to
global heap pressure.
Normal flushes due to one memstore reaching the flush size is a normal
op and should not get accounted in the above checks. Only the flushes because
of global heap pressure (blocked or no blocked) should get accounted
Also a configurable time interval in which this tuner will check the
condition. The config name can be
"hbase.regionserver.heapmemory.autotuner.period" which default to 5 mns.
Depending on the check it can increase/decrease the size of block cache
or memstore global size in steps. The step value can be something like 2% of
–Xmx. This also can be configurable.
There should be way to support MemoryTuner to turn ON/OFF the tuner facility.
> Automagically tweak global memstore and block cache sizes based on workload
> ---------------------------------------------------------------------------
>
> Key: HBASE-5349
> URL: https://issues.apache.org/jira/browse/HBASE-5349
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: Anoop Sam John
>
> Hypertable does a neat thing where it changes the size given to the CellCache
> (our MemStores) and Block Cache based on the workload. If you need an image,
> scroll down at the bottom of this link:
> http://www.hypertable.com/documentation/architecture/
> That'd be one less thing to configure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
