ramkrishna.s.vasudevan commented on HBASE-15787:

bq.Why am I doing ABOVE_OFFHEAP_HIGHER_MARK in the HeapMemoryManager (unless 
HeapMemoryManager is for both onheap and offheap?)
Am removing this for now.  Except that am seeing if this heap memory tuner is 
called for offheap memstore. If so I do some step size calculations.
bq.see if it settles or if it does a Tacoma Narrows Bridge behavior.
I first read what is this behaviour :). Interesting. And true that we should be 
careful that we don't end up there. But for that I think we need to run various 
work loads. So will that be good if we commit this and as part of testing we 
see what test cases we need to consider before making this auto tuner default 
for 2.0?
bq.RegionServerAccounting is memory only? Should it say that it is memory only? 
I think the purpose was not only memory.
 * RegionServerAccounting keeps record of some basic real time information about
 * the Region Server. 
The javadoc says basic real time info which includes memory. So am not sure if 
it is right to rename to MemoryAccounting. If you strongly feel about the 
rename I can do it.

> Change the flush related heuristics to work with offheap size configured
> ------------------------------------------------------------------------
>                 Key: HBASE-15787
>                 URL: https://issues.apache.org/jira/browse/HBASE-15787
>             Project: HBase
>          Issue Type: Sub-task
>          Components: regionserver
>            Reporter: ramkrishna.s.vasudevan
>            Assignee: ramkrishna.s.vasudevan
>             Fix For: 2.0.0
>         Attachments: HBASE-15787.patch, HBASE-15787_1.patch
> With offheap MSLAB in place we may have to change the flush related 
> heuristics to work with offheap size configured rather than the java heap 
> size.
> Since we now have clear seperation of the memstore data size and memstore 
> heap size, for offheap memstore
> -> Decide if the global.offheap.memstore.size is breached for blocking 
> updates and force flushes. 
> -> If the onheap global.memstore.size is breached (due to heap overhead) even 
> then block updates and force flushes.
> -> The global.memstore.size.lower.limit is now by default 95% of the 
> global.memstore.size. So now we apply this 95% on the 
> global.offheap.memstore.size and also on global.memstore.size (as it was done 
> for onheap case).
> -> We will have new FlushTypes introduced
> {code}
>   ABOVE_ONHEAP_LOWER_MARK, /* happens due to lower mark breach of onheap 
> memstore settings
>                               An offheap memstore can even breach the 
> onheap_lower_mark*/
>   ABOVE_ONHEAP_HIGHER_MARK,/* happens due to higher mark breach of onheap 
> memstore settings
>                               An offheap memstore can even breach the 
> onheap_higher_mark*/
>   ABOVE_OFFHEAP_LOWER_MARK,/* happens due to lower mark breach of offheap 
> memstore settings*/
> {code}
> -> regionServerAccounting does all the accounting.
> -> HeapMemoryTuner is what is litte tricky here. First thing to note is that 
> at no point it will try to increase or decrease the 
> global.offheap.memstore.size. If there is a heap pressure then it will try to 
> increase the memstore heap limit. 
> In case of offheap memstore there is always a chance that the heap pressure 
> does not increase. In that case we could ideally decrease the heap limit for 
> memstore. The current logic of heapmemory tuner is such that things will 
> naturally settle down. But on discussion what we thought is let us include 
> the flush count that happens due to offheap pressure but give that a lesser 
> weightage and thus ensure that the initial decrease on memstore heap limit 
> does not happen. Currently that fraction is set as 0.5. 

This message was sent by Atlassian JIRA

Reply via email to