[jira] [Updated] (HBASE-15787) Change the flush related heuristics to work with offheap size configured
[ https://issues.apache.org/jira/browse/HBASE-15787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15787: --- Resolution: Fixed Status: Resolved (was: Patch Available) Thanks for the reviews [~saint@gmail.com] and [~anoop.hbase]. Pushed to master. > Change the flush related heuristics to work with offheap size configured > > > Key: HBASE-15787 > URL: https://issues.apache.org/jira/browse/HBASE-15787 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-15787.patch, HBASE-15787_1.patch, > HBASE-15787_4.patch, HBASE-15787_5.patch, HBASE-15787_6.patch, > HBASE-15787_7.patch, HBASE-15787_8.patch, HBASE-15787_9.patch > > > With offheap MSLAB in place we may have to change the flush related > heuristics to work with offheap size configured rather than the java heap > size. > Since we now have clear seperation of the memstore data size and memstore > heap size, for offheap memstore > -> Decide if the global.offheap.memstore.size is breached for blocking > updates and force flushes. > -> If the onheap global.memstore.size is breached (due to heap overhead) even > then block updates and force flushes. > -> The global.memstore.size.lower.limit is now by default 95% of the > global.memstore.size. So now we apply this 95% on the > global.offheap.memstore.size and also on global.memstore.size (as it was done > for onheap case). > -> We will have new FlushTypes introduced > {code} > ABOVE_ONHEAP_LOWER_MARK, /* happens due to lower mark breach of onheap > memstore settings > An offheap memstore can even breach the > onheap_lower_mark*/ > ABOVE_ONHEAP_HIGHER_MARK,/* happens due to higher mark breach of onheap > memstore settings > An offheap memstore can even breach the > onheap_higher_mark*/ > ABOVE_OFFHEAP_LOWER_MARK,/* happens due to lower mark breach of offheap > memstore settings*/ > ABOVE_OFFHEAP_HIGHER_MARK; > {code} > -> regionServerAccounting does all the accounting. > -> HeapMemoryTuner is what is litte tricky here. First thing to note is that > at no point it will try to increase or decrease the > global.offheap.memstore.size. If there is a heap pressure then it will try to > increase the memstore heap limit. > In case of offheap memstore there is always a chance that the heap pressure > does not increase. In that case we could ideally decrease the heap limit for > memstore. The current logic of heapmemory tuner is such that things will > naturally settle down. But on discussion what we thought is let us include > the flush count that happens due to offheap pressure but give that a lesser > weightage and thus ensure that the initial decrease on memstore heap limit > does not happen. Currently that fraction is set as 0.5. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15787) Change the flush related heuristics to work with offheap size configured
[ https://issues.apache.org/jira/browse/HBASE-15787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15787: --- Status: Patch Available (was: Open) > Change the flush related heuristics to work with offheap size configured > > > Key: HBASE-15787 > URL: https://issues.apache.org/jira/browse/HBASE-15787 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-15787.patch, HBASE-15787_1.patch, > HBASE-15787_4.patch, HBASE-15787_5.patch, HBASE-15787_6.patch, > HBASE-15787_7.patch, HBASE-15787_8.patch, HBASE-15787_9.patch > > > With offheap MSLAB in place we may have to change the flush related > heuristics to work with offheap size configured rather than the java heap > size. > Since we now have clear seperation of the memstore data size and memstore > heap size, for offheap memstore > -> Decide if the global.offheap.memstore.size is breached for blocking > updates and force flushes. > -> If the onheap global.memstore.size is breached (due to heap overhead) even > then block updates and force flushes. > -> The global.memstore.size.lower.limit is now by default 95% of the > global.memstore.size. So now we apply this 95% on the > global.offheap.memstore.size and also on global.memstore.size (as it was done > for onheap case). > -> We will have new FlushTypes introduced > {code} > ABOVE_ONHEAP_LOWER_MARK, /* happens due to lower mark breach of onheap > memstore settings > An offheap memstore can even breach the > onheap_lower_mark*/ > ABOVE_ONHEAP_HIGHER_MARK,/* happens due to higher mark breach of onheap > memstore settings > An offheap memstore can even breach the > onheap_higher_mark*/ > ABOVE_OFFHEAP_LOWER_MARK,/* happens due to lower mark breach of offheap > memstore settings*/ > ABOVE_OFFHEAP_HIGHER_MARK; > {code} > -> regionServerAccounting does all the accounting. > -> HeapMemoryTuner is what is litte tricky here. First thing to note is that > at no point it will try to increase or decrease the > global.offheap.memstore.size. If there is a heap pressure then it will try to > increase the memstore heap limit. > In case of offheap memstore there is always a chance that the heap pressure > does not increase. In that case we could ideally decrease the heap limit for > memstore. The current logic of heapmemory tuner is such that things will > naturally settle down. But on discussion what we thought is let us include > the flush count that happens due to offheap pressure but give that a lesser > weightage and thus ensure that the initial decrease on memstore heap limit > does not happen. Currently that fraction is set as 0.5. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15787) Change the flush related heuristics to work with offheap size configured
[ https://issues.apache.org/jira/browse/HBASE-15787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15787: --- Attachment: HBASE-15787_9.patch This is what I will be committing. The getFlushPressure() is added in RegionserverAccounting. We could do a clean up of the API in RegionServerServices as a separate task. > Change the flush related heuristics to work with offheap size configured > > > Key: HBASE-15787 > URL: https://issues.apache.org/jira/browse/HBASE-15787 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-15787.patch, HBASE-15787_1.patch, > HBASE-15787_4.patch, HBASE-15787_5.patch, HBASE-15787_6.patch, > HBASE-15787_7.patch, HBASE-15787_8.patch, HBASE-15787_9.patch > > > With offheap MSLAB in place we may have to change the flush related > heuristics to work with offheap size configured rather than the java heap > size. > Since we now have clear seperation of the memstore data size and memstore > heap size, for offheap memstore > -> Decide if the global.offheap.memstore.size is breached for blocking > updates and force flushes. > -> If the onheap global.memstore.size is breached (due to heap overhead) even > then block updates and force flushes. > -> The global.memstore.size.lower.limit is now by default 95% of the > global.memstore.size. So now we apply this 95% on the > global.offheap.memstore.size and also on global.memstore.size (as it was done > for onheap case). > -> We will have new FlushTypes introduced > {code} > ABOVE_ONHEAP_LOWER_MARK, /* happens due to lower mark breach of onheap > memstore settings > An offheap memstore can even breach the > onheap_lower_mark*/ > ABOVE_ONHEAP_HIGHER_MARK,/* happens due to higher mark breach of onheap > memstore settings > An offheap memstore can even breach the > onheap_higher_mark*/ > ABOVE_OFFHEAP_LOWER_MARK,/* happens due to lower mark breach of offheap > memstore settings*/ > ABOVE_OFFHEAP_HIGHER_MARK; > {code} > -> regionServerAccounting does all the accounting. > -> HeapMemoryTuner is what is litte tricky here. First thing to note is that > at no point it will try to increase or decrease the > global.offheap.memstore.size. If there is a heap pressure then it will try to > increase the memstore heap limit. > In case of offheap memstore there is always a chance that the heap pressure > does not increase. In that case we could ideally decrease the heap limit for > memstore. The current logic of heapmemory tuner is such that things will > naturally settle down. But on discussion what we thought is let us include > the flush count that happens due to offheap pressure but give that a lesser > weightage and thus ensure that the initial decrease on memstore heap limit > does not happen. Currently that fraction is set as 0.5. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15787) Change the flush related heuristics to work with offheap size configured
[ https://issues.apache.org/jira/browse/HBASE-15787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15787: --- Status: Open (was: Patch Available) > Change the flush related heuristics to work with offheap size configured > > > Key: HBASE-15787 > URL: https://issues.apache.org/jira/browse/HBASE-15787 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-15787.patch, HBASE-15787_1.patch, > HBASE-15787_4.patch, HBASE-15787_5.patch, HBASE-15787_6.patch, > HBASE-15787_7.patch, HBASE-15787_8.patch > > > With offheap MSLAB in place we may have to change the flush related > heuristics to work with offheap size configured rather than the java heap > size. > Since we now have clear seperation of the memstore data size and memstore > heap size, for offheap memstore > -> Decide if the global.offheap.memstore.size is breached for blocking > updates and force flushes. > -> If the onheap global.memstore.size is breached (due to heap overhead) even > then block updates and force flushes. > -> The global.memstore.size.lower.limit is now by default 95% of the > global.memstore.size. So now we apply this 95% on the > global.offheap.memstore.size and also on global.memstore.size (as it was done > for onheap case). > -> We will have new FlushTypes introduced > {code} > ABOVE_ONHEAP_LOWER_MARK, /* happens due to lower mark breach of onheap > memstore settings > An offheap memstore can even breach the > onheap_lower_mark*/ > ABOVE_ONHEAP_HIGHER_MARK,/* happens due to higher mark breach of onheap > memstore settings > An offheap memstore can even breach the > onheap_higher_mark*/ > ABOVE_OFFHEAP_LOWER_MARK,/* happens due to lower mark breach of offheap > memstore settings*/ > ABOVE_OFFHEAP_HIGHER_MARK; > {code} > -> regionServerAccounting does all the accounting. > -> HeapMemoryTuner is what is litte tricky here. First thing to note is that > at no point it will try to increase or decrease the > global.offheap.memstore.size. If there is a heap pressure then it will try to > increase the memstore heap limit. > In case of offheap memstore there is always a chance that the heap pressure > does not increase. In that case we could ideally decrease the heap limit for > memstore. The current logic of heapmemory tuner is such that things will > naturally settle down. But on discussion what we thought is let us include > the flush count that happens due to offheap pressure but give that a lesser > weightage and thus ensure that the initial decrease on memstore heap limit > does not happen. Currently that fraction is set as 0.5. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15787) Change the flush related heuristics to work with offheap size configured
[ https://issues.apache.org/jira/browse/HBASE-15787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15787: --- Status: Patch Available (was: Open) > Change the flush related heuristics to work with offheap size configured > > > Key: HBASE-15787 > URL: https://issues.apache.org/jira/browse/HBASE-15787 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-15787.patch, HBASE-15787_1.patch, > HBASE-15787_4.patch, HBASE-15787_5.patch, HBASE-15787_6.patch, > HBASE-15787_7.patch, HBASE-15787_8.patch > > > With offheap MSLAB in place we may have to change the flush related > heuristics to work with offheap size configured rather than the java heap > size. > Since we now have clear seperation of the memstore data size and memstore > heap size, for offheap memstore > -> Decide if the global.offheap.memstore.size is breached for blocking > updates and force flushes. > -> If the onheap global.memstore.size is breached (due to heap overhead) even > then block updates and force flushes. > -> The global.memstore.size.lower.limit is now by default 95% of the > global.memstore.size. So now we apply this 95% on the > global.offheap.memstore.size and also on global.memstore.size (as it was done > for onheap case). > -> We will have new FlushTypes introduced > {code} > ABOVE_ONHEAP_LOWER_MARK, /* happens due to lower mark breach of onheap > memstore settings > An offheap memstore can even breach the > onheap_lower_mark*/ > ABOVE_ONHEAP_HIGHER_MARK,/* happens due to higher mark breach of onheap > memstore settings > An offheap memstore can even breach the > onheap_higher_mark*/ > ABOVE_OFFHEAP_LOWER_MARK,/* happens due to lower mark breach of offheap > memstore settings*/ > ABOVE_OFFHEAP_HIGHER_MARK; > {code} > -> regionServerAccounting does all the accounting. > -> HeapMemoryTuner is what is litte tricky here. First thing to note is that > at no point it will try to increase or decrease the > global.offheap.memstore.size. If there is a heap pressure then it will try to > increase the memstore heap limit. > In case of offheap memstore there is always a chance that the heap pressure > does not increase. In that case we could ideally decrease the heap limit for > memstore. The current logic of heapmemory tuner is such that things will > naturally settle down. But on discussion what we thought is let us include > the flush count that happens due to offheap pressure but give that a lesser > weightage and thus ensure that the initial decrease on memstore heap limit > does not happen. Currently that fraction is set as 0.5. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15787) Change the flush related heuristics to work with offheap size configured
[ https://issues.apache.org/jira/browse/HBASE-15787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15787: --- Attachment: HBASE-15787_8.patch Attaching the right patch. > Change the flush related heuristics to work with offheap size configured > > > Key: HBASE-15787 > URL: https://issues.apache.org/jira/browse/HBASE-15787 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-15787.patch, HBASE-15787_1.patch, > HBASE-15787_4.patch, HBASE-15787_5.patch, HBASE-15787_6.patch, > HBASE-15787_7.patch, HBASE-15787_8.patch > > > With offheap MSLAB in place we may have to change the flush related > heuristics to work with offheap size configured rather than the java heap > size. > Since we now have clear seperation of the memstore data size and memstore > heap size, for offheap memstore > -> Decide if the global.offheap.memstore.size is breached for blocking > updates and force flushes. > -> If the onheap global.memstore.size is breached (due to heap overhead) even > then block updates and force flushes. > -> The global.memstore.size.lower.limit is now by default 95% of the > global.memstore.size. So now we apply this 95% on the > global.offheap.memstore.size and also on global.memstore.size (as it was done > for onheap case). > -> We will have new FlushTypes introduced > {code} > ABOVE_ONHEAP_LOWER_MARK, /* happens due to lower mark breach of onheap > memstore settings > An offheap memstore can even breach the > onheap_lower_mark*/ > ABOVE_ONHEAP_HIGHER_MARK,/* happens due to higher mark breach of onheap > memstore settings > An offheap memstore can even breach the > onheap_higher_mark*/ > ABOVE_OFFHEAP_LOWER_MARK,/* happens due to lower mark breach of offheap > memstore settings*/ > ABOVE_OFFHEAP_HIGHER_MARK; > {code} > -> regionServerAccounting does all the accounting. > -> HeapMemoryTuner is what is litte tricky here. First thing to note is that > at no point it will try to increase or decrease the > global.offheap.memstore.size. If there is a heap pressure then it will try to > increase the memstore heap limit. > In case of offheap memstore there is always a chance that the heap pressure > does not increase. In that case we could ideally decrease the heap limit for > memstore. The current logic of heapmemory tuner is such that things will > naturally settle down. But on discussion what we thought is let us include > the flush count that happens due to offheap pressure but give that a lesser > weightage and thus ensure that the initial decrease on memstore heap limit > does not happen. Currently that fraction is set as 0.5. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15787) Change the flush related heuristics to work with offheap size configured
[ https://issues.apache.org/jira/browse/HBASE-15787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15787: --- Status: Open (was: Patch Available) > Change the flush related heuristics to work with offheap size configured > > > Key: HBASE-15787 > URL: https://issues.apache.org/jira/browse/HBASE-15787 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-15787.patch, HBASE-15787_1.patch, > HBASE-15787_4.patch, HBASE-15787_5.patch, HBASE-15787_6.patch, > HBASE-15787_7.patch > > > With offheap MSLAB in place we may have to change the flush related > heuristics to work with offheap size configured rather than the java heap > size. > Since we now have clear seperation of the memstore data size and memstore > heap size, for offheap memstore > -> Decide if the global.offheap.memstore.size is breached for blocking > updates and force flushes. > -> If the onheap global.memstore.size is breached (due to heap overhead) even > then block updates and force flushes. > -> The global.memstore.size.lower.limit is now by default 95% of the > global.memstore.size. So now we apply this 95% on the > global.offheap.memstore.size and also on global.memstore.size (as it was done > for onheap case). > -> We will have new FlushTypes introduced > {code} > ABOVE_ONHEAP_LOWER_MARK, /* happens due to lower mark breach of onheap > memstore settings > An offheap memstore can even breach the > onheap_lower_mark*/ > ABOVE_ONHEAP_HIGHER_MARK,/* happens due to higher mark breach of onheap > memstore settings > An offheap memstore can even breach the > onheap_higher_mark*/ > ABOVE_OFFHEAP_LOWER_MARK,/* happens due to lower mark breach of offheap > memstore settings*/ > ABOVE_OFFHEAP_HIGHER_MARK; > {code} > -> regionServerAccounting does all the accounting. > -> HeapMemoryTuner is what is litte tricky here. First thing to note is that > at no point it will try to increase or decrease the > global.offheap.memstore.size. If there is a heap pressure then it will try to > increase the memstore heap limit. > In case of offheap memstore there is always a chance that the heap pressure > does not increase. In that case we could ideally decrease the heap limit for > memstore. The current logic of heapmemory tuner is such that things will > naturally settle down. But on discussion what we thought is let us include > the flush count that happens due to offheap pressure but give that a lesser > weightage and thus ensure that the initial decrease on memstore heap limit > does not happen. Currently that fraction is set as 0.5. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15787) Change the flush related heuristics to work with offheap size configured
[ https://issues.apache.org/jira/browse/HBASE-15787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15787: --- Status: Patch Available (was: Open) > Change the flush related heuristics to work with offheap size configured > > > Key: HBASE-15787 > URL: https://issues.apache.org/jira/browse/HBASE-15787 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-15787.patch, HBASE-15787_1.patch, > HBASE-15787_4.patch, HBASE-15787_5.patch, HBASE-15787_6.patch, > HBASE-15787_7.patch > > > With offheap MSLAB in place we may have to change the flush related > heuristics to work with offheap size configured rather than the java heap > size. > Since we now have clear seperation of the memstore data size and memstore > heap size, for offheap memstore > -> Decide if the global.offheap.memstore.size is breached for blocking > updates and force flushes. > -> If the onheap global.memstore.size is breached (due to heap overhead) even > then block updates and force flushes. > -> The global.memstore.size.lower.limit is now by default 95% of the > global.memstore.size. So now we apply this 95% on the > global.offheap.memstore.size and also on global.memstore.size (as it was done > for onheap case). > -> We will have new FlushTypes introduced > {code} > ABOVE_ONHEAP_LOWER_MARK, /* happens due to lower mark breach of onheap > memstore settings > An offheap memstore can even breach the > onheap_lower_mark*/ > ABOVE_ONHEAP_HIGHER_MARK,/* happens due to higher mark breach of onheap > memstore settings > An offheap memstore can even breach the > onheap_higher_mark*/ > ABOVE_OFFHEAP_LOWER_MARK,/* happens due to lower mark breach of offheap > memstore settings*/ > ABOVE_OFFHEAP_HIGHER_MARK; > {code} > -> regionServerAccounting does all the accounting. > -> HeapMemoryTuner is what is litte tricky here. First thing to note is that > at no point it will try to increase or decrease the > global.offheap.memstore.size. If there is a heap pressure then it will try to > increase the memstore heap limit. > In case of offheap memstore there is always a chance that the heap pressure > does not increase. In that case we could ideally decrease the heap limit for > memstore. The current logic of heapmemory tuner is such that things will > naturally settle down. But on discussion what we thought is let us include > the flush count that happens due to offheap pressure but give that a lesser > weightage and thus ensure that the initial decrease on memstore heap limit > does not happen. Currently that fraction is set as 0.5. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15787) Change the flush related heuristics to work with offheap size configured
[ https://issues.apache.org/jira/browse/HBASE-15787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15787: --- Status: Open (was: Patch Available) > Change the flush related heuristics to work with offheap size configured > > > Key: HBASE-15787 > URL: https://issues.apache.org/jira/browse/HBASE-15787 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-15787.patch, HBASE-15787_1.patch, > HBASE-15787_4.patch, HBASE-15787_5.patch, HBASE-15787_6.patch, > HBASE-15787_7.patch > > > With offheap MSLAB in place we may have to change the flush related > heuristics to work with offheap size configured rather than the java heap > size. > Since we now have clear seperation of the memstore data size and memstore > heap size, for offheap memstore > -> Decide if the global.offheap.memstore.size is breached for blocking > updates and force flushes. > -> If the onheap global.memstore.size is breached (due to heap overhead) even > then block updates and force flushes. > -> The global.memstore.size.lower.limit is now by default 95% of the > global.memstore.size. So now we apply this 95% on the > global.offheap.memstore.size and also on global.memstore.size (as it was done > for onheap case). > -> We will have new FlushTypes introduced > {code} > ABOVE_ONHEAP_LOWER_MARK, /* happens due to lower mark breach of onheap > memstore settings > An offheap memstore can even breach the > onheap_lower_mark*/ > ABOVE_ONHEAP_HIGHER_MARK,/* happens due to higher mark breach of onheap > memstore settings > An offheap memstore can even breach the > onheap_higher_mark*/ > ABOVE_OFFHEAP_LOWER_MARK,/* happens due to lower mark breach of offheap > memstore settings*/ > ABOVE_OFFHEAP_HIGHER_MARK; > {code} > -> regionServerAccounting does all the accounting. > -> HeapMemoryTuner is what is litte tricky here. First thing to note is that > at no point it will try to increase or decrease the > global.offheap.memstore.size. If there is a heap pressure then it will try to > increase the memstore heap limit. > In case of offheap memstore there is always a chance that the heap pressure > does not increase. In that case we could ideally decrease the heap limit for > memstore. The current logic of heapmemory tuner is such that things will > naturally settle down. But on discussion what we thought is let us include > the flush count that happens due to offheap pressure but give that a lesser > weightage and thus ensure that the initial decrease on memstore heap limit > does not happen. Currently that fraction is set as 0.5. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15787) Change the flush related heuristics to work with offheap size configured
[ https://issues.apache.org/jira/browse/HBASE-15787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15787: --- Attachment: HBASE-15787_7.patch Updated patch. > Change the flush related heuristics to work with offheap size configured > > > Key: HBASE-15787 > URL: https://issues.apache.org/jira/browse/HBASE-15787 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-15787.patch, HBASE-15787_1.patch, > HBASE-15787_4.patch, HBASE-15787_5.patch, HBASE-15787_6.patch, > HBASE-15787_7.patch > > > With offheap MSLAB in place we may have to change the flush related > heuristics to work with offheap size configured rather than the java heap > size. > Since we now have clear seperation of the memstore data size and memstore > heap size, for offheap memstore > -> Decide if the global.offheap.memstore.size is breached for blocking > updates and force flushes. > -> If the onheap global.memstore.size is breached (due to heap overhead) even > then block updates and force flushes. > -> The global.memstore.size.lower.limit is now by default 95% of the > global.memstore.size. So now we apply this 95% on the > global.offheap.memstore.size and also on global.memstore.size (as it was done > for onheap case). > -> We will have new FlushTypes introduced > {code} > ABOVE_ONHEAP_LOWER_MARK, /* happens due to lower mark breach of onheap > memstore settings > An offheap memstore can even breach the > onheap_lower_mark*/ > ABOVE_ONHEAP_HIGHER_MARK,/* happens due to higher mark breach of onheap > memstore settings > An offheap memstore can even breach the > onheap_higher_mark*/ > ABOVE_OFFHEAP_LOWER_MARK,/* happens due to lower mark breach of offheap > memstore settings*/ > ABOVE_OFFHEAP_HIGHER_MARK; > {code} > -> regionServerAccounting does all the accounting. > -> HeapMemoryTuner is what is litte tricky here. First thing to note is that > at no point it will try to increase or decrease the > global.offheap.memstore.size. If there is a heap pressure then it will try to > increase the memstore heap limit. > In case of offheap memstore there is always a chance that the heap pressure > does not increase. In that case we could ideally decrease the heap limit for > memstore. The current logic of heapmemory tuner is such that things will > naturally settle down. But on discussion what we thought is let us include > the flush count that happens due to offheap pressure but give that a lesser > weightage and thus ensure that the initial decrease on memstore heap limit > does not happen. Currently that fraction is set as 0.5. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15787) Change the flush related heuristics to work with offheap size configured
[ https://issues.apache.org/jira/browse/HBASE-15787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15787: --- Attachment: HBASE-15787_6.patch Correct findbugs and the testcase. > Change the flush related heuristics to work with offheap size configured > > > Key: HBASE-15787 > URL: https://issues.apache.org/jira/browse/HBASE-15787 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-15787.patch, HBASE-15787_1.patch, > HBASE-15787_4.patch, HBASE-15787_5.patch, HBASE-15787_6.patch > > > With offheap MSLAB in place we may have to change the flush related > heuristics to work with offheap size configured rather than the java heap > size. > Since we now have clear seperation of the memstore data size and memstore > heap size, for offheap memstore > -> Decide if the global.offheap.memstore.size is breached for blocking > updates and force flushes. > -> If the onheap global.memstore.size is breached (due to heap overhead) even > then block updates and force flushes. > -> The global.memstore.size.lower.limit is now by default 95% of the > global.memstore.size. So now we apply this 95% on the > global.offheap.memstore.size and also on global.memstore.size (as it was done > for onheap case). > -> We will have new FlushTypes introduced > {code} > ABOVE_ONHEAP_LOWER_MARK, /* happens due to lower mark breach of onheap > memstore settings > An offheap memstore can even breach the > onheap_lower_mark*/ > ABOVE_ONHEAP_HIGHER_MARK,/* happens due to higher mark breach of onheap > memstore settings > An offheap memstore can even breach the > onheap_higher_mark*/ > ABOVE_OFFHEAP_LOWER_MARK,/* happens due to lower mark breach of offheap > memstore settings*/ > ABOVE_OFFHEAP_HIGHER_MARK; > {code} > -> regionServerAccounting does all the accounting. > -> HeapMemoryTuner is what is litte tricky here. First thing to note is that > at no point it will try to increase or decrease the > global.offheap.memstore.size. If there is a heap pressure then it will try to > increase the memstore heap limit. > In case of offheap memstore there is always a chance that the heap pressure > does not increase. In that case we could ideally decrease the heap limit for > memstore. The current logic of heapmemory tuner is such that things will > naturally settle down. But on discussion what we thought is let us include > the flush count that happens due to offheap pressure but give that a lesser > weightage and thus ensure that the initial decrease on memstore heap limit > does not happen. Currently that fraction is set as 0.5. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15787) Change the flush related heuristics to work with offheap size configured
[ https://issues.apache.org/jira/browse/HBASE-15787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15787: --- Hadoop Flags: Reviewed Status: Patch Available (was: Open) > Change the flush related heuristics to work with offheap size configured > > > Key: HBASE-15787 > URL: https://issues.apache.org/jira/browse/HBASE-15787 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-15787.patch, HBASE-15787_1.patch, > HBASE-15787_4.patch, HBASE-15787_5.patch, HBASE-15787_6.patch > > > With offheap MSLAB in place we may have to change the flush related > heuristics to work with offheap size configured rather than the java heap > size. > Since we now have clear seperation of the memstore data size and memstore > heap size, for offheap memstore > -> Decide if the global.offheap.memstore.size is breached for blocking > updates and force flushes. > -> If the onheap global.memstore.size is breached (due to heap overhead) even > then block updates and force flushes. > -> The global.memstore.size.lower.limit is now by default 95% of the > global.memstore.size. So now we apply this 95% on the > global.offheap.memstore.size and also on global.memstore.size (as it was done > for onheap case). > -> We will have new FlushTypes introduced > {code} > ABOVE_ONHEAP_LOWER_MARK, /* happens due to lower mark breach of onheap > memstore settings > An offheap memstore can even breach the > onheap_lower_mark*/ > ABOVE_ONHEAP_HIGHER_MARK,/* happens due to higher mark breach of onheap > memstore settings > An offheap memstore can even breach the > onheap_higher_mark*/ > ABOVE_OFFHEAP_LOWER_MARK,/* happens due to lower mark breach of offheap > memstore settings*/ > ABOVE_OFFHEAP_HIGHER_MARK; > {code} > -> regionServerAccounting does all the accounting. > -> HeapMemoryTuner is what is litte tricky here. First thing to note is that > at no point it will try to increase or decrease the > global.offheap.memstore.size. If there is a heap pressure then it will try to > increase the memstore heap limit. > In case of offheap memstore there is always a chance that the heap pressure > does not increase. In that case we could ideally decrease the heap limit for > memstore. The current logic of heapmemory tuner is such that things will > naturally settle down. But on discussion what we thought is let us include > the flush count that happens due to offheap pressure but give that a lesser > weightage and thus ensure that the initial decrease on memstore heap limit > does not happen. Currently that fraction is set as 0.5. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15787) Change the flush related heuristics to work with offheap size configured
[ https://issues.apache.org/jira/browse/HBASE-15787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15787: --- Status: Open (was: Patch Available) > Change the flush related heuristics to work with offheap size configured > > > Key: HBASE-15787 > URL: https://issues.apache.org/jira/browse/HBASE-15787 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-15787.patch, HBASE-15787_1.patch, > HBASE-15787_4.patch, HBASE-15787_5.patch > > > With offheap MSLAB in place we may have to change the flush related > heuristics to work with offheap size configured rather than the java heap > size. > Since we now have clear seperation of the memstore data size and memstore > heap size, for offheap memstore > -> Decide if the global.offheap.memstore.size is breached for blocking > updates and force flushes. > -> If the onheap global.memstore.size is breached (due to heap overhead) even > then block updates and force flushes. > -> The global.memstore.size.lower.limit is now by default 95% of the > global.memstore.size. So now we apply this 95% on the > global.offheap.memstore.size and also on global.memstore.size (as it was done > for onheap case). > -> We will have new FlushTypes introduced > {code} > ABOVE_ONHEAP_LOWER_MARK, /* happens due to lower mark breach of onheap > memstore settings > An offheap memstore can even breach the > onheap_lower_mark*/ > ABOVE_ONHEAP_HIGHER_MARK,/* happens due to higher mark breach of onheap > memstore settings > An offheap memstore can even breach the > onheap_higher_mark*/ > ABOVE_OFFHEAP_LOWER_MARK,/* happens due to lower mark breach of offheap > memstore settings*/ > ABOVE_OFFHEAP_HIGHER_MARK; > {code} > -> regionServerAccounting does all the accounting. > -> HeapMemoryTuner is what is litte tricky here. First thing to note is that > at no point it will try to increase or decrease the > global.offheap.memstore.size. If there is a heap pressure then it will try to > increase the memstore heap limit. > In case of offheap memstore there is always a chance that the heap pressure > does not increase. In that case we could ideally decrease the heap limit for > memstore. The current logic of heapmemory tuner is such that things will > naturally settle down. But on discussion what we thought is let us include > the flush count that happens due to offheap pressure but give that a lesser > weightage and thus ensure that the initial decrease on memstore heap limit > does not happen. Currently that fraction is set as 0.5. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15787) Change the flush related heuristics to work with offheap size configured
[ https://issues.apache.org/jira/browse/HBASE-15787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15787: --- Attachment: HBASE-15787_5.patch Updated patch as in RB. > Change the flush related heuristics to work with offheap size configured > > > Key: HBASE-15787 > URL: https://issues.apache.org/jira/browse/HBASE-15787 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-15787.patch, HBASE-15787_1.patch, > HBASE-15787_4.patch, HBASE-15787_5.patch > > > With offheap MSLAB in place we may have to change the flush related > heuristics to work with offheap size configured rather than the java heap > size. > Since we now have clear seperation of the memstore data size and memstore > heap size, for offheap memstore > -> Decide if the global.offheap.memstore.size is breached for blocking > updates and force flushes. > -> If the onheap global.memstore.size is breached (due to heap overhead) even > then block updates and force flushes. > -> The global.memstore.size.lower.limit is now by default 95% of the > global.memstore.size. So now we apply this 95% on the > global.offheap.memstore.size and also on global.memstore.size (as it was done > for onheap case). > -> We will have new FlushTypes introduced > {code} > ABOVE_ONHEAP_LOWER_MARK, /* happens due to lower mark breach of onheap > memstore settings > An offheap memstore can even breach the > onheap_lower_mark*/ > ABOVE_ONHEAP_HIGHER_MARK,/* happens due to higher mark breach of onheap > memstore settings > An offheap memstore can even breach the > onheap_higher_mark*/ > ABOVE_OFFHEAP_LOWER_MARK,/* happens due to lower mark breach of offheap > memstore settings*/ > ABOVE_OFFHEAP_HIGHER_MARK; > {code} > -> regionServerAccounting does all the accounting. > -> HeapMemoryTuner is what is litte tricky here. First thing to note is that > at no point it will try to increase or decrease the > global.offheap.memstore.size. If there is a heap pressure then it will try to > increase the memstore heap limit. > In case of offheap memstore there is always a chance that the heap pressure > does not increase. In that case we could ideally decrease the heap limit for > memstore. The current logic of heapmemory tuner is such that things will > naturally settle down. But on discussion what we thought is let us include > the flush count that happens due to offheap pressure but give that a lesser > weightage and thus ensure that the initial decrease on memstore heap limit > does not happen. Currently that fraction is set as 0.5. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15787) Change the flush related heuristics to work with offheap size configured
[ https://issues.apache.org/jira/browse/HBASE-15787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15787: --- Status: Patch Available (was: Open) > Change the flush related heuristics to work with offheap size configured > > > Key: HBASE-15787 > URL: https://issues.apache.org/jira/browse/HBASE-15787 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-15787.patch, HBASE-15787_1.patch, > HBASE-15787_4.patch, HBASE-15787_5.patch > > > With offheap MSLAB in place we may have to change the flush related > heuristics to work with offheap size configured rather than the java heap > size. > Since we now have clear seperation of the memstore data size and memstore > heap size, for offheap memstore > -> Decide if the global.offheap.memstore.size is breached for blocking > updates and force flushes. > -> If the onheap global.memstore.size is breached (due to heap overhead) even > then block updates and force flushes. > -> The global.memstore.size.lower.limit is now by default 95% of the > global.memstore.size. So now we apply this 95% on the > global.offheap.memstore.size and also on global.memstore.size (as it was done > for onheap case). > -> We will have new FlushTypes introduced > {code} > ABOVE_ONHEAP_LOWER_MARK, /* happens due to lower mark breach of onheap > memstore settings > An offheap memstore can even breach the > onheap_lower_mark*/ > ABOVE_ONHEAP_HIGHER_MARK,/* happens due to higher mark breach of onheap > memstore settings > An offheap memstore can even breach the > onheap_higher_mark*/ > ABOVE_OFFHEAP_LOWER_MARK,/* happens due to lower mark breach of offheap > memstore settings*/ > ABOVE_OFFHEAP_HIGHER_MARK; > {code} > -> regionServerAccounting does all the accounting. > -> HeapMemoryTuner is what is litte tricky here. First thing to note is that > at no point it will try to increase or decrease the > global.offheap.memstore.size. If there is a heap pressure then it will try to > increase the memstore heap limit. > In case of offheap memstore there is always a chance that the heap pressure > does not increase. In that case we could ideally decrease the heap limit for > memstore. The current logic of heapmemory tuner is such that things will > naturally settle down. But on discussion what we thought is let us include > the flush count that happens due to offheap pressure but give that a lesser > weightage and thus ensure that the initial decrease on memstore heap limit > does not happen. Currently that fraction is set as 0.5. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15787) Change the flush related heuristics to work with offheap size configured
[ https://issues.apache.org/jira/browse/HBASE-15787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15787: --- Attachment: HBASE-15787_4.patch New patch. Remove OFFHEAP_HIGH(LOW)_MARK from Heapmemory tuner. Except that it knows whether the current memstore is offheap memstore. Will add to RB. > Change the flush related heuristics to work with offheap size configured > > > Key: HBASE-15787 > URL: https://issues.apache.org/jira/browse/HBASE-15787 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-15787.patch, HBASE-15787_1.patch, > HBASE-15787_4.patch > > > With offheap MSLAB in place we may have to change the flush related > heuristics to work with offheap size configured rather than the java heap > size. > Since we now have clear seperation of the memstore data size and memstore > heap size, for offheap memstore > -> Decide if the global.offheap.memstore.size is breached for blocking > updates and force flushes. > -> If the onheap global.memstore.size is breached (due to heap overhead) even > then block updates and force flushes. > -> The global.memstore.size.lower.limit is now by default 95% of the > global.memstore.size. So now we apply this 95% on the > global.offheap.memstore.size and also on global.memstore.size (as it was done > for onheap case). > -> We will have new FlushTypes introduced > {code} > ABOVE_ONHEAP_LOWER_MARK, /* happens due to lower mark breach of onheap > memstore settings > An offheap memstore can even breach the > onheap_lower_mark*/ > ABOVE_ONHEAP_HIGHER_MARK,/* happens due to higher mark breach of onheap > memstore settings > An offheap memstore can even breach the > onheap_higher_mark*/ > ABOVE_OFFHEAP_LOWER_MARK,/* happens due to lower mark breach of offheap > memstore settings*/ > ABOVE_OFFHEAP_HIGHER_MARK; > {code} > -> regionServerAccounting does all the accounting. > -> HeapMemoryTuner is what is litte tricky here. First thing to note is that > at no point it will try to increase or decrease the > global.offheap.memstore.size. If there is a heap pressure then it will try to > increase the memstore heap limit. > In case of offheap memstore there is always a chance that the heap pressure > does not increase. In that case we could ideally decrease the heap limit for > memstore. The current logic of heapmemory tuner is such that things will > naturally settle down. But on discussion what we thought is let us include > the flush count that happens due to offheap pressure but give that a lesser > weightage and thus ensure that the initial decrease on memstore heap limit > does not happen. Currently that fraction is set as 0.5. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15787) Change the flush related heuristics to work with offheap size configured
[ https://issues.apache.org/jira/browse/HBASE-15787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15787: --- Attachment: HBASE-15787_1.patch this patch adds test case and does have better mechanism to track offheap memstore affecting the heap memory manager. If we have offheap memstore and we are sure that there are no flushes due to onheap pressure, the 'step' function that we apply to decrease the memstore size incase we need to increase the block_cache size (under read heavy cases) we do that with minstep function. this wil ensure that we don't reduce the heap limit of the offheap memstore which will lead to heap overhead flushes. In cases where there are only flushes due to offheap pressure and it is write scenario we won't adjust the memstore size at all. All cases covered in test cases. Pls have a look at it. > Change the flush related heuristics to work with offheap size configured > > > Key: HBASE-15787 > URL: https://issues.apache.org/jira/browse/HBASE-15787 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-15787.patch, HBASE-15787_1.patch > > > With offheap MSLAB in place we may have to change the flush related > heuristics to work with offheap size configured rather than the java heap > size. > Since we now have clear seperation of the memstore data size and memstore > heap size, for offheap memstore > -> Decide if the global.offheap.memstore.size is breached for blocking > updates and force flushes. > -> If the onheap global.memstore.size is breached (due to heap overhead) even > then block updates and force flushes. > -> The global.memstore.size.lower.limit is now by default 95% of the > global.memstore.size. So now we apply this 95% on the > global.offheap.memstore.size and also on global.memstore.size (as it was done > for onheap case). > -> We will have new FlushTypes introduced > {code} > ABOVE_ONHEAP_LOWER_MARK, /* happens due to lower mark breach of onheap > memstore settings > An offheap memstore can even breach the > onheap_lower_mark*/ > ABOVE_ONHEAP_HIGHER_MARK,/* happens due to higher mark breach of onheap > memstore settings > An offheap memstore can even breach the > onheap_higher_mark*/ > ABOVE_OFFHEAP_LOWER_MARK,/* happens due to lower mark breach of offheap > memstore settings*/ > ABOVE_OFFHEAP_HIGHER_MARK; > {code} > -> regionServerAccounting does all the accounting. > -> HeapMemoryTuner is what is litte tricky here. First thing to note is that > at no point it will try to increase or decrease the > global.offheap.memstore.size. If there is a heap pressure then it will try to > increase the memstore heap limit. > In case of offheap memstore there is always a chance that the heap pressure > does not increase. In that case we could ideally decrease the heap limit for > memstore. The current logic of heapmemory tuner is such that things will > naturally settle down. But on discussion what we thought is let us include > the flush count that happens due to offheap pressure but give that a lesser > weightage and thus ensure that the initial decrease on memstore heap limit > does not happen. Currently that fraction is set as 0.5. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15787) Change the flush related heuristics to work with offheap size configured
[ https://issues.apache.org/jira/browse/HBASE-15787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15787: --- Description: With offheap MSLAB in place we may have to change the flush related heuristics to work with offheap size configured rather than the java heap size. Since we now have clear seperation of the memstore data size and memstore heap size, for offheap memstore -> Decide if the global.offheap.memstore.size is breached for blocking updates and force flushes. -> If the onheap global.memstore.size is breached (due to heap overhead) even then block updates and force flushes. -> The global.memstore.size.lower.limit is now by default 95% of the global.memstore.size. So now we apply this 95% on the global.offheap.memstore.size and also on global.memstore.size (as it was done for onheap case). -> We will have new FlushTypes introduced {code} ABOVE_ONHEAP_LOWER_MARK, /* happens due to lower mark breach of onheap memstore settings An offheap memstore can even breach the onheap_lower_mark*/ ABOVE_ONHEAP_HIGHER_MARK,/* happens due to higher mark breach of onheap memstore settings An offheap memstore can even breach the onheap_higher_mark*/ ABOVE_OFFHEAP_LOWER_MARK,/* happens due to lower mark breach of offheap memstore settings*/ ABOVE_OFFHEAP_HIGHER_MARK; {code} -> regionServerAccounting does all the accounting. -> HeapMemoryTuner is what is litte tricky here. First thing to note is that at no point it will try to increase or decrease the global.offheap.memstore.size. If there is a heap pressure then it will try to increase the memstore heap limit. In case of offheap memstore there is always a chance that the heap pressure does not increase. In that case we could ideally decrease the heap limit for memstore. The current logic of heapmemory tuner is such that things will naturally settle down. But on discussion what we thought is let us include the flush count that happens due to offheap pressure but give that a lesser weightage and thus ensure that the initial decrease on memstore heap limit does not happen. Currently that fraction is set as 0.5. was: With offheap MSLAB in place we may have to change the flush related heuristics to work with offheap size configured rather than the java heap size. Since we now have clear seperation of the memstore data size and memstore heap size, for offheap memstore -> Decide if the global.offheap.memstore.size is breached for blocking updates and force flushes. -> If the onheap global.memstore.size is breached (due to heap overhead) even then block updates and force flushes. -> The global.memstore.size.lower.limit is now by default 95% of the global.memstore.size. So now we apply this 95% on the global.offheap.memstore.size and also on global.memstore.size (as it was done for onheap case). -> We will have new FlushTypes introduces {code} ABOVE_ONHEAP_LOWER_MARK, /* happens due to lower mark breach of onheap memstore settings An offheap memstore can even breach the onheap_lower_mark*/ ABOVE_ONHEAP_HIGHER_MARK,/* happens due to higher mark breach of onheap memstore settings An offheap memstore can even breach the onheap_higher_mark*/ ABOVE_OFFHEAP_LOWER_MARK,/* happens due to lower mark breach of offheap memstore settings*/ ABOVE_OFFHEAP_HIGHER_MARK; {code} -> regionServerAccounting does all the accounting. -> HeapMemoryTuner is what is litte tricky here. First thing to note is that at no point it will try to increase or decrease the global.offheap.memstore.size. If there is a heap pressure then it will try to increase the memstore heap limit. In case of offheap memstore there is always a chance that the heap pressure does not increase. In that case we could ideally decrease the heap limit for memstore. The current logic of heapmemory tuner is such that things will naturally settle down. But on discussion what we thought is let us include the flush count that happens due to offheap pressure but give that a lesser weightage and thus ensure that the initial decrease on memstore heap limit does not happen. Currently that fraction is set as 0.5. > Change the flush related heuristics to work with offheap size configured > > > Key: HBASE-15787 > URL: https://issues.apache.org/jira/browse/HBASE-15787 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-15787.patch > > > With offheap MSLAB in place we may have to change the flush related > heuristics to work with offheap size configured
[jira] [Updated] (HBASE-15787) Change the flush related heuristics to work with offheap size configured
[ https://issues.apache.org/jira/browse/HBASE-15787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15787: --- Description: With offheap MSLAB in place we may have to change the flush related heuristics to work with offheap size configured rather than the java heap size. Since we now have clear seperation of the memstore data size and memstore heap size, for offheap memstore -> Decide if the global.offheap.memstore.size is breached for blocking updates and force flushes. -> If the onheap global.memstore.size is breached (due to heap overhead) even then block updates and force flushes. -> The global.memstore.size.lower.limit is now by default 95% of the global.memstore.size. So now we apply this 95% on the global.offheap.memstore.size and also on global.memstore.size (as it was done for onheap case). -> We will have new FlushTypes introduces {code} ABOVE_ONHEAP_LOWER_MARK, /* happens due to lower mark breach of onheap memstore settings An offheap memstore can even breach the onheap_lower_mark*/ ABOVE_ONHEAP_HIGHER_MARK,/* happens due to higher mark breach of onheap memstore settings An offheap memstore can even breach the onheap_higher_mark*/ ABOVE_OFFHEAP_LOWER_MARK,/* happens due to lower mark breach of offheap memstore settings*/ ABOVE_OFFHEAP_HIGHER_MARK; {code} -> regionServerAccounting does all the accounting. -> HeapMemoryTuner is what is litte tricky here. First thing to note is that at no point it will try to increase or decrease the global.offheap.memstore.size. If there is a heap pressure then it will try to increase the memstore heap limit. In case of offheap memstore there is always a chance that the heap pressure does not increase. In that case we could ideally decrease the heap limit for memstore. The current logic of heapmemory tuner is such that things will naturally settle down. But on discussion what we thought is let us include the flush count that happens due to offheap pressure but give that a lesser weightage and thus ensure that the initial decrease on memstore heap limit does not happen. Currently that fraction is set as 0.5. was:With offheap MSLAB in place we may have to change the flush related heuristics to work with offheap size configured rather than the java heap size. > Change the flush related heuristics to work with offheap size configured > > > Key: HBASE-15787 > URL: https://issues.apache.org/jira/browse/HBASE-15787 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-15787.patch > > > With offheap MSLAB in place we may have to change the flush related > heuristics to work with offheap size configured rather than the java heap > size. > Since we now have clear seperation of the memstore data size and memstore > heap size, for offheap memstore > -> Decide if the global.offheap.memstore.size is breached for blocking > updates and force flushes. > -> If the onheap global.memstore.size is breached (due to heap overhead) even > then block updates and force flushes. > -> The global.memstore.size.lower.limit is now by default 95% of the > global.memstore.size. So now we apply this 95% on the > global.offheap.memstore.size and also on global.memstore.size (as it was done > for onheap case). > -> We will have new FlushTypes introduces > {code} > ABOVE_ONHEAP_LOWER_MARK, /* happens due to lower mark breach of onheap > memstore settings > An offheap memstore can even breach the > onheap_lower_mark*/ > ABOVE_ONHEAP_HIGHER_MARK,/* happens due to higher mark breach of onheap > memstore settings > An offheap memstore can even breach the > onheap_higher_mark*/ > ABOVE_OFFHEAP_LOWER_MARK,/* happens due to lower mark breach of offheap > memstore settings*/ > ABOVE_OFFHEAP_HIGHER_MARK; > {code} > -> regionServerAccounting does all the accounting. > -> HeapMemoryTuner is what is litte tricky here. First thing to note is that > at no point it will try to increase or decrease the > global.offheap.memstore.size. If there is a heap pressure then it will try to > increase the memstore heap limit. > In case of offheap memstore there is always a chance that the heap pressure > does not increase. In that case we could ideally decrease the heap limit for > memstore. The current logic of heapmemory tuner is such that things will > naturally settle down. But on discussion what we thought is let us include > the flush count that happens due to offheap pressure but give that a lesser > weightage and thus ensure
[jira] [Updated] (HBASE-15787) Change the flush related heuristics to work with offheap size configured
[ https://issues.apache.org/jira/browse/HBASE-15787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15787: --- Attachment: HBASE-15787.patch This patch is built on top of HBASE-15786. There are no tests added for now. Will add in subsequent patches. Will update the description. > Change the flush related heuristics to work with offheap size configured > > > Key: HBASE-15787 > URL: https://issues.apache.org/jira/browse/HBASE-15787 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-15787.patch > > > With offheap MSLAB in place we may have to change the flush related > heuristics to work with offheap size configured rather than the java heap > size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)