[ https://issues.apache.org/jira/browse/HBASE-15787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15748174#comment-15748174 ]
Hadoop QA commented on HBASE-15787: ----------------------------------- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s {color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 4 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 54s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 41s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 13s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 42s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 41s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 43s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 14s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 25m 34s {color} | {color:green} Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 49s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 88m 52s {color} | {color:green} hbase-server in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 126m 10s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12843197/HBASE-15787_8.patch | | JIRA Issue | HBASE-15787 | | Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 7267434fa456 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh | | git revision | master / a73b0b3 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/4914/testReport/ | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/4914/console | | Powered by | Apache Yetus 0.3.0 http://yetus.apache.org | This message was automatically generated. > Change the flush related heuristics to work with offheap size configured > ------------------------------------------------------------------------ > > Key: HBASE-15787 > URL: https://issues.apache.org/jira/browse/HBASE-15787 > Project: HBase > Issue Type: Sub-task > Components: regionserver > Reporter: ramkrishna.s.vasudevan > Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-15787.patch, HBASE-15787_1.patch, > HBASE-15787_4.patch, HBASE-15787_5.patch, HBASE-15787_6.patch, > HBASE-15787_7.patch, HBASE-15787_8.patch > > > With offheap MSLAB in place we may have to change the flush related > heuristics to work with offheap size configured rather than the java heap > size. > Since we now have clear seperation of the memstore data size and memstore > heap size, for offheap memstore > -> Decide if the global.offheap.memstore.size is breached for blocking > updates and force flushes. > -> If the onheap global.memstore.size is breached (due to heap overhead) even > then block updates and force flushes. > -> The global.memstore.size.lower.limit is now by default 95% of the > global.memstore.size. So now we apply this 95% on the > global.offheap.memstore.size and also on global.memstore.size (as it was done > for onheap case). > -> We will have new FlushTypes introduced > {code} > ABOVE_ONHEAP_LOWER_MARK, /* happens due to lower mark breach of onheap > memstore settings > An offheap memstore can even breach the > onheap_lower_mark*/ > ABOVE_ONHEAP_HIGHER_MARK,/* happens due to higher mark breach of onheap > memstore settings > An offheap memstore can even breach the > onheap_higher_mark*/ > ABOVE_OFFHEAP_LOWER_MARK,/* happens due to lower mark breach of offheap > memstore settings*/ > ABOVE_OFFHEAP_HIGHER_MARK; > {code} > -> regionServerAccounting does all the accounting. > -> HeapMemoryTuner is what is litte tricky here. First thing to note is that > at no point it will try to increase or decrease the > global.offheap.memstore.size. If there is a heap pressure then it will try to > increase the memstore heap limit. > In case of offheap memstore there is always a chance that the heap pressure > does not increase. In that case we could ideally decrease the heap limit for > memstore. The current logic of heapmemory tuner is such that things will > naturally settle down. But on discussion what we thought is let us include > the flush count that happens due to offheap pressure but give that a lesser > weightage and thus ensure that the initial decrease on memstore heap limit > does not happen. Currently that fraction is set as 0.5. -- This message was sent by Atlassian JIRA (v6.3.4#6332)