[ https://issues.apache.org/jira/browse/HBASE-10501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900003#comment-13900003 ]
Lars Hofhansl commented on HBASE-10501: --------------------------------------- Thinking about this a bit more. It is probably fairly unlikely that an RS will see 3 or 4 regions of the same table, unless that table is already spread over the cluster. So I think we should ensure we're pretty close to the max file when we see that many regions. In most setups the ration of maxFileSize to flushSize is somewhere around 100:1. (10g regions and 128mb flushsize makes this 80). So if we start with 3*flushsize we get an initial 30:1 ratio and thus we'd reach the max at about 5 (sqrt(30) = 5.4...) That would also avoid the initial split storm after each flush. Should I change the initial size to 3 or 4 x the flush size, instead of making it configurable at all? > Make IncreasingToUpperBoundRegionSplitPolicy configurable > --------------------------------------------------------- > > Key: HBASE-10501 > URL: https://issues.apache.org/jira/browse/HBASE-10501 > Project: HBase > Issue Type: Bug > Reporter: Lars Hofhansl > Attachments: 10501-0.94.txt > > > During some (admittedly artificial) load testing we found a large amount > split activity, which we tracked down the > IncreasingToUpperBoundRegionSplitPolicy. > The current logic is this (from the comments): > "regions that are on this server that all are of the same table, squared, > times the region flush size OR the maximum region split size, whichever is > smaller" > So with a flush size of 128mb and max file size of 20gb, we'd need 13 region > of the same table on an RS to reach the max size. > With 10gb file sized it is still 9 regions of the same table. > Considering that the number of regions that an RS can carry is limited and > there might be multiple tables, this should be more configurable. > I think the squaring is smart and we do not need to change it. > We could > * Make the start size configurable and default it to the flush size > * Add multiplier for the initial size, i.e. start with n * flushSize > * Also change the default to start with 2*flush size > Of course one can override the default split policy, but these seem like > simple tweaks. > Or we could instead set the goal of how many regions of the same table would > need to be present in order to reach the max size. In that case we'd start > with maxSize/goal^2. So if max size is 20gb and the goal is three we'd start > with 20g/9 = 2.2g for the initial region size. > [~stack], I'm especially interested in your opinion. -- This message was sent by Atlassian JIRA (v6.1.5#6160)