Minor compactions don't compact all files, only a few of them depending on what the algorithm computes, so flushing before the compaction (which might already be running!) could very well put you in the same situation at the end.
Something we know we need to do is a merging flush: https://issues.apache.org/jira/browse/HBASE-3656 Patches are welcome 24/7 :) J-D On Tue, Jun 14, 2011 at 12:14 AM, Weihua JIANG <[email protected]> wrote: > I looked at the code. This warn message is printed by > MemStoreFlusher.flushRegion(). If there are too many store files, it > first request compaction and wait 90s, then flush mem store. > > My question is: why not flush mem store before compaction? In current > mode, the result is a compacted store file + a new flush store file. > This makes it easier to reach compaction criteria later. If flush > before compaction, then the result is a compacted store file. > > Thanks > Weihua > > 2011/6/13 Sheng Chen <[email protected]>: >> I've met with the same problem. >> Update operations are blocked by memstore flushing, and memstore flushing is >> blocked by a compaction ("too many store files, delay flushing for 90s"). >> >> Have you got any solutions? >> >> 2011/5/23 Wayne <[email protected]> >> >>> We have 4 CFs, but only 1 is ever used for a given region. What about >>> upping >>> the size per memstore file to 1G? We have 5x limit of 256m which results in >>> lots of messages like "memstore size 1.3g is >= than blocking 1.2g size". >>> Maybe given the bigger region size we need a bigger memstore size? >>> >>> Here is a region server log snippet for this occurring 2x in less than a 2 >>> minute period. >>> >>> http://pastebin.com/CxAQSXTt >>> >>> >>> On Mon, May 23, 2011 at 11:33 AM, Stack <[email protected]> wrote: >>> >>> > On Mon, May 23, 2011 at 6:40 AM, Wayne <[email protected]> wrote: >>> > > In order to reduce the total number of regions we have up'd the max >>> > region >>> > > size to 5g. This has kept us below 100 regions per node but the side >>> > affect >>> > > is pauses occurring every 1-2 min under heavy writes to a single >>> region. >>> > We >>> > > see the "too many store files delaying flush up to 90sec" warning every >>> > > couple of minutes. We have upped the size of the memstore flush size >>> > (256m) >>> > > as well as upped the blockingstorefiles (15) but these pauses >>> > > are occurring more than writes are occurring. In the end our write >>> > > through-put has degraded considerably. >>> > > >>> > >>> > How many column families? Pastebin a regionserver log. You could up >>> > the number of store files before we put up the blocking writes gate >>> > but then you might have runaway files to compact. >>> > >>> > St.Ack >>> > >>> >> >
