We have 4 CFs, but only 1 is ever used for a given region. What about upping the size per memstore file to 1G? We have 5x limit of 256m which results in lots of messages like "memstore size 1.3g is >= than blocking 1.2g size". Maybe given the bigger region size we need a bigger memstore size?
Here is a region server log snippet for this occurring 2x in less than a 2 minute period. http://pastebin.com/CxAQSXTt On Mon, May 23, 2011 at 11:33 AM, Stack <[email protected]> wrote: > On Mon, May 23, 2011 at 6:40 AM, Wayne <[email protected]> wrote: > > In order to reduce the total number of regions we have up'd the max > region > > size to 5g. This has kept us below 100 regions per node but the side > affect > > is pauses occurring every 1-2 min under heavy writes to a single region. > We > > see the "too many store files delaying flush up to 90sec" warning every > > couple of minutes. We have upped the size of the memstore flush size > (256m) > > as well as upped the blockingstorefiles (15) but these pauses > > are occurring more than writes are occurring. In the end our write > > through-put has degraded considerably. > > > > How many column families? Pastebin a regionserver log. You could up > the number of store files before we put up the blocking writes gate > but then you might have runaway files to compact. > > St.Ack >
