Wayne mentions a few of them in his original message in this thread, also please have a look at this chapter of the online book: http://hbase.apache.org/book/performance.html
J-D On Mon, Jun 13, 2011 at 11:21 PM, Sheng Chen <[email protected]> wrote: > Thank you JD. > > My hbase works as a log archive storage, there is a continuing stream of > inserting. > So it is normally write heavy. Is there any advice for this scenario? > Thanks. > > Sean > > > > 2011/6/14 Jean-Daniel Cryans <[email protected]> > >> Unless your normal workload is very heavy on writes (which is Wayne's >> case), you're better off using bulk loading: >> http://hbase.apache.org/bulk-loads.html >> >> J-D >> >> On Mon, Jun 13, 2011 at 12:26 AM, Sheng Chen <[email protected]> >> wrote: >> > I've met with the same problem. >> > Update operations are blocked by memstore flushing, and memstore flushing >> is >> > blocked by a compaction ("too many store files, delay flushing for 90s"). >> > >> > Have you got any solutions? >> > >> > 2011/5/23 Wayne <[email protected]> >> > >> >> We have 4 CFs, but only 1 is ever used for a given region. What about >> >> upping >> >> the size per memstore file to 1G? We have 5x limit of 256m which results >> in >> >> lots of messages like "memstore size 1.3g is >= than blocking 1.2g >> size". >> >> Maybe given the bigger region size we need a bigger memstore size? >> >> >> >> Here is a region server log snippet for this occurring 2x in less than a >> 2 >> >> minute period. >> >> >> >> http://pastebin.com/CxAQSXTt >> >> >> >> >> >> On Mon, May 23, 2011 at 11:33 AM, Stack <[email protected]> wrote: >> >> >> >> > On Mon, May 23, 2011 at 6:40 AM, Wayne <[email protected]> wrote: >> >> > > In order to reduce the total number of regions we have up'd the max >> >> > region >> >> > > size to 5g. This has kept us below 100 regions per node but the side >> >> > affect >> >> > > is pauses occurring every 1-2 min under heavy writes to a single >> >> region. >> >> > We >> >> > > see the "too many store files delaying flush up to 90sec" warning >> every >> >> > > couple of minutes. We have upped the size of the memstore flush size >> >> > (256m) >> >> > > as well as upped the blockingstorefiles (15) but these pauses >> >> > > are occurring more than writes are occurring. In the end our write >> >> > > through-put has degraded considerably. >> >> > > >> >> > >> >> > How many column families? Pastebin a regionserver log. You could up >> >> > the number of store files before we put up the blocking writes gate >> >> > but then you might have runaway files to compact. >> >> > >> >> > St.Ack >> >> > >> >> >> > >> >
