Thank Cryans, I'll try them! On Fri, Mar 18, 2011 at 12:20 AM, Jean-Daniel Cryans <[email protected]>wrote:
> You can limit the number of WALs and their size on the region server by > tuning: > > hbase.regionserver.maxlogs the default is 32 > hbase.regionserver.hlog.blocksize the default is whatever your HDFS > blocksize times 0.95 > > You can limit the number of parallel threads in the master by tuning: > > hbase.regionserver.hlog.splitlog.writer.threads the default is 3 > hbase.regionserver.hlog.splitlog.buffersize the default is 1024*1024*!28 > > J-D > > On Wed, Mar 16, 2011 at 11:57 PM, 茅旭峰 <[email protected]> wrote: > > Hi, > > > > In our tests, we've accumulated lots of WAL logs, in .logs, which leads > to > > quite long time pause or even > > OOME when restarting either master or region server. We're doing sort of > > bulk import and have not using > > bulk import tricks, like turning off WAL feature. We think it's unknown > how > > our application really use the > > hbase, so it is possible that users doing batch import unless we're > running > > out of space. I wonder if there > > is any property to set to control the size of WAL, would setting smaller > > 'hbase.regionserver.logroll.period' > > help? > > > > On the other hand, since we have lots of regions, the master is easy to > run > > into OOME, due to the occupied > > memory by the instance of Assignment.regions. When we were trying to > restart > > the master, it always died > > with OOME. I think, from the hprof file, it is because the instance of > > HLogSplitter$OutputSink holds too many > > HLogSplitter$WriterAndPaths in logWriters, which even hold the buffer of > > wal.SequenceFileLogWriter. > > Is there any trick to avoid such kind of scenario? > > > > Thanks and regards, > > > > Mao Xu-Feng > > >
