You can limit the number of WALs and their size on the region server by tuning:
hbase.regionserver.maxlogs the default is 32 hbase.regionserver.hlog.blocksize the default is whatever your HDFS blocksize times 0.95 You can limit the number of parallel threads in the master by tuning: hbase.regionserver.hlog.splitlog.writer.threads the default is 3 hbase.regionserver.hlog.splitlog.buffersize the default is 1024*1024*!28 J-D On Wed, Mar 16, 2011 at 11:57 PM, 茅旭峰 <[email protected]> wrote: > Hi, > > In our tests, we've accumulated lots of WAL logs, in .logs, which leads to > quite long time pause or even > OOME when restarting either master or region server. We're doing sort of > bulk import and have not using > bulk import tricks, like turning off WAL feature. We think it's unknown how > our application really use the > hbase, so it is possible that users doing batch import unless we're running > out of space. I wonder if there > is any property to set to control the size of WAL, would setting smaller > 'hbase.regionserver.logroll.period' > help? > > On the other hand, since we have lots of regions, the master is easy to run > into OOME, due to the occupied > memory by the instance of Assignment.regions. When we were trying to restart > the master, it always died > with OOME. I think, from the hprof file, it is because the instance of > HLogSplitter$OutputSink holds too many > HLogSplitter$WriterAndPaths in logWriters, which even hold the buffer of > wal.SequenceFileLogWriter. > Is there any trick to avoid such kind of scenario? > > Thanks and regards, > > Mao Xu-Feng >
