mslab was introduced after 0.20.6 Read Todd's series: http://www.cloudera.com/blog/2011/03/avoiding-full-gcs-in-hbase-with-memstore-local-allocation-buffers-part-3/
Cheers On Thu, Dec 29, 2011 at 12:19 AM, Seraph Imalia <[email protected]> wrote: > Region Servers > > Address Start Code Load > dynobuntu10:60030 1325081250180 requests=43, regions=224, > usedHeap=3946, maxHeap=4087 > dynobuntu12:60030 1325081249966 requests=32, regions=224, > usedHeap=3821, maxHeap=4087 > dynobuntu17:60030 1325081248407 requests=39, regions=225, > usedHeap=4016, maxHeap=4087 > Total: servers: 3 requests=114, regions=673 > > I restarted them yesterday and the number of regions increased from 667 to > 673 and they are about to run out of heap again :(. Should I set that > property to false? - what does mslab do? - is it new after 0.20.6? > > Regards, > Seraph > > On 28 Dec 2011, at 5:46 PM, Ted Yu wrote: > > > Can you tell me how many regions each region server hosts ? > > > > In 0.90.4 there is this parameter: > > <name>hbase.hregion.memstore.mslab.enabled</name> > > <value>true</value> > > mslab tends to consume heap if region count is high. > > > > Cheers > > > > On Wed, Dec 28, 2011 at 6:27 AM, Seraph Imalia <[email protected]> > wrote: > > > >> Hi Guys, > >> > >> After updating from 0.20.6 to 0.90.4, we have been having serious RAM > >> issues. I had hbase-env.sh set to use 3 Gigs of RAM with 0.20.6 but > with > >> 0.90.4 even 4.5 Gigs seems not enough. It does not matter how much load > >> the hbase services are under, it just crashes after 24-48 hours. The > only > >> difference the load makes is how quickly the services crash. Even over > >> this holiday season with our lowest load of the year, it crashes just > after > >> 36 hours of being started. To fix it, I have to run the stop-hbase.sh > >> command, wait a while and kill -9 any hbase processes that have stopped > >> outputting logs or stopped responding, and then run start-hbase.sh > again. > >> > >> Attached are my logs from the latest "start-to-crash". There are 3 > >> servers and hbase is being used for storing URL's - 7 client servers > >> connect to hbase and perform URL Lookups at about 40 requests per second > >> (this is the low load over this holiday season). If the URL does not > >> exist, it gets added. The Key on the HTable is the URL and there are a > few > >> fields stored against it - e.g. DateDiscovered, Host, Script, > QueryString, > >> etc. > >> > >> Each server has a hadoop datanode and an hbase regionserver and 1 of the > >> servers additionally has the namenode, master and zookeeper. On first > >> start, each regionserver uses 2 Gigs (usedHeap) and as soon as I restart > >> the clients, the usedHeap slowly climes until it reaches the maxHeap and > >> shortly after that, the regionservers start crashing - sometimes they > >> actually shutdown gracefully by themselves. > >> > >> Originally, we had hbase.regionserver.handler.count set to 100 and I > have > >> now removed that to leave it as default which has not helped. > >> > >> We have not made any changes to the clients and we have a mirrored > >> instance of this in our UK Data Centre which is still running 0.20.6 and > >> servicing 10 clients currently at over 300 requests per second (again > low > >> load over the holidays) and it is 100% stable. > >> > >> What do I do now? - your website says I cannot downgrade? > >> > >> Please help > >> > >> Regards, > >> Seraph > >> > >> > >> > >> > >> > >> > >
