Region Servers Address Start Code Load dynobuntu10:60030 1325081250180 requests=43, regions=224, usedHeap=3946, maxHeap=4087 dynobuntu12:60030 1325081249966 requests=32, regions=224, usedHeap=3821, maxHeap=4087 dynobuntu17:60030 1325081248407 requests=39, regions=225, usedHeap=4016, maxHeap=4087 Total: servers: 3 requests=114, regions=673
I restarted them yesterday and the number of regions increased from 667 to 673 and they are about to run out of heap again :(. Should I set that property to false? - what does mslab do? - is it new after 0.20.6? Regards, Seraph On 28 Dec 2011, at 5:46 PM, Ted Yu wrote: > Can you tell me how many regions each region server hosts ? > > In 0.90.4 there is this parameter: > <name>hbase.hregion.memstore.mslab.enabled</name> > <value>true</value> > mslab tends to consume heap if region count is high. > > Cheers > > On Wed, Dec 28, 2011 at 6:27 AM, Seraph Imalia <[email protected]> wrote: > >> Hi Guys, >> >> After updating from 0.20.6 to 0.90.4, we have been having serious RAM >> issues. I had hbase-env.sh set to use 3 Gigs of RAM with 0.20.6 but with >> 0.90.4 even 4.5 Gigs seems not enough. It does not matter how much load >> the hbase services are under, it just crashes after 24-48 hours. The only >> difference the load makes is how quickly the services crash. Even over >> this holiday season with our lowest load of the year, it crashes just after >> 36 hours of being started. To fix it, I have to run the stop-hbase.sh >> command, wait a while and kill -9 any hbase processes that have stopped >> outputting logs or stopped responding, and then run start-hbase.sh again. >> >> Attached are my logs from the latest "start-to-crash". There are 3 >> servers and hbase is being used for storing URL's - 7 client servers >> connect to hbase and perform URL Lookups at about 40 requests per second >> (this is the low load over this holiday season). If the URL does not >> exist, it gets added. The Key on the HTable is the URL and there are a few >> fields stored against it - e.g. DateDiscovered, Host, Script, QueryString, >> etc. >> >> Each server has a hadoop datanode and an hbase regionserver and 1 of the >> servers additionally has the namenode, master and zookeeper. On first >> start, each regionserver uses 2 Gigs (usedHeap) and as soon as I restart >> the clients, the usedHeap slowly climes until it reaches the maxHeap and >> shortly after that, the regionservers start crashing - sometimes they >> actually shutdown gracefully by themselves. >> >> Originally, we had hbase.regionserver.handler.count set to 100 and I have >> now removed that to leave it as default which has not helped. >> >> We have not made any changes to the clients and we have a mirrored >> instance of this in our UK Data Centre which is still running 0.20.6 and >> servicing 10 clients currently at over 300 requests per second (again low >> load over the holidays) and it is 100% stable. >> >> What do I do now? - your website says I cannot downgrade? >> >> Please help >> >> Regards, >> Seraph >> >> >> >> >> >>
