Hi Kevin, We have about 160 regions per server with 16Gig region size and 10 drives for Hbase. I've looked at disk IO and that doesn't seem to be any problem ( % utilization is < 2 across all disk)
Any suggestion what heap size I should allocation, normally I allocate 16GB. Also, I read increasing hbase.hstore.blockingStoreFiles and hbase.hregion.memstore.block.multiplier is good idea for write-heavy cluster, but in my case it seem to be heading to wrong direction. Thanks On Fri, Jan 24, 2014 at 6:31 PM, Kevin O'dell <[email protected]> wrote: > Rohit, > > 64GB heap is not ideal, you will run into some weird issues. How many > regions are you running per server, how many drives in each node, any other > settings you changed from default? > On Jan 24, 2014 6:22 PM, "Rohit Dev" <[email protected]> wrote: > >> Hi, >> >> We are running Opentsdb on CDH 4.3 hbase cluster, with most of the >> default settings. The cluster is heavy on write and I'm trying to see >> what parameters I can tune to optimize the write performance. >> >> >> # I get messages related to Memstore[1] and Slow Response[2] very >> often, is this an indication of any issue ? >> >> I tried increasing some parameters on one node: >> - hbase.hstore.blockingStoreFiles - from default 7 to 15 >> - hbase.hregion.memstore.block.multiplier - from default 2 to 8 >> - and heap size from 16GB to 64GB >> >> * 'Compaction queue' went up to ~200 within 60 mins after restarting >> region server with new parameters and the log started to get even more >> noisy. >> >> Can anyone please suggest if I'm going to right direction with these >> new settings ? or if there are other thing that I could monitor or >> change to make it better. >> >> Thank you! >> >> >> [1] >> INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates >> for 'IPC Server handler 19 on 60020' on region >> >> tsdb,\x008XR\xE0i\x90\x00\x00\x02Q\x7F\x1D\x00\x00(\x00\x0B]\x00\x008M(r\x00\x00Bl\xA7\x8C,1390556781703.0771bf90cab25c503d3400206417f6bf.: >> memstore size 256.3 M is >= than blocking 256 M size >> >> [2] >> WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): >> >> {"processingtimems":17887,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@586940ea >> ), >> rpc version=1, client version=29, >> methodsFingerPrint=0","client":"192.168.10.10:54132 >> ","starttimems":1390587959182,"queuetimems":1498,"class":"HRegionServer","responsesize":0,"method":"multi"} >>
