On Wed, Nov 21, 2012 at 1:04 AM, Vincent Barat <[email protected]> wrote: > Hi, > > I've checked my 30 RPC handlers, they are all in a WAITING state: > > Thread 89 (PRI IPC Server handler 6 on 60020): > State: WAITING > Blocked count: 238 > Waited count: 617 > Waiting on > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@131f139b > Stack: > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1299) >
So Vincent, the servers are quiet? Which would match your low CPU observation. Clients are unable to send them load for some reason? How many disks. What is your block cache hit number (see regionserver log -- it gets printed every so often .... or in the below I see 99% so your numbers should be good coming out of the regionserver). > > > Here is some extract for one of our RS (this is similar to all of them): > > requestsPerSecond=593, numberOfOnlineRegions=584, numberOfStores=1147, > numberOfStorefiles=1980, storefileIndexSizeMB=15, rootIndexSizeKB=16219, > totalStaticIndexSizeKB=246127, totalStaticBloomSizeKB=12936, > memstoreSizeMB=1421, readRequestsCount=633241097, > writeRequestsCount=9375846, compactionQueueSize=0, flushQueueSize=0, > usedHeapMB=3042, maxHeapMB=4591, blockCacheSizeMB=890.19, > blockCacheFreeMB=257.65, blockCacheCount=14048, > blockCacheHitCount=5854936149, blockCacheMissCount=14761288, > blockCacheEvictedCount=4870523, blockCacheHitRatio=99%, > blockCacheHitCachingRatio=99%, hdfsBlocksLocalityIndex=29 > 600 regions is a lot per server. You should put it on your TODO list to have less per server -- bigger regions which you can do now you are on 0.92. If you major compact -- do it when site is less heavily loaded -- does our performance go up. Are all query types slow or just certain types? St.Ack
