The slow response took about 1.5 minutes. During this period, did you observe high latency ?
If you have Ganglia installed on master / NN node, do you observe abnormal spike ? BTW did you presplit your table ? Thanks On Fri, Feb 15, 2013 at 7:14 PM, Viral Bajaria <[email protected]>wrote: > Hi, > > (using hbase 0.94.4 and hadoop 1.0.4) > > I have been seeing a lot of the following WARN in my logs: > > 2013-02-16 02:37:11,409 DEBUG > org.apache.hadoop.hbase.io.hfile.LruBlockCache: Stats: total=25.18 MB, > free=2.97 GB, max=3 GB, blocks=1, accesses=52, hits=51, hitRatio=98.07%, , > cachingAccesses=52, cachingHits=51, cachingHitsRatio=98.07%, , evictions=0, > evicted=0, evictedPerRun=NaN > 2013-02-16 02:37:33,368 WARN org.apache.hadoop.ipc.HBaseServer: > (responseTooSlow): > > {"processingtimems":97509,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@1c3308bd > ), > rpc version=1, client version=29, > methodsFingerPrint=-1368823753","client":" > 10.149.10.10:41009 > > ","starttimems":1360982155855,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} > 2013-02-16 02:38:37,377 WARN org.apache.hadoop.ipc.HBaseServer: > (responseTooSlow): > > {"processingtimems":97191,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@3eafc7ae > ), > rpc version=1, client version=29, > methodsFingerPrint=-1368823753","client":" > 10.149.10.10:41014 > > ","starttimems":1360982220183,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} > 2013-02-16 02:39:29,842 WARN org.apache.hadoop.ipc.HBaseServer: > (responseTooSlow): > > {"processingtimems":85300,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@3d615428 > ), > rpc version=1, client version=29, > methodsFingerPrint=-1368823753","client":" > 10.149.10.10:41017 > > ","starttimems":1360982284538,"queuetimems":1,"class":"HRegionServer","responsesize":0,"method":"multi"} > > It's strange because this is a new hbase setup with almost no traffic on > it. I am running a perf test and would not expect this to happen. The > regionservers have 12GB heap space and are only using 1GB when that error > happens. I just pushed close to 33K rows via a batch and I see the > responseTooSlow. > > I enabled GC logging, but I don't see any GC lockups, and each GC attempt > is only taking a few 100 ms. > > What else could be happening here, any pointers on debugging ? My setup is > 1 Master running with 1 NN (on the same server) with 3 regionservers > running alongside the datanodes. > > Thanks, > Viral >
