Answers below, prefixed by "geoff:"
-----Original Message-----
From: saint....@gmail.com [mailto:saint....@gmail.com] On Behalf Of Stack
Sent: Tuesday, April 20, 2010 11:23 AM
To: hbase-user@hadoop.apache.org
Subject: Re: extremely sluggish hbase
On Tue, Apr 20, 2010 at 10:29 AM, Geoff Hendrey <ghend...@decarta.com> wrote:
> Hbase shell is taking 63 seconds to scan a table with {LIMIT=>1}!
Is MR job running concurrently?
Geoff: no
Whats happening on your servers? High load?
Geoff: no, 99% idle on both servers
I see
> this error occur frequently in the region server logs. Any ideas on
> what this might be>
>
> 2010-04-20 04:19:41,401 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server handler 2 on 60020, call next(-750587486574522252) from
> 10.241.6.80:51850: error:
> org.apache.hadoop.hbase.UnknownScannerException: Name:
> -750587486574522252
>
> I also see this in the regions server logs:
>
> 2010-04-20 04:21:44,559 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
> 5849633296569445699 lease expired
> 2010-04-20 04:21:44,560 INFO org.apache.hadoop.hdfs.DFSClient: Could
> not obtain block blk_1799401938583830364_69702 from any node:
> java.io.IOException: No live nodes contain current block
>
So, this is usually because the client took long between 'next'
invocations on the scanner or the server is under such load its holding on to
the 'next' call for so long that the next time 'next' is called, the scanner
lease has expired.
> However "hadoop dfsadmin -report" doesn't show any HDFS issues. Looks
> totally healthy. When I do "status" from HBase shell I get
> "hbase(main):008:0> status
> 2 servers, 0 dead, 484.0000 average load" which also seems healthy to
> me.
>
Your servers are carrying 500 regions each.
Geoff: Is this high, moderate, or low for a typical installation?
> Any suggestions?
>
Look at top. Look for loading. Are you swapping?
Geoff: I will look into the swapping and see if I can get some numbers.
Look in hbase logs. Whats it say its doing? Fat GC pauses?
Geoff: I monitor all the logs and I don't see any GC pauses. I am running 64
bit java with 8GB of heap. I'll look into GC further and see if I can get some
concrete data.
St.Ack
