On Fri, Aug 13, 2010 at 3:59 PM, James Kennedy <[email protected]> wrote:
> We've recently updated the hbase-transactional-tableindexed extension to work
> with the latest 0.89.20100726 version of HBase (still to be pushed).
Good. Please update the list when you push out the change.
> All HBase tests are passing but then when we started to write our own and
> test true sudden HRegionServer death we ran into trouble.
> It seems that the HMaster does not recognize the kill even after many
> minutes. Client requests are blocked and the log continues to repeat the
> logs below.
>
> We realized that HBase's own tests that require RegionServer death use
> abort() and not kill() which does enough cleanup to inadequately simulate a
> sudden (e.g. JVM crash) death.
>
> As an experiment I made HRegionServer.kill() public and modified
> HBaseMiniCluster to call that from abort() instead. Now a test like
> TestMasterTransitions will exhibit similar behaviour: The HMaster never
> notices the RegionServer is gone.
>
Odd. I added kill. Its used in TestMasterTransistions currently.
Are you sure the master doesn't notice the server gone? Maybe the RS
is dead in all but the ZK Client session thread so its lease is not
timing out against the ZK ensemble?
The other issue that can get in the way of a RS recovery is replay of
logs; the master needs to grab the lease on the WAL log that the dead
regionserver was writing. This can take a little while. It won't
work at all if you are not using an hdfs that doesn't support
'append'; i.e. the hadoop that is in hbase lib dir or build your own
out of apache hadoop branch-0.20-append branch (or get cdh3b2). Is
this the issue (Tail the log and look for failure to obtain file
lease).
An incomplete server crash recovery will show the exceptions you pasted below.
> Could it really be that sudden region server death is not handled in hbase?
> Or more likely is this a failure of the testing framework to adequately
> simulate kill -9?
>
Its kinda hard to simulate kill -9 in unit tests. The 'kill' method
above was a pale attempt at it in our unit testing context. I've
manually kill -9'd nodes -- RS and DN -- to simulate node death
testing 0.89.x builds and all seems to be basically working.
On making it so master notices downed servers the faster so it'll
start up recovery the sooner, thats a case of playing with the zk
session timeout but coming in the other direction is tuning GC so GC
pause is not > ZK session expiration.
You also want to ensure that there is not too much of a backlog of WAL
logs else recovery which involves splitting all WALs that had
outstanding edits on the crashed server has less work to do when it
runs. Default is maximum backlog of 32 WALs. You might want to tune
this down though there is a currently an issue because it seems like
we can overrun this limit.
Hope this helps James,
St.Ack
> James Kennedy
> Project Manager
> Troove Inc.
>
>
> -------------------------------
>
> [13/08/10 15:12:12] 259494 [n.serverMonitor] INFO
> oop.hbase.master.ServerManager - 2 region servers, 0 dead, average load 3.5
> [13/08/10 15:12:12] 259560 [ger.metaScanner] INFO
> adoop.hbase.master.BaseScanner - RegionManager.metaScanner scanning meta
> region {server: 10.0.1.4:56908, regionname: .META.,,1.1028785192, startKey:
> <>}
> [13/08/10 15:12:12] 259561 [ger.metaScanner] WARN
> adoop.hbase.master.BaseScanner - Scan one META region: {server:
> 10.0.1.4:56908, regionname: .META.,,1.1028785192, startKey: <>}
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
> at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:309)
> at
> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:857)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:725)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:253)
> at $Proxy10.openScanner(Unknown Source)
> at
> org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:182)
> at
> org.apache.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.java:73)
> at
> org.apache.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.java:129)
> at
> org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:156)
> at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
>
>
>
>
