hbase version: 0.98.10
when a rs died, master will use ServerShutDownHanlder to process this dead rs
in ServerShutDownHandler， will call RegionStates.serverOffline to get all
regions need to reAssign
in method RegionStates.serverOffline, line 538
LOG.warn("THIS SHOULD NOT HANPPEN......");
but, there is a scenario can make line 538 happen and leading to region in RIT
setps to reproduce this problem:
1. hmaster asks rs1 to open region1
2. rs1 opens region1 successfully
3. hmaster use method "handleRegion" to process RS_ZK_REGION_OPEN zk-event
such as : delete ZNode, update RIT, make region to online state, update
lastAssignment
4. before hmaster process node-deleted event, rs1 died
then hmaster will skip the following steps to make region online
5. hmaster submit serverShutDownHanlder to process dead rs1
find region1 in RIT with online state, and the serverName in RIT equals the
dead server
then, in method RegionStates.serverOffline, line 538
LOG.warn("THIS SHOULD NOT HANPPEN......"); will happen
finally, region1 will stay in RIT. unless we restart hmaster to refresh
hmaster's memory
