[
https://issues.apache.org/jira/browse/HBASE-5094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5094:
------------------------------------------
Description:
R1 is reassigned to RS3 during RS1 shutdown, even though R1 was just assigned
to RS2 by load balancer. So .META. table indicated R1 is on RS3. Both RS2 and
RS3 think they have R1. Later when RS3 shutdown, R1 is reassigned to RS2. RS2
will indicate ALREADY_OPENED. Thus the region is considered assigned to RS2
even though .META. indicates it is on RS3.
1) Region R1 - Assigned from RS1 to RS2.
2) RS1 goes down and ServerShutDownHandler. ServerShutDwonHandler finds RS1
with R1 from META as still META is not yet updated to RS2.
3) As RS1 goes down R1 is assigned from RS1 to RS3.
4) RS3 goes down. ServerShutdownHandler processes R1 and tries to assign it to
RS2.
5) RS2 says ALREADY_OPENED but META shows RS3.
I was able to reproduce the scenario.
was:
R1 is reassigned to RS3 during RS1 shutdown, even though R1 was just assigned
to RS2 by load balancer. So .META. table indicated R1 is on RS3. Both RS2 and
RS3 think they have R1. Later when RS3 shutdown, R1 is reassigned to RS2. RS2
will indicate ALREADY_OPENED. Thus the region is considered assigned to RS2
even though .META. indicates it is on RS3.
T1: Load balancer tried to move R1 from RS1 to RS2
. 2011-11-21 14:03:20,812 INFO org.apache.hadoop.hbase.master.HMaster: balance
hri=tableXY,\xB8Q\xEB\x85\x1E\xB8Q\xDF,1321573099841.ee2e205a60f1bb06cc73bc9df06289df.,
src=skynet-1,60020,1321912978281, dest=skynet-4,60020,1321912999305
T2: RS1 shutdown. 2011-11-21 14:03:24,759 DEBUG
org.apache.hadoop.hbase.master.ServerManager:
Added=skynet-1,60020,1321912978281 to dead servers, submitted shutdown handler
to be executed, root=false, meta=true
T3:R1 is opened on RS2. 2011-11-21 14:03:26,131 DEBUG
org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: The master has
opened the region
tableXY,\xB8Q\xEB\x85\x1E\xB8Q\xDF,1321573099841.ee2e205a60f1bb06cc73bc9df06289df.
that was online on skynet-4,60020,1321912999305
T4: As part of RS1 shutdown handling, region reassignment starts. It uses the
region location captured at T2. 2011-11-21 14:03:26,152 INFO
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Reassigning 32
region(s) that skynet-1,60020,1321912978281 was carrying (skipping 0 regions(s)
that are already in transition)
T5: R1 is assigned to RS3. 2011-11-21 14:03:27,404 DEBUG
org.apache.hadoop.hbase.zookeeper.ZKUtil: master:60000-0x133b84f9f490000
Retrieved 115 byte(s) of data from znode
/hbase/unassigned/ee2e205a60f1bb06cc73bc9df06289df;
data=region=tableXY,\xB8Q\xEB\x85\x1E\xB8Q\xDF,1321573099841.ee2e205a60f1bb06cc73bc9df06289df.,
origin=skynet-3,60020,1321912991430, state=RS_ZK_REGION_OPENED
T6: RS3 shutdown. R1 is reassigned to RS2. 2011-11-21 14:03:37,899 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: ALREADY_OPENED region
tableXY,\xB8Q\xEB\x85\x1E\xB8Q\xDF,1321573099841.ee2e205a60f1bb06cc73bc9df06289df.
to skynet-4,60020,1321912999305
>From AssignmentManager point of view, the R1 is assigned to RS2. The .META.
>table indicates the location is RS3.
> The META can hold an entry for a region with a different server name from the
> one actually in the AssignmentManager thus making the region inaccessible.
> --------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-5094
> URL: https://issues.apache.org/jira/browse/HBASE-5094
> Project: HBase
> Issue Type: Bug
> Reporter: ramkrishna.s.vasudevan
>
> R1 is reassigned to RS3 during RS1 shutdown, even though R1 was just assigned
> to RS2 by load balancer. So .META. table indicated R1 is on RS3. Both RS2 and
> RS3 think they have R1. Later when RS3 shutdown, R1 is reassigned to RS2. RS2
> will indicate ALREADY_OPENED. Thus the region is considered assigned to RS2
> even though .META. indicates it is on RS3.
> 1) Region R1 - Assigned from RS1 to RS2.
> 2) RS1 goes down and ServerShutDownHandler. ServerShutDwonHandler finds RS1
> with R1 from META as still META is not yet updated to RS2.
> 3) As RS1 goes down R1 is assigned from RS1 to RS3.
> 4) RS3 goes down. ServerShutdownHandler processes R1 and tries to assign it
> to RS2.
> 5) RS2 says ALREADY_OPENED but META shows RS3.
> I was able to reproduce the scenario.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
