[
https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13183915#comment-13183915
]
chunhui shen commented on HBASE-5179:
-------------------------------------
Master logs, Let's see the region a04d0ac0a360e8cf5edf74af4ce64b16.
{code}
2011-12-30 02:20:05,285 INFO org.apache.hadoop.hbase.master.HMaster: Master
startup proceeding: master failover
2011-12-30 02:20:06,779 INFO org.apache.hadoop.hbase.master.ServerManager:
Server start rejected; we already have dw83.kgb.sqa.cm4:60020 registered;
existingServer=serverName=dw83.kgb.sqa.cm4,60020,1325180976942,
load=(requests=0, regions=7, usedHeap=10831, maxHeap=15872),
newServer=serverName=dw83.kgb.sqa.cm4,60020,1325182806080, load=(requests=0,
regions=0, usedHeap=230, maxHeap=15872)
2011-12-30 02:20:06,779 INFO org.apache.hadoop.hbase.master.ServerManager:
Triggering server recovery; existingServer dw83.kgb.sqa.cm4,60020,1325180976942
looks stale
2011-12-30 02:20:06,780 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
based on AM, current region=-ROOT-,,0.70236052 is on
server=serverName=dw80.kgb.sqa.cm4,60020,1325180470774, load=(requests=0,
regions=0, usedHeap=0, maxHeap=0) server being checked:
dw83.kgb.sqa.cm4,60020,1325180976942
2011-12-30 02:20:06,780 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
based on AM, current region=.META.,,1.1028785192 is on
server=serverName=dw80.kgb.sqa.cm4,60020,1325180470774, load=(requests=0,
regions=0, usedHeap=0, maxHeap=0) server being checked:
dw83.kgb.sqa.cm4,60020,1325180976942
2011-12-30 02:20:06,781 DEBUG org.apache.hadoop.hbase.master.ServerManager:
Added=dw83.kgb.sqa.cm4,60020,1325180976942 to dead servers, submitted shutdown
handler to be executed, root=false, meta=false
2011-12-30 02:20:07,839 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
path=hdfs://dw74.kgb.sqa.cm4:9000/hbase-common/writetest/a04d0ac0a360e8cf5edf74af4ce64b16/recovered.edits/0000000000965355783.temp
region=a04d0ac0a360e8cf5edf74af4ce64b16
2011-12-30 02:20:08,965 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:60000-0x134784f727b0543 Creating (or updating) unassigned node for
a04d0ac0a360e8cf5edf74af4ce64b16 with OFFLINE state
2011-12-30 02:20:08,988 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Failed-over master needs to process 14 regions in transition
2011-12-30 02:20:09,017 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Processing region
writetest,B8VCH6I7EP0SLJA6KU8VTE75RCCZMZ14GTBRSTC7QOW9L2Q818R1O4PLA9ZX64JD5ZZTSAK021NUYUUHJ0BS9NTTCQ09PBRZMZPL,1325179237366.a04d0ac0a360e8cf5edf74af4ce64b16.
in state M_ZK_REGION_OFFLINE
2011-12-30 02:20:09,017 DEBUG
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED
event for a04d0ac0a360e8cf5edf74af4ce64b16
2011-12-30 02:20:09,017 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Forcing OFFLINE;
was=writetest,B8VCH6I7EP0SLJA6KU8VTE75RCCZMZ14GTBRSTC7QOW9L2Q818R1O4PLA9ZX64JD5ZZTSAK021NUYUUHJ0BS9NTTCQ09PBRZMZPL,1325179237366.a04d0ac0a360e8cf5edf74af4ce64b16.
state=OFFLINE, ts=1325182808966
2011-12-30 02:20:09,020 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Assigning region
writetest,B8VCH6I7EP0SLJA6KU8VTE75RCCZMZ14GTBRSTC7QOW9L2Q818R1O4PLA9ZX64JD5ZZTSAK021NUYUUHJ0BS9NTTCQ09PBRZMZPL,1325179237366.a04d0ac0a360e8cf5edf74af4ce64b16.
to dw81.kgb.sqa.cm4,60020,1325181205124
2011-12-30 02:20:09,365 DEBUG
org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: Opened region
writetest,B8VCH6I7EP0SLJA6KU8VTE75RCCZMZ14GTBRSTC7QOW9L2Q818R1O4PLA9ZX64JD5ZZTSAK021NUYUUHJ0BS9NTTCQ09PBRZMZPL,1325179237366.a04d0ac0a360e8cf5edf74af4ce64b16.
on dw81.kgb.sqa.cm4,60020,1325181205124
2011-12-30 02:20:20,144 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
hdfs://dw74.kgb.sqa.cm4:9000/hbase-common/writetest/a04d0ac0a360e8cf5edf74af4ce64b16/recovered.edits/0000000000965355783.temp
(wrote 146434 edits in 1761ms)
{code}
> Concurrent processing of processFaileOver and ServerShutdownHandler may
> cause region is assigned before completing split log, it would cause data loss
> -------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-5179
> URL: https://issues.apache.org/jira/browse/HBASE-5179
> Project: HBase
> Issue Type: Bug
> Components: master
> Reporter: chunhui shen
> Assignee: chunhui shen
>
> If master's processing its failover and ServerShutdownHandler's processing
> happen concurrently, it may appear following case.
> 1.master completing splitLogAfterStartup()
> 2.RegionserverA restarts, and ServerShutdownHandler is processing.
> 3.master starts to rebuildUserRegions, and RegionserverA is considered as
> dead server.
> 4.master starts to assign regions of RegionserverA because it is a dead
> server by step3.
> However, when doing step4(assigning region), ServerShutdownHandler may be
> doing split log, Therefore, it may cause data loss.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
