[
https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189641#comment-13189641
]
gaojinchao commented on HBASE-5179:
-----------------------------------
@chunhui
The first test case failed, we start a cluster, the patch is split a new region
server's Hlog.
2012-01-20 00:34:39,462 INFO org.mortbay.log: Started
[email protected]:20010
2012-01-20 00:34:39,462 DEBUG org.apache.hadoop.hbase.master.HMaster: Started
service threads
2012-01-20 00:34:40,158 INFO org.apache.hadoop.hbase.master.ServerManager:
Registering server=C3S32,20020,1327037679721, regionCount=0, userLoad=false
2012-01-20 00:34:40,296 INFO org.apache.hadoop.hbase.master.ServerManager:
Registering server=C3S33,20020,1327037679059, regionCount=0, userLoad=false
2012-01-20 00:34:40,488 INFO org.apache.hadoop.hbase.master.ServerManager:
Registering server=C3S31,20020,1327037679673, regionCount=0, userLoad=false
2012-01-20 00:34:40,962 INFO org.apache.hadoop.hbase.master.ServerManager:
Waiting on regionserver(s) count to settle; currently=3
2012-01-20 00:34:42,462 INFO org.apache.hadoop.hbase.master.ServerManager:
Finished waiting for regionserver count to settle; count=3, sleptFor=3000
2012-01-20 00:34:42,463 INFO org.apache.hadoop.hbase.master.ServerManager:
Exiting wait on regionserver(s) to checkin; count=3, stopped=false, count of
regions out on cluster=0
2012-01-20 00:34:42,463 INFO org.apache.hadoop.hbase.master.HMaster:
------------------sleep 60s-----------------
2012-01-20 00:35:42,469 INFO org.apache.hadoop.hbase.master.MasterFileSystem:
Log folder hdfs://C3S31:9000/hbase/.logs/C3S31,20020,1327037679673 belongs to
an existing region server
2012-01-20 00:35:42,470 INFO org.apache.hadoop.hbase.master.MasterFileSystem:
Log folder hdfs://C3S31:9000/hbase/.logs/C3S32,20020,1327037679721 belongs to
an existing region server
2012-01-20 00:35:42,470 INFO org.apache.hadoop.hbase.master.MasterFileSystem:
Log folder hdfs://C3S31:9000/hbase/.logs/C3S33,20020,1327037679059 belongs to
an existing region server
2012-01-20 00:35:42,504 INFO org.apache.hadoop.hbase.catalog.CatalogTracker:
Failed verification of -ROOT-,,0 at address=C3S32:20020;
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: Region is not online:
-ROOT-,,0
2012-01-20 00:36:42,610 FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled
exception. Starting shutdown.
java.lang.RuntimeException: Timed out waiting to finish splitting log for
C3S32,20020,1327037679721
at
org.apache.hadoop.hbase.master.HMaster.waitUntilNoLogDir(HMaster.java:578)
at
org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:478)
at
org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:422)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:283)
2012-01-20 00:36:42,613 INFO org.apache.hadoop.hbase.master.HMaster: Aborting
2012-01-20 00:36:42,613 DEBUG org.apache.hadoop.hbase.master.HMaster: Stopping
service threads
2012-01-20 00:36:42,613 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server
on 20000
> Concurrent processing of processFaileOver and ServerShutdownHandler may cause
> region to be assigned before log splitting is completed, causing data loss
> --------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-5179
> URL: https://issues.apache.org/jira/browse/HBASE-5179
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.90.2
> Reporter: chunhui shen
> Assignee: chunhui shen
> Priority: Critical
> Fix For: 0.92.0, 0.94.0, 0.90.6
>
> Attachments: 5179-90.txt, 5179-90v10.patch, 5179-90v11.patch,
> 5179-90v12.patch, 5179-90v13.txt, 5179-90v14.patch, 5179-90v15.patch,
> 5179-90v2.patch, 5179-90v3.patch, 5179-90v4.patch, 5179-90v5.patch,
> 5179-90v6.patch, 5179-90v7.patch, 5179-90v8.patch, 5179-90v9.patch,
> 5179-v11-92.txt, 5179-v11.txt, 5179-v2.txt, 5179-v3.txt, 5179-v4.txt,
> Errorlog, hbase-5179.patch, hbase-5179v10.patch, hbase-5179v12.patch,
> hbase-5179v5.patch, hbase-5179v6.patch, hbase-5179v7.patch,
> hbase-5179v8.patch, hbase-5179v9.patch
>
>
> If master's processing its failover and ServerShutdownHandler's processing
> happen concurrently, it may appear following case.
> 1.master completed splitLogAfterStartup()
> 2.RegionserverA restarts, and ServerShutdownHandler is processing.
> 3.master starts to rebuildUserRegions, and RegionserverA is considered as
> dead server.
> 4.master starts to assign regions of RegionserverA because it is a dead
> server by step3.
> However, when doing step4(assigning region), ServerShutdownHandler may be
> doing split log, Therefore, it may cause data loss.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
