[
https://issues.apache.org/jira/browse/HBASE-14664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Samir Ahmic resolved HBASE-14664.
---------------------------------
Resolution: Fixed
Slow failover detection time and staled values in zk was reason for this issue.
This was resolved in HBASE-14861.
> Master failover issue: Backup master is unable to start if active master is
> killed and started in short time interval
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-14664
> URL: https://issues.apache.org/jira/browse/HBASE-14664
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 2.0.0
> Reporter: Samir Ahmic
> Assignee: Samir Ahmic
> Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-14664.patch, HBASE-14664.patch
>
>
> I notice this issue while running IntegrationTestDDLMasterFailover, it can be
> simply reproduced by executing this on active master (tested on two masters +
> 3rs cluster setup)
> {code}
> $ kill -9 master_pid; hbase-daemon.sh start master
> {code}
> Logs show that new active master is trying to locate hbase:meta table on
> restarted active master
> {code}
> 2015-10-21 19:28:20,804 INFO [hnode2:16000.activeMasterManager]
> zookeeper.MetaTableLocator: Failed verification of hbase:meta,,1 at
> address=hnode1,16000,1445447051681,
> exception=org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is
> not running yet
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1092)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionInfo(RSRpcServices.java:1330)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.getRegionInfo(MasterRpcServices.java:1525)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22233)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2136)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> at java.lang.Thread.run(Thread.java:745)
> 2015-10-21 19:28:20,805 INFO [hnode2:16000.activeMasterManager]
> master.HMaster: Meta was in transition on hnode1,16000,1445447051681
> 2015-10-21 19:28:20,805 INFO [hnode2:16000.activeMasterManager]
> master.AssignmentManager: Processing {1588230740 state=OPEN,
> ts=1445448500598, server=hnode1,16000,1445447051681
> {code}
> and because of above master is unable to read hbase:meta table:
> {code}
> 2015-10-21 19:28:49,429 INFO [hconnection-0x6e9cebcc-shared--pool6-t1]
> client.AsyncProcess: #2, table=hbase:meta, attempt=10/351 failed=1ops, last
> exception: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException:
> org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not
> running yet
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1092)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2083)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32462)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2136)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> which cause master is unable to complete start.
> I have also notices that in this case value of /hbase/meta-region-server
> znode is always pointing on restarted active master (hnode1 in my cluster ).
> I was able to workaround this issue by repeating same scenario with following:
> {code}
> $ kill -9 master_pid; hbase zkcli rmr /hbase/meta-region-server;
> hbase-daemon.sh start master
> {code}
> So issue is probably caused by staled value in /hbase/meta-region-server
> znode. I will try to create patch based on above.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
