[
https://issues.apache.org/jira/browse/HBASE-13845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14579569#comment-14579569
]
Hudson commented on HBASE-13845:
--------------------------------
FAILURE: Integrated in HBase-TRUNK #6556 (See
[https://builds.apache.org/job/HBase-TRUNK/6556/])
HBASE-13845 Expire of one region server carrying meta can bring down the
master: test case (jerryjch: rev 14fe23254a78c52bdaef0da819268c8b405059cb)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMetaShutdownHandler.java
> Expire of one region server carrying meta can bring down the master
> -------------------------------------------------------------------
>
> Key: HBASE-13845
> URL: https://issues.apache.org/jira/browse/HBASE-13845
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 2.0.0, 1.1.0, 1.2.0
> Reporter: Jerry He
> Assignee: Jerry He
> Fix For: 2.0.0, 1.2.0, 1.1.1
>
> Attachments: HBASE-13845-branch-1.1.patch,
> HBASE-13845-branch-1.patch, HBASE-13845-master-test-case-only.patch
>
>
> There seems to be a code bug that can cause expiration of one region server
> carrying meta to bring down the master under certain case.
> Here is the sequence of event.
> a) The master detects the expiration of a region server on ZK, and starts to
> expire the region server.
> b) Since the failed region server carries meta, the shutdown handler will
> call verifyAndAssignMetaWithRetries() during processing the expired rs.
> c) In verifyAndAssignMeta(), there is a logic to verifyMetaRegionLocation
> {code}
> (!server.getMetaTableLocator().verifyMetaRegionLocation(server.getConnection(),
> this.server.getZooKeeper(), timeout)) {
> this.services.getAssignmentManager().assignMeta
> (HRegionInfo.FIRST_META_REGIONINFO);
> } else if
> (serverName.equals(server.getMetaTableLocator().getMetaRegionLocation(
> this.server.getZooKeeper()))) {
> throw new IOException("hbase:meta is onlined on the dead server "
> + serverName);
> {code}
> If we see the meta region is still alive on the expired rs, we throw an
> exception.
> We do some retries (default 10x1000ms) for verifyAndAssignMeta.
> If we still get the exception after retries, we abort the master.
> {code}
> 2015-05-27 06:58:30,156 FATAL
> [MASTER_META_SERVER_OPERATIONS-bdvs1163:60000-0] master.HMaster: Master
> server abort: loaded coprocessors are: []
> 2015-05-27 06:58:30,156 FATAL
> [MASTER_META_SERVER_OPERATIONS-bdvs1163:60000-0] master.HMaster:
> verifyAndAssignMeta failed after10 times retries, aborting
> java.io.IOException: hbase:meta is onlined on the dead server
> bdvs1164.svl.ibm.com,16020,1432681743203
> at
> org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler.verifyAndAssignMeta(MetaServerShutdownHandler.java:162)
> at
> org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler.verifyAndAssignMetaWithRetries(MetaServerShutdownHandler.java:184)
> at
> org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler.process(MetaServerShutdownHandler.java:93)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 2015-05-27 06:58:30,156 INFO
> [MASTER_META_SERVER_OPERATIONS-bdvs1163:60000-0] regionserver.HRegionServer:
> STOPPED: verifyAndAssignMeta failed after10 times retries, aborting
> {code}
> The problem happens when the expired is slow processing its own expiration or
> has a slow death, and is still able to respond to master's meta verification
> in the meantime
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
