Pankaj Kumar created HBASE-14207:
------------------------------------
Summary: Region was hijacked and remained in transition when RS
failed to open a region and later regionplan changed to new RS on retry
Key: HBASE-14207
URL: https://issues.apache.org/jira/browse/HBASE-14207
Project: HBase
Issue Type: Bug
Components: master
Reporter: Pankaj Kumar
Assignee: Pankaj Kumar
On production environment, following events happened
1. Master is trying to assign a region to RS, but due to
KeeperException$SessionExpiredException RS failed to open the region.
In RS log, saw multiple WARN log related to
KeeperException$SessionExpiredException
> KeeperErrorCode = Session expired for
/hbase/region-in-transition/08f1935d652e5dbdac09b423b8f9401b
> Unable to get data of znode
/hbase/region-in-transition/08f1935d652e5dbdac09b423b8f9401b
2. Master retried to assign the region to same RS, but RS again failed.
3. On second retry new plan formed and this time plan destination (RS) is
different, so master send the request to new RS to open the region. But new RS
failed to open the region as there was server mismatch in ZNODE than the
expected current server name.
Logs Snippet:
{noformat}
HM
2015-07-14 03:50:29,759 | INFO | master:T101PC03VM13:21300 | Processing
08f1935d652e5dbdac09b423b8f9401b in state: M_ZK_REGION_OFFLINE |
org.apache.hadoop.hbase.master.AssignmentManager.processRegionsInTransition(AssignmentManager.java:644)
2015-07-14 03:50:29,759 | INFO | master:T101PC03VM13:21300 | Transitioned
{08f1935d652e5dbdac09b423b8f9401b state=OFFLINE, ts=1436817029679, server=null}
to {08f1935d652e5dbdac09b423b8f9401b state=PENDING_OPEN, ts=1436817029759,
server=T101PC03VM13,21302,1436816690692} |
org.apache.hadoop.hbase.master.RegionStates.updateRegionState(RegionStates.java:327)
2015-07-14 03:50:29,760 | INFO | master:T101PC03VM13:21300 | Processed region
08f1935d652e5dbdac09b423b8f9401b in state M_ZK_REGION_OFFLINE, on server:
T101PC03VM13,21302,1436816690692 |
org.apache.hadoop.hbase.master.AssignmentManager.processRegionsInTransition(AssignmentManager.java:768)
2015-07-14 03:50:29,800 | INFO | MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3
| Assigning
INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
T101PC03VM13,21302,1436816690692 |
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1983)
2015-07-14 03:50:29,801 | WARN | MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3
| Failed assignment of
INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
T101PC03VM13,21302,1436816690692, trying to assign elsewhere instead; try=1 of
10 |
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2077)
2015-07-14 03:50:29,802 | INFO | MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3
| Trying to re-assign
INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
the same failed server. |
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2123)
2015-07-14 03:50:31,804 | INFO | MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3
| Assigning
INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
T101PC03VM13,21302,1436816690692 |
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1983)
2015-07-14 03:50:31,806 | WARN | MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3
| Failed assignment of
INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
T101PC03VM13,21302,1436816690692, trying to assign elsewhere instead; try=2 of
10 |
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2077)
2015-07-14 03:50:31,807 | INFO | MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3
| Transitioned {08f1935d652e5dbdac09b423b8f9401b state=PENDING_OPEN,
ts=1436817031804, server=T101PC03VM13,21302,1436816690692} to
{08f1935d652e5dbdac09b423b8f9401b state=OFFLINE, ts=1436817031807,
server=T101PC03VM13,21302,1436816690692} |
org.apache.hadoop.hbase.master.RegionStates.updateRegionState(RegionStates.java:327)
2015-07-14 03:50:31,807 | INFO | MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3
| Assigning
INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
T101PC03VM14,21302,1436816997967 |
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1983)
2015-07-14 03:50:31,807 | INFO | MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3
| Transitioned {08f1935d652e5dbdac09b423b8f9401b state=OFFLINE,
ts=1436817031807, server=T101PC03VM13,21302,1436816690692} to
{08f1935d652e5dbdac09b423b8f9401b state=PENDING_OPEN, ts=1436817031807,
server=T101PC03VM14,21302,1436816997967} |
org.apache.hadoop.hbase.master.RegionStates.updateRegionState(RegionStates.java:327)
2015-07-14 03:51:09,501 | INFO | MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-4
| Skip assigning region in transition on other
server{08f1935d652e5dbdac09b423b8f9401b state=PENDING_OPEN, ts=1436817031807,
server=T101PC03VM14,21302,1436816997967} |
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:250)
{noformat}
{noformat}
RS - T101PC03VM14
2015-07-14 03:50:31,809 | INFO |
PriorityRpcServer.handler=2,queue=0,port=21302 | Open
INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. |
org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:3671)
2015-07-14 03:50:31,830 | WARN | RS_OPEN_REGION-T101PC03VM14:21302-2 |
regionserver:21302-0xe4e88f6f1b70002,
quorum=t101pc03vm12:24002,t101pc03vm13:24002,t101pc03vm14:24002,
baseZNode=/hbase Attempt to transition the unassigned node for
08f1935d652e5dbdac09b423b8f9401b from M_ZK_REGION_OFFLINE to
RS_ZK_REGION_OPENING failed, the server that tried to transition was
T101PC03VM14,21302,1436816997967 not the expected
T101PC03VM13,21302,1436816690692 |
org.apache.hadoop.hbase.zookeeper.ZKAssign.transitionNode(ZKAssign.java:875)
2015-07-14 03:50:31,830 | WARN | RS_OPEN_REGION-T101PC03VM14:21302-2 | Failed
transition from OFFLINE to OPENING for region=08f1935d652e5dbdac09b423b8f9401b
|
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.transitionZookeeperOfflineToOpening(OpenRegionHandler.java:539)
2015-07-14 03:50:31,831 | WARN | RS_OPEN_REGION-T101PC03VM14:21302-2 | Region
was hijacked? Opening cancelled for
encodedName=08f1935d652e5dbdac09b423b8f9401b |
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:132)
2015-07-14 03:50:31,831 | INFO | RS_OPEN_REGION-T101PC03VM14:21302-2 | Opening
of region {ENCODED => 08f1935d652e5dbdac09b423b8f9401b, NAME =>
'INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b.',
STARTKEY => '', ENDKEY => '200'} failed, transitioning from OFFLINE to
FAILED_OPEN in ZK, expecting version -1 |
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.tryTransitionFromOfflineToFailedOpen(OpenRegionHandler.java:436)
2015-07-14 03:50:31,834 | WARN | RS_OPEN_REGION-T101PC03VM14:21302-2 |
regionserver:21302-0xe4e88f6f1b70002,
quorum=t101pc03vm12:24002,t101pc03vm13:24002,t101pc03vm14:24002,
baseZNode=/hbase Attempt to transition the unassigned node for
08f1935d652e5dbdac09b423b8f9401b from M_ZK_REGION_OFFLINE to
RS_ZK_REGION_FAILED_OPEN failed, the server that tried to transition was
T101PC03VM14,21302,1436816997967 not the expected
T101PC03VM13,21302,1436816690692 |
org.apache.hadoop.hbase.zookeeper.ZKAssign.transitionNode(ZKAssign.java:875)
2015-07-14 03:50:31,834 | WARN | RS_OPEN_REGION-T101PC03VM14:21302-2 | Unable
to mark region {ENCODED => 08f1935d652e5dbdac09b423b8f9401b, NAME =>
'INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b.',
STARTKEY => '', ENDKEY => '200'} as FAILED_OPEN. It's likely that the master
already timed out this open attempt, and thus another RS already has the
region. |
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.tryTransitionFromOfflineToFailedOpen(OpenRegionHandler.java:444)
{noformat}
Since ZNODE was not modified with new destination server detail, so RS failed
to open it.
On Plan change we should set true to 'setOfflineInZK', so that ZNODE will be
modified with new destination server detail after resetting
'versionOfOfflineNode' to -1.
{code}
if (plan != newPlan &&
!plan.getDestination().equals(newPlan.getDestination())) {
// Clean out plan we failed execute and one that doesn't look like
it'll
// succeed anyways; we need a new plan!
// Transition back to OFFLINE
currentState = regionStates.updateRegionState(region,
State.OFFLINE);
versionOfOfflineNode = -1;
plan = newPlan;
}
{code}
Please correct me if I am wrong.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
