[
https://issues.apache.org/jira/browse/HBASE-14207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pankaj Kumar updated HBASE-14207:
---------------------------------
Description:
On production environment, following events happened
1. Master is trying to assign a region to RS, but due to
KeeperException$SessionExpiredException RS failed to open the region.
In RS log, saw multiple WARN log related to
KeeperException$SessionExpiredException
> KeeperErrorCode = Session expired for
/hbase/region-in-transition/08f1935d652e5dbdac09b423b8f9401b
> Unable to get data of znode
/hbase/region-in-transition/08f1935d652e5dbdac09b423b8f9401b
2. Master retried to assign the region to same RS, but RS again failed.
3. On second retry new plan formed and this time plan destination (RS) is
different, so master send the request to new RS to open the region. But new RS
failed to open the region as there was server mismatch in ZNODE than the
expected current server name.
Logs Snippet:
{noformat}
HM
2015-07-14 03:50:29,759 | INFO | master:T101PC03VM13:21300 | Processing
08f1935d652e5dbdac09b423b8f9401b in state: M_ZK_REGION_OFFLINE |
org.apache.hadoop.hbase.master.AssignmentManager.processRegionsInTransition(AssignmentManager.java:644)
2015-07-14 03:50:29,759 | INFO | master:T101PC03VM13:21300 | Transitioned
{08f1935d652e5dbdac09b423b8f9401b state=OFFLINE, ts=1436817029679, server=null}
to {08f1935d652e5dbdac09b423b8f9401b state=PENDING_OPEN, ts=1436817029759,
server=T101PC03VM13,21302,1436816690692} |
org.apache.hadoop.hbase.master.RegionStates.updateRegionState(RegionStates.java:327)
2015-07-14 03:50:29,760 | INFO | master:T101PC03VM13:21300 | Processed region
08f1935d652e5dbdac09b423b8f9401b in state M_ZK_REGION_OFFLINE, on server:
T101PC03VM13,21302,1436816690692 |
org.apache.hadoop.hbase.master.AssignmentManager.processRegionsInTransition(AssignmentManager.java:768)
2015-07-14 03:50:29,800 | INFO | MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3
| Assigning
INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
T101PC03VM13,21302,1436816690692 |
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1983)
2015-07-14 03:50:29,801 | WARN | MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3
| Failed assignment of
INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
T101PC03VM13,21302,1436816690692, trying to assign elsewhere instead; try=1 of
10 |
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2077)
2015-07-14 03:50:29,802 | INFO | MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3
| Trying to re-assign
INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
the same failed server. |
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2123)
2015-07-14 03:50:31,804 | INFO | MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3
| Assigning
INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
T101PC03VM13,21302,1436816690692 |
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1983)
2015-07-14 03:50:31,806 | WARN | MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3
| Failed assignment of
INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
T101PC03VM13,21302,1436816690692, trying to assign elsewhere instead; try=2 of
10 |
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2077)
2015-07-14 03:50:31,807 | INFO | MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3
| Transitioned {08f1935d652e5dbdac09b423b8f9401b state=PENDING_OPEN,
ts=1436817031804, server=T101PC03VM13,21302,1436816690692} to
{08f1935d652e5dbdac09b423b8f9401b state=OFFLINE, ts=1436817031807,
server=T101PC03VM13,21302,1436816690692} |
org.apache.hadoop.hbase.master.RegionStates.updateRegionState(RegionStates.java:327)
2015-07-14 03:50:31,807 | INFO | MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3
| Assigning
INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
T101PC03VM14,21302,1436816997967 |
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1983)
2015-07-14 03:50:31,807 | INFO | MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3
| Transitioned {08f1935d652e5dbdac09b423b8f9401b state=OFFLINE,
ts=1436817031807, server=T101PC03VM13,21302,1436816690692} to
{08f1935d652e5dbdac09b423b8f9401b state=PENDING_OPEN, ts=1436817031807,
server=T101PC03VM14,21302,1436816997967} |
org.apache.hadoop.hbase.master.RegionStates.updateRegionState(RegionStates.java:327)
2015-07-14 03:51:09,501 | INFO | MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-4
| Skip assigning region in transition on other
server{08f1935d652e5dbdac09b423b8f9401b state=PENDING_OPEN, ts=1436817031807,
server=T101PC03VM14,21302,1436816997967} |
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:250)
{noformat}
{noformat}
RS - T101PC03VM14
2015-07-14 03:50:31,809 | INFO |
PriorityRpcServer.handler=2,queue=0,port=21302 | Open
INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. |
org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:3671)
2015-07-14 03:50:31,830 | WARN | RS_OPEN_REGION-T101PC03VM14:21302-2 |
regionserver:21302-0xe4e88f6f1b70002,
quorum=t101pc03vm12:24002,t101pc03vm13:24002,t101pc03vm14:24002,
baseZNode=/hbase Attempt to transition the unassigned node for
08f1935d652e5dbdac09b423b8f9401b from M_ZK_REGION_OFFLINE to
RS_ZK_REGION_OPENING failed, the server that tried to transition was
T101PC03VM14,21302,1436816997967 not the expected
T101PC03VM13,21302,1436816690692 |
org.apache.hadoop.hbase.zookeeper.ZKAssign.transitionNode(ZKAssign.java:875)
2015-07-14 03:50:31,830 | WARN | RS_OPEN_REGION-T101PC03VM14:21302-2 | Failed
transition from OFFLINE to OPENING for region=08f1935d652e5dbdac09b423b8f9401b
|
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.transitionZookeeperOfflineToOpening(OpenRegionHandler.java:539)
2015-07-14 03:50:31,831 | WARN | RS_OPEN_REGION-T101PC03VM14:21302-2 | Region
was hijacked? Opening cancelled for
encodedName=08f1935d652e5dbdac09b423b8f9401b |
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:132)
2015-07-14 03:50:31,831 | INFO | RS_OPEN_REGION-T101PC03VM14:21302-2 | Opening
of region {ENCODED => 08f1935d652e5dbdac09b423b8f9401b, NAME =>
'INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b.',
STARTKEY => '', ENDKEY => '200'} failed, transitioning from OFFLINE to
FAILED_OPEN in ZK, expecting version -1 |
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.tryTransitionFromOfflineToFailedOpen(OpenRegionHandler.java:436)
2015-07-14 03:50:31,834 | WARN | RS_OPEN_REGION-T101PC03VM14:21302-2 |
regionserver:21302-0xe4e88f6f1b70002,
quorum=t101pc03vm12:24002,t101pc03vm13:24002,t101pc03vm14:24002,
baseZNode=/hbase Attempt to transition the unassigned node for
08f1935d652e5dbdac09b423b8f9401b from M_ZK_REGION_OFFLINE to
RS_ZK_REGION_FAILED_OPEN failed, the server that tried to transition was
T101PC03VM14,21302,1436816997967 not the expected
T101PC03VM13,21302,1436816690692 |
org.apache.hadoop.hbase.zookeeper.ZKAssign.transitionNode(ZKAssign.java:875)
2015-07-14 03:50:31,834 | WARN | RS_OPEN_REGION-T101PC03VM14:21302-2 | Unable
to mark region {ENCODED => 08f1935d652e5dbdac09b423b8f9401b, NAME =>
'INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b.',
STARTKEY => '', ENDKEY => '200'} as FAILED_OPEN. It's likely that the master
already timed out this open attempt, and thus another RS already has the
region. |
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.tryTransitionFromOfflineToFailedOpen(OpenRegionHandler.java:444)
{noformat}
was:
On production environment, following events happened
1. Master is trying to assign a region to RS, but due to
KeeperException$SessionExpiredException RS failed to open the region.
In RS log, saw multiple WARN log related to
KeeperException$SessionExpiredException
> KeeperErrorCode = Session expired for
/hbase/region-in-transition/08f1935d652e5dbdac09b423b8f9401b
> Unable to get data of znode
/hbase/region-in-transition/08f1935d652e5dbdac09b423b8f9401b
2. Master retried to assign the region to same RS, but RS again failed.
3. On second retry new plan formed and this time plan destination (RS) is
different, so master send the request to new RS to open the region. But new RS
failed to open the region as there was server mismatch in ZNODE than the
expected current server name.
Logs Snippet:
{noformat}
HM
2015-07-14 03:50:29,759 | INFO | master:T101PC03VM13:21300 | Processing
08f1935d652e5dbdac09b423b8f9401b in state: M_ZK_REGION_OFFLINE |
org.apache.hadoop.hbase.master.AssignmentManager.processRegionsInTransition(AssignmentManager.java:644)
2015-07-14 03:50:29,759 | INFO | master:T101PC03VM13:21300 | Transitioned
{08f1935d652e5dbdac09b423b8f9401b state=OFFLINE, ts=1436817029679, server=null}
to {08f1935d652e5dbdac09b423b8f9401b state=PENDING_OPEN, ts=1436817029759,
server=T101PC03VM13,21302,1436816690692} |
org.apache.hadoop.hbase.master.RegionStates.updateRegionState(RegionStates.java:327)
2015-07-14 03:50:29,760 | INFO | master:T101PC03VM13:21300 | Processed region
08f1935d652e5dbdac09b423b8f9401b in state M_ZK_REGION_OFFLINE, on server:
T101PC03VM13,21302,1436816690692 |
org.apache.hadoop.hbase.master.AssignmentManager.processRegionsInTransition(AssignmentManager.java:768)
2015-07-14 03:50:29,800 | INFO | MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3
| Assigning
INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
T101PC03VM13,21302,1436816690692 |
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1983)
2015-07-14 03:50:29,801 | WARN | MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3
| Failed assignment of
INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
T101PC03VM13,21302,1436816690692, trying to assign elsewhere instead; try=1 of
10 |
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2077)
2015-07-14 03:50:29,802 | INFO | MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3
| Trying to re-assign
INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
the same failed server. |
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2123)
2015-07-14 03:50:31,804 | INFO | MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3
| Assigning
INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
T101PC03VM13,21302,1436816690692 |
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1983)
2015-07-14 03:50:31,806 | WARN | MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3
| Failed assignment of
INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
T101PC03VM13,21302,1436816690692, trying to assign elsewhere instead; try=2 of
10 |
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2077)
2015-07-14 03:50:31,807 | INFO | MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3
| Transitioned {08f1935d652e5dbdac09b423b8f9401b state=PENDING_OPEN,
ts=1436817031804, server=T101PC03VM13,21302,1436816690692} to
{08f1935d652e5dbdac09b423b8f9401b state=OFFLINE, ts=1436817031807,
server=T101PC03VM13,21302,1436816690692} |
org.apache.hadoop.hbase.master.RegionStates.updateRegionState(RegionStates.java:327)
2015-07-14 03:50:31,807 | INFO | MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3
| Assigning
INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
T101PC03VM14,21302,1436816997967 |
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1983)
2015-07-14 03:50:31,807 | INFO | MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3
| Transitioned {08f1935d652e5dbdac09b423b8f9401b state=OFFLINE,
ts=1436817031807, server=T101PC03VM13,21302,1436816690692} to
{08f1935d652e5dbdac09b423b8f9401b state=PENDING_OPEN, ts=1436817031807,
server=T101PC03VM14,21302,1436816997967} |
org.apache.hadoop.hbase.master.RegionStates.updateRegionState(RegionStates.java:327)
2015-07-14 03:51:09,501 | INFO | MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-4
| Skip assigning region in transition on other
server{08f1935d652e5dbdac09b423b8f9401b state=PENDING_OPEN, ts=1436817031807,
server=T101PC03VM14,21302,1436816997967} |
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:250)
{noformat}
{noformat}
RS - T101PC03VM14
2015-07-14 03:50:31,809 | INFO |
PriorityRpcServer.handler=2,queue=0,port=21302 | Open
INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. |
org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:3671)
2015-07-14 03:50:31,830 | WARN | RS_OPEN_REGION-T101PC03VM14:21302-2 |
regionserver:21302-0xe4e88f6f1b70002,
quorum=t101pc03vm12:24002,t101pc03vm13:24002,t101pc03vm14:24002,
baseZNode=/hbase Attempt to transition the unassigned node for
08f1935d652e5dbdac09b423b8f9401b from M_ZK_REGION_OFFLINE to
RS_ZK_REGION_OPENING failed, the server that tried to transition was
T101PC03VM14,21302,1436816997967 not the expected
T101PC03VM13,21302,1436816690692 |
org.apache.hadoop.hbase.zookeeper.ZKAssign.transitionNode(ZKAssign.java:875)
2015-07-14 03:50:31,830 | WARN | RS_OPEN_REGION-T101PC03VM14:21302-2 | Failed
transition from OFFLINE to OPENING for region=08f1935d652e5dbdac09b423b8f9401b
|
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.transitionZookeeperOfflineToOpening(OpenRegionHandler.java:539)
2015-07-14 03:50:31,831 | WARN | RS_OPEN_REGION-T101PC03VM14:21302-2 | Region
was hijacked? Opening cancelled for
encodedName=08f1935d652e5dbdac09b423b8f9401b |
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:132)
2015-07-14 03:50:31,831 | INFO | RS_OPEN_REGION-T101PC03VM14:21302-2 | Opening
of region {ENCODED => 08f1935d652e5dbdac09b423b8f9401b, NAME =>
'INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b.',
STARTKEY => '', ENDKEY => '200'} failed, transitioning from OFFLINE to
FAILED_OPEN in ZK, expecting version -1 |
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.tryTransitionFromOfflineToFailedOpen(OpenRegionHandler.java:436)
2015-07-14 03:50:31,834 | WARN | RS_OPEN_REGION-T101PC03VM14:21302-2 |
regionserver:21302-0xe4e88f6f1b70002,
quorum=t101pc03vm12:24002,t101pc03vm13:24002,t101pc03vm14:24002,
baseZNode=/hbase Attempt to transition the unassigned node for
08f1935d652e5dbdac09b423b8f9401b from M_ZK_REGION_OFFLINE to
RS_ZK_REGION_FAILED_OPEN failed, the server that tried to transition was
T101PC03VM14,21302,1436816997967 not the expected
T101PC03VM13,21302,1436816690692 |
org.apache.hadoop.hbase.zookeeper.ZKAssign.transitionNode(ZKAssign.java:875)
2015-07-14 03:50:31,834 | WARN | RS_OPEN_REGION-T101PC03VM14:21302-2 | Unable
to mark region {ENCODED => 08f1935d652e5dbdac09b423b8f9401b, NAME =>
'INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b.',
STARTKEY => '', ENDKEY => '200'} as FAILED_OPEN. It's likely that the master
already timed out this open attempt, and thus another RS already has the
region. |
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.tryTransitionFromOfflineToFailedOpen(OpenRegionHandler.java:444)
{noformat}
Since ZNODE was not modified with new destination server detail, so RS failed
to open it.
On Plan change we should set true to 'setOfflineInZK', so that ZNODE will be
modified with new destination server detail after resetting
'versionOfOfflineNode' to -1.
{code}
if (plan != newPlan &&
!plan.getDestination().equals(newPlan.getDestination())) {
// Clean out plan we failed execute and one that doesn't look like
it'll
// succeed anyways; we need a new plan!
// Transition back to OFFLINE
currentState = regionStates.updateRegionState(region,
State.OFFLINE);
versionOfOfflineNode = -1;
plan = newPlan;
}
{code}
Please correct me if I am wrong.
> Region was hijacked and remained in transition when RS failed to open a
> region and later regionplan changed to new RS on retry
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-14207
> URL: https://issues.apache.org/jira/browse/HBASE-14207
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.98.6
> Reporter: Pankaj Kumar
> Assignee: Pankaj Kumar
> Priority: Critical
>
> On production environment, following events happened
> 1. Master is trying to assign a region to RS, but due to
> KeeperException$SessionExpiredException RS failed to open the region.
> In RS log, saw multiple WARN log related to
> KeeperException$SessionExpiredException
> > KeeperErrorCode = Session expired for
> /hbase/region-in-transition/08f1935d652e5dbdac09b423b8f9401b
> > Unable to get data of znode
> /hbase/region-in-transition/08f1935d652e5dbdac09b423b8f9401b
> 2. Master retried to assign the region to same RS, but RS again failed.
> 3. On second retry new plan formed and this time plan destination (RS) is
> different, so master send the request to new RS to open the region. But new
> RS failed to open the region as there was server mismatch in ZNODE than the
> expected current server name.
> Logs Snippet:
> {noformat}
> HM
> 2015-07-14 03:50:29,759 | INFO | master:T101PC03VM13:21300 | Processing
> 08f1935d652e5dbdac09b423b8f9401b in state: M_ZK_REGION_OFFLINE |
> org.apache.hadoop.hbase.master.AssignmentManager.processRegionsInTransition(AssignmentManager.java:644)
> 2015-07-14 03:50:29,759 | INFO | master:T101PC03VM13:21300 | Transitioned
> {08f1935d652e5dbdac09b423b8f9401b state=OFFLINE, ts=1436817029679,
> server=null} to {08f1935d652e5dbdac09b423b8f9401b state=PENDING_OPEN,
> ts=1436817029759, server=T101PC03VM13,21302,1436816690692} |
> org.apache.hadoop.hbase.master.RegionStates.updateRegionState(RegionStates.java:327)
> 2015-07-14 03:50:29,760 | INFO | master:T101PC03VM13:21300 | Processed
> region 08f1935d652e5dbdac09b423b8f9401b in state M_ZK_REGION_OFFLINE, on
> server: T101PC03VM13,21302,1436816690692 |
> org.apache.hadoop.hbase.master.AssignmentManager.processRegionsInTransition(AssignmentManager.java:768)
> 2015-07-14 03:50:29,800 | INFO |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Assigning
> INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
> T101PC03VM13,21302,1436816690692 |
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1983)
> 2015-07-14 03:50:29,801 | WARN |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Failed assignment of
> INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
> T101PC03VM13,21302,1436816690692, trying to assign elsewhere instead; try=1
> of 10 |
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2077)
> 2015-07-14 03:50:29,802 | INFO |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Trying to re-assign
> INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
> the same failed server. |
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2123)
> 2015-07-14 03:50:31,804 | INFO |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Assigning
> INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
> T101PC03VM13,21302,1436816690692 |
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1983)
> 2015-07-14 03:50:31,806 | WARN |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Failed assignment of
> INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
> T101PC03VM13,21302,1436816690692, trying to assign elsewhere instead; try=2
> of 10 |
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2077)
> 2015-07-14 03:50:31,807 | INFO |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Transitioned
> {08f1935d652e5dbdac09b423b8f9401b state=PENDING_OPEN, ts=1436817031804,
> server=T101PC03VM13,21302,1436816690692} to {08f1935d652e5dbdac09b423b8f9401b
> state=OFFLINE, ts=1436817031807, server=T101PC03VM13,21302,1436816690692} |
> org.apache.hadoop.hbase.master.RegionStates.updateRegionState(RegionStates.java:327)
> 2015-07-14 03:50:31,807 | INFO |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Assigning
> INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
> T101PC03VM14,21302,1436816997967 |
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1983)
> 2015-07-14 03:50:31,807 | INFO |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Transitioned
> {08f1935d652e5dbdac09b423b8f9401b state=OFFLINE, ts=1436817031807,
> server=T101PC03VM13,21302,1436816690692} to {08f1935d652e5dbdac09b423b8f9401b
> state=PENDING_OPEN, ts=1436817031807,
> server=T101PC03VM14,21302,1436816997967} |
> org.apache.hadoop.hbase.master.RegionStates.updateRegionState(RegionStates.java:327)
> 2015-07-14 03:51:09,501 | INFO |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-4 | Skip assigning region in
> transition on other server{08f1935d652e5dbdac09b423b8f9401b
> state=PENDING_OPEN, ts=1436817031807,
> server=T101PC03VM14,21302,1436816997967} |
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:250)
> {noformat}
> {noformat}
> RS - T101PC03VM14
> 2015-07-14 03:50:31,809 | INFO |
> PriorityRpcServer.handler=2,queue=0,port=21302 | Open
> INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. |
> org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:3671)
> 2015-07-14 03:50:31,830 | WARN | RS_OPEN_REGION-T101PC03VM14:21302-2 |
> regionserver:21302-0xe4e88f6f1b70002,
> quorum=t101pc03vm12:24002,t101pc03vm13:24002,t101pc03vm14:24002,
> baseZNode=/hbase Attempt to transition the unassigned node for
> 08f1935d652e5dbdac09b423b8f9401b from M_ZK_REGION_OFFLINE to
> RS_ZK_REGION_OPENING failed, the server that tried to transition was
> T101PC03VM14,21302,1436816997967 not the expected
> T101PC03VM13,21302,1436816690692 |
> org.apache.hadoop.hbase.zookeeper.ZKAssign.transitionNode(ZKAssign.java:875)
> 2015-07-14 03:50:31,830 | WARN | RS_OPEN_REGION-T101PC03VM14:21302-2 |
> Failed transition from OFFLINE to OPENING for
> region=08f1935d652e5dbdac09b423b8f9401b |
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.transitionZookeeperOfflineToOpening(OpenRegionHandler.java:539)
> 2015-07-14 03:50:31,831 | WARN | RS_OPEN_REGION-T101PC03VM14:21302-2 |
> Region was hijacked? Opening cancelled for
> encodedName=08f1935d652e5dbdac09b423b8f9401b |
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:132)
> 2015-07-14 03:50:31,831 | INFO | RS_OPEN_REGION-T101PC03VM14:21302-2 |
> Opening of region {ENCODED => 08f1935d652e5dbdac09b423b8f9401b, NAME =>
> 'INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b.',
> STARTKEY => '', ENDKEY => '200'} failed, transitioning from OFFLINE to
> FAILED_OPEN in ZK, expecting version -1 |
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.tryTransitionFromOfflineToFailedOpen(OpenRegionHandler.java:436)
> 2015-07-14 03:50:31,834 | WARN | RS_OPEN_REGION-T101PC03VM14:21302-2 |
> regionserver:21302-0xe4e88f6f1b70002,
> quorum=t101pc03vm12:24002,t101pc03vm13:24002,t101pc03vm14:24002,
> baseZNode=/hbase Attempt to transition the unassigned node for
> 08f1935d652e5dbdac09b423b8f9401b from M_ZK_REGION_OFFLINE to
> RS_ZK_REGION_FAILED_OPEN failed, the server that tried to transition was
> T101PC03VM14,21302,1436816997967 not the expected
> T101PC03VM13,21302,1436816690692 |
> org.apache.hadoop.hbase.zookeeper.ZKAssign.transitionNode(ZKAssign.java:875)
> 2015-07-14 03:50:31,834 | WARN | RS_OPEN_REGION-T101PC03VM14:21302-2 |
> Unable to mark region {ENCODED => 08f1935d652e5dbdac09b423b8f9401b, NAME =>
> 'INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b.',
> STARTKEY => '', ENDKEY => '200'} as FAILED_OPEN. It's likely that the master
> already timed out this open attempt, and thus another RS already has the
> region. |
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.tryTransitionFromOfflineToFailedOpen(OpenRegionHandler.java:444)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
