[
https://issues.apache.org/jira/browse/HBASE-8912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13810240#comment-13810240
]
Sergey Kirichenko commented on HBASE-8912:
------------------------------------------
May be this helps (HBase from cloudera - 0.94.6-cdh4.4.0):
grep by region caused exception on master:
{noformat}
2013-10-31 00:07:52,871 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Region 3a476d37da81f620a3e53179d7d9192b has null regionLocation. But its table
table_x isn't in ENABLING state.
2013-10-31 00:07:53,057 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:60000-0x242045137a20070 Async create of unassigned node for
3a476d37da81f620a3e53179d7d9192b with OFFLINE state
2013-10-31 00:07:53,467 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager$CreateUnassignedAsyncCallback:
rs=table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
state=OFFLINE, ts=1383163673057, server=null, server=xxx100,60020,1383163665902
2013-10-31 00:07:53,495 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager$ExistsUnassignedAsyncCallback:
rs=table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
state=OFFLINE, ts=1383163673057, server=null
2013-10-31 00:07:54,834 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Handling transition=RS_ZK_REGION_OPENING, server=xxx100,60020,1383163665902,
region=3a476d37da81f620a3e53179d7d9192b
2013-10-31 00:07:56,953 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Handling transition=RS_ZK_REGION_FAILED_OPEN,
server=xxx100,60020,1383163665902, region=3a476d37da81f620a3e53179d7d9192b
2013-10-31 00:07:56,953 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Found an existing plan for
table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
destination server is xxx100,60020,1383163665902
2013-10-31 00:07:56,953 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
No previous transition plan was found (or we are ignoring an existing plan) for
table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
so generated a random one;
hri=table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.,
src=, dest=xxx108,60020,1383163666006; 9 (online=9, available=8) available
servers
2013-10-31 00:07:56,955 DEBUG
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED
event for 3a476d37da81f620a3e53179d7d9192b
2013-10-31 00:07:56,956 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Forcing OFFLINE;
was=table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
state=CLOSED, ts=1383163675624, server=xxx100,60020,1383163665902
2013-10-31 00:07:56,956 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:60000-0x242045137a20070 Creating (or updating) unassigned node for
3a476d37da81f620a3e53179d7d9192b with OFFLINE state
2013-10-31 00:07:57,003 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Found an existing plan for
table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
destination server is xxx108,60020,1383163666006
2013-10-31 00:07:57,003 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Using pre-existing plan for region
table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.;
plan=hri=table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.,
src=, dest=xxx108,60020,1383163666006
2013-10-31 00:07:57,003 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Assigning region
table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
to xxx108,60020,1383163666006
2013-10-31 00:07:58,545 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Handling transition=RS_ZK_REGION_FAILED_OPEN,
server=xxx108,60020,1383163666006, region=3a476d37da81f620a3e53179d7d9192b
2013-10-31 00:07:58,545 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Found an existing plan for
table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
destination server is xxx108,60020,1383163666006
2013-10-31 00:07:58,545 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
No previous transition plan was found (or we are ignoring an existing plan) for
table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
so generated a random one;
hri=table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.,
src=, dest=xxx106,60020,1383163666003; 9 (online=9, available=8) available
servers
2013-10-31 00:07:58,546 DEBUG
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED
event for 3a476d37da81f620a3e53179d7d9192b
2013-10-31 00:07:58,546 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Forcing OFFLINE;
was=table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
state=CLOSED, ts=1383163677110, server=xxx108,60020,1383163666006
2013-10-31 00:07:58,546 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:60000-0x242045137a20070 Creating (or updating) unassigned node for
3a476d37da81f620a3e53179d7d9192b with OFFLINE state
2013-10-31 00:07:58,553 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Handling transition=RS_ZK_REGION_FAILED_OPEN,
server=xxx108,60020,1383163666006, region=3a476d37da81f620a3e53179d7d9192b
2013-10-31 00:07:58,554 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Found an existing plan for
table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
destination server is xxx106,60020,1383163666003
2013-10-31 00:07:58,554 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
No previous transition plan was found (or we are ignoring an existing plan) for
table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
so generated a random one;
hri=table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.,
src=, dest=xxx104,60020,1383163665976; 9 (online=9, available=8) available
servers
2013-10-31 00:07:58,554 DEBUG
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED
event for 3a476d37da81f620a3e53179d7d9192b
2013-10-31 00:07:58,554 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Forcing OFFLINE;
was=table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
state=CLOSED, ts=1383163677110, server=xxx108,60020,1383163666006
2013-10-31 00:07:58,571 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Found an existing plan for
table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
destination server is xxx104,60020,1383163665976
2013-10-31 00:07:58,571 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Using pre-existing plan for region
table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.;
plan=hri=table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.,
src=, dest=xxx104,60020,1383163665976
2013-10-31 00:07:58,571 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Assigning region
table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
to xxx104,60020,1383163665976
2013-10-31 00:07:58,595 FATAL org.apache.hadoop.hbase.master.HMaster:
Unexpected state :
table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
state=PENDING_OPEN, ts=1383163678594, server=xxx104,60020,1383163665976 ..
Cannot transit it to OFFLINE.
java.lang.IllegalStateException: Unexpected state :
table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
state=PENDING_OPEN, ts=1383163678594, server=xxx104,60020,1383163665976 ..
Cannot transit it to OFFLINE.
at
org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1831)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1661)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1426)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1398)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1393)
at
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
{noformat}
grep by region caused exception on xxx100:
{noformat}
2013-10-31 00:07:54,000 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open
region:
table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
2013-10-31 00:07:54,000 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
regionserver:60020-0x242045137a20071 Attempting to transition node
3a476d37da81f620a3e53179d7d9192b from M_ZK_REGION_OFFLINE to
RS_ZK_REGION_OPENING
2013-10-31 00:07:54,029 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
regionserver:60020-0x242045137a20071 Successfully transitioned node
3a476d37da81f620a3e53179d7d9192b from M_ZK_REGION_OFFLINE to
RS_ZK_REGION_OPENING
2013-10-31 00:07:55,439 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Opening region: {NAME =>
'table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.',
STARTKEY => '6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8', ENDKEY =>
'71c71c71c71c71c71c71c71c71c71c71c71c71c0', ENCODED =>
3a476d37da81f620a3e53179d7d9192b,}
2013-10-31 00:07:55,439 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Instantiated
table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
2013-10-31 00:07:55,447 DEBUG org.apache.hadoop.hbase.regionserver.StoreFile:
Store file
hdfs://xxx-master/hbase/table_x/3a476d37da81f620a3e53179d7d9192b/A/table_y=abca20169d0fdd22f2a13d6caf41d83d-0729d4dbd652443bbd44381db5b7b26b
is a link
2013-10-31 00:07:55,501 DEBUG org.apache.hadoop.hbase.regionserver.Store:
loaded
hdfs://xxx-master/hbase/table_x/3a476d37da81f620a3e53179d7d9192b/A/table_y=abca20169d0fdd22f2a13d6caf41d83d-0729d4dbd652443bbd44381db5b7b26b,
isReference=false, isBulkLoadResult=false, seqid=46816, majorCompaction=true
2013-10-31 00:07:55,546 DEBUG org.apache.hadoop.hbase.regionserver.StoreFile:
Store file
hdfs://xxx-master/hbase/table_x/3a476d37da81f620a3e53179d7d9192b/B/table_y=abca20169d0fdd22f2a13d6caf41d83d-8606885898507153833
is a link
2013-10-31 00:07:55,602 DEBUG org.apache.hadoop.hbase.regionserver.StoreFile:
Store file
hdfs://xxx-master/hbase/table_x/3a476d37da81f620a3e53179d7d9192b/B/table_y=abca20169d0fdd22f2a13d6caf41d83d-b2449476f78e42b7ba0bba2ac69a24b8
is a link
2013-10-31 00:07:55,613 DEBUG org.apache.hadoop.hbase.regionserver.Store:
loaded
hdfs://xxx-master/hbase/table_x/3a476d37da81f620a3e53179d7d9192b/B/table_y=abca20169d0fdd22f2a13d6caf41d83d-b2449476f78e42b7ba0bba2ac69a24b8,
isReference=false, isBulkLoadResult=false, seqid=46816, majorCompaction=false
2013-10-31 00:07:55,618 ERROR
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open of
region=table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.,
starting to roll back the global memstore size.
2013-10-31 00:07:55,621 INFO
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Opening of
region {NAME =>
'table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.',
STARTKEY => '6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8', ENDKEY =>
'71c71c71c71c71c71c71c71c71c71c71c71c71c0', ENCODED =>
3a476d37da81f620a3e53179d7d9192b,} failed, marking as FAILED_OPEN in ZK
2013-10-31 00:07:55,621 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
regionserver:60020-0x242045137a20071 Attempting to transition node
3a476d37da81f620a3e53179d7d9192b from RS_ZK_REGION_OPENING to
RS_ZK_REGION_FAILED_OPEN
2013-10-31 00:07:55,630 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
regionserver:60020-0x242045137a20071 Successfully transitioned node
3a476d37da81f620a3e53179d7d9192b from RS_ZK_REGION_OPENING to
RS_ZK_REGION_FAILED_OPEN
{noformat}
grep by region caused exception on xxx108:
{noformat}
2013-10-31 00:07:57,003 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open
region:
table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
2013-10-31 00:07:57,010 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
regionserver:60020-0x242045137a20074 Attempting to transition node
3a476d37da81f620a3e53179d7d9192b from M_ZK_REGION_OFFLINE to
RS_ZK_REGION_OPENING
2013-10-31 00:07:57,042 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
regionserver:60020-0x242045137a20074 Successfully transitioned node
3a476d37da81f620a3e53179d7d9192b from M_ZK_REGION_OFFLINE to
RS_ZK_REGION_OPENING
2013-10-31 00:07:57,043 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Opening region: {NAME =>
'table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.',
STARTKEY => '6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8', ENDKEY =>
'71c71c71c71c71c71c71c71c71c71c71c71c71c0', ENCODED =>
3a476d37da81f620a3e53179d7d9192b,}
2013-10-31 00:07:57,043 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Instantiated
table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
2013-10-31 00:07:57,049 DEBUG org.apache.hadoop.hbase.regionserver.StoreFile:
Store file
hdfs://xxx-master/hbase/table_x/3a476d37da81f620a3e53179d7d9192b/A/table_y=abca20169d0fdd22f2a13d6caf41d83d-0729d4dbd652443bbd44381db5b7b26b
is a link
2013-10-31 00:07:57,060 DEBUG org.apache.hadoop.hbase.regionserver.Store:
loaded
hdfs://xxx-master/hbase/table_x/3a476d37da81f620a3e53179d7d9192b/A/table_y=abca20169d0fdd22f2a13d6caf41d83d-0729d4dbd652443bbd44381db5b7b26b,
isReference=false, isBulkLoadResult=false, seqid=46816, majorCompaction=true
2013-10-31 00:07:57,065 DEBUG org.apache.hadoop.hbase.regionserver.StoreFile:
Store file
hdfs://xxx-master/hbase/table_x/3a476d37da81f620a3e53179d7d9192b/B/table_y=abca20169d0fdd22f2a13d6caf41d83d-8606885898507153833
is a link
2013-10-31 00:07:57,095 DEBUG org.apache.hadoop.hbase.regionserver.StoreFile:
Store file
hdfs://xxx-master/hbase/table_x/3a476d37da81f620a3e53179d7d9192b/B/table_y=abca20169d0fdd22f2a13d6caf41d83d-b2449476f78e42b7ba0bba2ac69a24b8
is a link
2013-10-31 00:07:57,105 DEBUG org.apache.hadoop.hbase.regionserver.Store:
loaded
hdfs://xxx-master/hbase/table_x/3a476d37da81f620a3e53179d7d9192b/B/table_y=abca20169d0fdd22f2a13d6caf41d83d-b2449476f78e42b7ba0bba2ac69a24b8,
isReference=false, isBulkLoadResult=false, seqid=46816, majorCompaction=false
2013-10-31 00:07:57,107 ERROR
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open of
region=table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.,
starting to roll back the global memstore size.
2013-10-31 00:07:57,108 INFO
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Opening of
region {NAME =>
'table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.',
STARTKEY => '6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8', ENDKEY =>
'71c71c71c71c71c71c71c71c71c71c71c71c71c0', ENCODED =>
3a476d37da81f620a3e53179d7d9192b,} failed, marking as FAILED_OPEN in ZK
2013-10-31 00:07:57,108 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
regionserver:60020-0x242045137a20074 Attempting to transition node
3a476d37da81f620a3e53179d7d9192b from RS_ZK_REGION_OPENING to
RS_ZK_REGION_FAILED_OPEN
2013-10-31 00:07:57,125 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
regionserver:60020-0x242045137a20074 Successfully transitioned node
3a476d37da81f620a3e53179d7d9192b from RS_ZK_REGION_OPENING to
RS_ZK_REGION_FAILED_OPEN
{noformat}
grep by region caused exception on xxx104:
{noformat}
2013-10-31 00:07:58,581 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open
region:
table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
2013-10-31 00:07:58,587 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
regionserver:60020-0x420451326a0070 Attempting to transition node
3a476d37da81f620a3e53179d7d9192b from M_ZK_REGION_OFFLINE to
RS_ZK_REGION_OPENING
2013-10-31 00:07:58,602 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
regionserver:60020-0x420451326a0070 Successfully transitioned node
3a476d37da81f620a3e53179d7d9192b from M_ZK_REGION_OFFLINE to
RS_ZK_REGION_OPENING
2013-10-31 00:07:58,603 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Opening region: {NAME =>
'table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.',
STARTKEY => '6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8', ENDKEY =>
'71c71c71c71c71c71c71c71c71c71c71c71c71c0', ENCODED =>
3a476d37da81f620a3e53179d7d9192b,}
2013-10-31 00:07:58,604 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Instantiated
table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
2013-10-31 00:07:58,610 DEBUG org.apache.hadoop.hbase.regionserver.StoreFile:
Store file
hdfs://xxx-master/hbase/table_x/3a476d37da81f620a3e53179d7d9192b/A/table_y=abca20169d0fdd22f2a13d6caf41d83d-0729d4dbd652443bbd44381db5b7b26b
is a link
2013-10-31 00:07:58,621 DEBUG org.apache.hadoop.hbase.regionserver.Store:
loaded
hdfs://xxx-master/hbase/table_x/3a476d37da81f620a3e53179d7d9192b/A/table_y=abca20169d0fdd22f2a13d6caf41d83d-0729d4dbd652443bbd44381db5b7b26b,
isReference=false, isBulkLoadResult=false, seqid=46816, majorCompaction=true
2013-10-31 00:07:58,627 DEBUG org.apache.hadoop.hbase.regionserver.StoreFile:
Store file
hdfs://xxx-master/hbase/table_x/3a476d37da81f620a3e53179d7d9192b/B/table_y=abca20169d0fdd22f2a13d6caf41d83d-8606885898507153833
is a link
2013-10-31 00:07:58,639 DEBUG org.apache.hadoop.hbase.regionserver.StoreFile:
Store file
hdfs://xxx-master/hbase/table_x/3a476d37da81f620a3e53179d7d9192b/B/table_y=abca20169d0fdd22f2a13d6caf41d83d-b2449476f78e42b7ba0bba2ac69a24b8
is a link
2013-10-31 00:07:58,650 DEBUG org.apache.hadoop.hbase.regionserver.Store:
loaded
hdfs://xxx-master/hbase/table_x/3a476d37da81f620a3e53179d7d9192b/B/table_y=abca20169d0fdd22f2a13d6caf41d83d-b2449476f78e42b7ba0bba2ac69a24b8,
isReference=false, isBulkLoadResult=false, seqid=46816, majorCompaction=false
2013-10-31 00:07:58,652 ERROR
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open of
region=table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.,
starting to roll back the global memstore size.
2013-10-31 00:07:58,653 INFO
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Opening of
region {NAME =>
'table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.',
STARTKEY => '6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8', ENDKEY =>
'71c71c71c71c71c71c71c71c71c71c71c71c71c0', ENCODED =>
3a476d37da81f620a3e53179d7d9192b,} failed, marking as FAILED_OPEN in ZK
2013-10-31 00:07:58,653 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
regionserver:60020-0x420451326a0070 Attempting to transition node
3a476d37da81f620a3e53179d7d9192b from RS_ZK_REGION_OPENING to
RS_ZK_REGION_FAILED_OPEN
2013-10-31 00:07:58,670 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
regionserver:60020-0x420451326a0070 Successfully transitioned node
3a476d37da81f620a3e53179d7d9192b from RS_ZK_REGION_OPENING to
RS_ZK_REGION_FAILED_OPEN
{noformat}
1) Initially AM try to assign region with 'bulk assign' on xxx100 (PENDING_OPEN
=> RS_ZK_REGION_OPENING); but xxx100 failed to open region and AM handles this
event (RS_ZK_REGION_FAILED_OPEN => CLOSED => OFFLINE)
2) AM try to assign region in ClosedRegionHandler on xxx108 (there is no
RS_ZK_REGION_OPENING event in master's logs, but we see it in regionserver's
logs); it fails again
3) AM chose xxx106 for region assignment but receives RS_ZK_REGION_FAILED_OPEN
before sending request => CLOSED => ClosedRegionHandler => xxx104 => exception
> [0.94] AssignmentManager throws IllegalStateException from PENDING_OPEN to
> OFFLINE
> ----------------------------------------------------------------------------------
>
> Key: HBASE-8912
> URL: https://issues.apache.org/jira/browse/HBASE-8912
> Project: HBase
> Issue Type: Bug
> Reporter: Enis Soztutar
> Fix For: 0.94.14
>
> Attachments: HBase-0.94 #1036 test - testRetrying [Jenkins].html
>
>
> AM throws this exception which subsequently causes the master to abort:
> {code}
> java.lang.IllegalStateException: Unexpected state :
> testRetrying,jjj,1372891751115.9b828792311001062a5ff4b1038fe33b.
> state=PENDING_OPEN, ts=1372891751912,
> server=hemera.apache.org,39064,1372891746132 .. Cannot transit it to OFFLINE.
> at
> org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1879)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1688)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394)
> at
> org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> This exception trace is from the failing test TestMetaReaderEditor which is
> failing pretty frequently, but looking at the test code, I think this is not
> a test-only issue, but affects the main code path.
> https://builds.apache.org/job/HBase-0.94/1036/testReport/junit/org.apache.hadoop.hbase.catalog/TestMetaReaderEditor/testRetrying/
--
This message was sent by Atlassian JIRA
(v6.1#6144)
