Hi
We are running hbase 0.94.6 cdh 4.4 and have a problem with one table not
being assigned to any region. This is the SYSTEM.TABLE in Phoenix so all
tables are basically non functional at the moment.
When running hbck repair we get the following...
ERROR: Region { meta =>
SYSTEM.TABLE,,1385477659546.b739cf1ef14dd664ae873bf5b16e7a35., hdfs =>
hdfs://primecluster/hbase/SYSTEM.TABLE/b739cf1ef14dd664ae873bf5b16e7a35,
deployed => } not deployed on any region server.
And when hbck -repair (hbck_repair.txt attachment) it never assigns a
region and everything seems stuck.
Master log keeps spewing out the following at very rapid pace (master.log
attachement)...
Any pointers?
Cheers,
-Kristoffer
;; This buffer is for notes you don't want to save, and for Lisp evaluation.
;; If you want to create a file, visit that file with C-x C-f,
;; then enter the text in that file's own buffer.
4/08/14 16:27:51 INFO util.HBaseFsck: Loading regioninfos HDFS
14/08/14 16:27:51 INFO util.HBaseFsck: Loading HBase regioninfo from HDFS...
14/08/14 16:27:51 DEBUG util.HBaseFsck: Loading region dirs from
hdfs://primecluster/hbase/-ROOT-
14/08/14 16:27:51 DEBUG util.HBaseFsck: Loading region dirs from
hdfs://primecluster/hbase/.META.
14/08/14 16:27:51 DEBUG util.HBaseFsck: Loading region dirs from
hdfs://primecluster/hbase/SYSTEM.TABLE
14/08/14 16:27:51 DEBUG util.HBaseFsck: Loading region info from
hdfs:hdfs://primecluster/hbase/.META./1028785192
14/08/14 16:27:51 DEBUG util.HBaseFsck: Loading region info from
hdfs:hdfs://primecluster/hbase/SYSTEM.TABLE/b739cf1ef14dd664ae873bf5b16e7a35
14/08/14 16:27:51 DEBUG util.HBaseFsck: Loading region info from
hdfs:hdfs://primecluster/hbase/-ROOT-/70236052
14/08/14 16:27:51 DEBUG util.HBaseFsck: HRegionInfo read: {NAME => '-ROOT-,,0',
STARTKEY => '', ENDKEY => '', ENCODED => 70236052,}
14/08/14 16:27:51 DEBUG util.HBaseFsck: HRegionInfo read: {NAME =>
'SYSTEM.TABLE,,1385477659546.b739cf1ef14dd664ae873bf5b16e7a35.', STARTKEY =>
'', ENDKEY => '', ENCODED => b739cf1ef14dd664ae873bf5b16e7a35,}
14/08/14 16:27:51 DEBUG util.HBaseFsck: HRegionInfo read: {NAME => '.META.,,1',
STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,}
14/08/14 16:27:51 INFO util.HBaseFsck: Checking HBase region split map from
HDFS data...
14/08/14 16:27:51 INFO util.HBaseFsck: No integrity errors. We are done with
this phase. Glorious.
14/08/14 16:27:51 INFO util.HBaseFsck: Loading regionsinfo from the .META. table
14/08/14 16:27:51 DEBUG client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@7979a49f;
serverName=hdfs-ix15.se-ix.delta.prod,60020,1408022977976
14/08/14 16:27:51 DEBUG client.MetaScanner: Scanning -ROOT- starting at row=
for max=2147483647 rows using
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@7979a49f
14/08/14 16:27:51 DEBUG client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@7979a49f;
serverName=hdfs-ix15.se-ix.delta.prod,60020,1408022977976
14/08/14 16:27:51 DEBUG client.MetaScanner: Scanning .META. starting at row=
for max=2147483647 rows using
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@7979a49f
14/08/14 16:27:52 INFO util.HBaseFsck: getHTableDescriptors == tableNames =>
[SYSTEM.TABLE]
14/08/14 16:27:52 WARN conf.Configuration: fs.default.name is deprecated.
Instead, use fs.defaultFS
Number of Tables: 1
Number of live region servers: 23
Number of dead region servers: 0
Master: nn-ix01.se-ix.delta.prod,60000,1408024947592
Number of backup masters: 0
..14/08/14 16:27:52 DEBUG util.HBaseFsck: Loading region dirs from
hdfs://primecluster/hbase/-ROOT-
14/08/14 16:27:52 DEBUG util.HBaseFsck: Loading region dirs from
hdfs://primecluster/hbase/.META.
14/08/14 16:27:52 DEBUG util.HBaseFsck: Loading region dirs from
hdfs://primecluster/hbase/SYSTEM.TABLE
14/08/14 16:27:52 DEBUG util.HBaseFsck: Loading region info from
hdfs:hdfs://primecluster/hbase/.META./1028785192
14/08/14 16:27:52 DEBUG util.HBaseFsck: Loading region info from
hdfs:hdfs://primecluster/hbase/-ROOT-/70236052
14/08/14 16:27:52 DEBUG util.HBaseFsck: Loading region info from
hdfs:hdfs://primecluster/hbase/SYSTEM.TABLE/b739cf1ef14dd664ae873bf5b16e7a35
14/08/14 16:27:52 DEBUG util.HBaseFsck: HRegionInfo read: {NAME => '-ROOT-,,0',
STARTKEY => '', ENDKEY => '', ENCODED => 70236052,}
14/08/14 16:27:52 DEBUG util.HBaseFsck: HRegionInfo read: {NAME =>
'SYSTEM.TABLE,,1385477659546.b739cf1ef14dd664ae873bf5b16e7a35.', STARTKEY =>
'', ENDKEY => '', ENCODED => b739cf1ef14dd664ae873bf5b16e7a35,}
14/08/14 16:27:52 DEBUG util.HBaseFsck: HRegionInfo read: {NAME => '.META.,,1',
STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,}
Number of empty REGIONINFO_QUALIFIER rows in .META.: 0
ERROR: Region { meta =>
SYSTEM.TABLE,,1385477659546.b739cf1ef14dd664ae873bf5b16e7a35., hdfs =>
hdfs://primecluster/hbase/SYSTEM.TABLE/b739cf1ef14dd664ae873bf5b16e7a35,
deployed => } not deployed on any region server.
Trying to fix unassigned region...
14/08/14 16:27:52 INFO util.HBaseFsckRepair: Region still in transition,
waiting for it to become assigned: {NAME =>
'SYSTEM.TABLE,,1385477659546.b739cf1ef14dd664ae873bf5b16e7a35.', STARTKEY =>
'', ENDKEY => '', ENCODED => b739cf1ef14dd664ae873bf5b16e7a35,}
14/08/14 16:27:53 INFO util.HBaseFsckRepair: Region still in transition,
waiting for it to become assigned: {NAME =>
'SYSTEM.TABLE,,1385477659546.b739cf1ef14dd664ae873bf5b16e7a35.', STARTKEY =>
'', ENDKEY => '', ENCODED => b739cf1ef14dd664ae873bf5b16e7a35,}
14/08/14 16:27:54 INFO util.HBaseFsckRepair: Region still in transition,
waiting for it to become assigned: {NAME =>
'SYSTEM.TABLE,,1385477659546.b739cf1ef14dd664ae873bf5b16e7a35.', STARTKEY =>
'', ENDKEY => '', ENCODED => b739cf1ef14dd664ae873bf5b16e7a35,}
14/08/14 16:27:55 INFO util.HBaseFsckRepair: Region still in transition,
waiting for it to become assigned: {NAME =>
'SYSTEM.TABLE,,1385477659546.b739cf1ef14dd664ae873bf5b16e7a35.', STARTKEY =>
'', ENDKEY => '', ENCODED => b739cf1ef14dd664ae873bf5b16e7a35,}
14/08/14 16:27:56 INFO util.HBaseFsckRepair: Region still in transition,
waiting for it to become assigned: {NAME =>
'SYSTEM.TABLE,,1385477659546.b739cf1ef14dd664ae873bf5b16e7a35.', STARTKEY =>
'', ENDKEY => '', ENCODED => b739cf1ef14dd664ae873bf5b16e7a35,}
2014-08-14 16:20:45,977 DEBUG org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED event for b739cf1ef14dd664ae873bf5b16e7a35
2014-08-14 16:20:45,977 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=SYSTEM.TABLE,,1385477659546.b739cf1ef14dd664ae873bf5b16e7a35. state=CLOSED, ts=1408026045972, server=hdfs-ix11.se-ix.delta.prod,60020,1408022981002
2014-08-14 16:20:45,977 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x74407b3bc4a090b Creating (or updating) unassigned node for b739cf1ef14dd664ae873bf5b16e7a35 with OFFLINE state
2014-08-14 16:20:45,982 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=M_ZK_REGION_OFFLINE, server=nn-ix01.se-ix.delta.prod,60000,1408024947592, region=b739cf1ef14dd664ae873bf5b16e7a35
2014-08-14 16:20:45,982 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan for SYSTEM.TABLE,,1385477659546.b739cf1ef14dd664ae873bf5b16e7a35. destination server is hdfs-ix22.se-ix.delta.prod,60020,1408022967725
2014-08-14 16:20:45,982 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Using pre-existing plan for region SYSTEM.TABLE,,1385477659546.b739cf1ef14dd664ae873bf5b16e7a35.; plan=hri=SYSTEM.TABLE,,1385477659546.b739cf1ef14dd664ae873bf5b16e7a35., src=, dest=hdfs-ix22.se-ix.delta.prod,60020,1408022967725
2014-08-14 16:20:45,982 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Assigning region SYSTEM.TABLE,,1385477659546.b739cf1ef14dd664ae873bf5b16e7a35. to hdfs-ix22.se-ix.delta.prod,60020,1408022967725
2014-08-14 16:20:46,040 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENING, server=hdfs-ix22.se-ix.delta.prod,60020,1408022967725, region=b739cf1ef14dd664ae873bf5b16e7a35
2014-08-14 16:20:46,101 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_FAILED_OPEN, server=hdfs-ix22.se-ix.delta.prod,60020,1408022967725, region=b739cf1ef14dd664ae873bf5b16e7a35
2014-08-14 16:20:46,101 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan for SYSTEM.TABLE,,1385477659546.b739cf1ef14dd664ae873bf5b16e7a35. destination server is hdfs-ix22.se-ix.delta.prod,60020,1408022967725
2014-08-14 16:20:46,101 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: No previous transition plan was found (or we are ignoring an existing plan) for SYSTEM.TABLE,,1385477659546.b739cf1ef14dd664ae873bf5b16e7a35. so generated a random one; hri=SYSTEM.TABLE,,1385477659546.b739cf1ef14dd664ae873bf5b16e7a35., src=, dest=hdfs-ix14.se-ix.delta.prod,60020,1408022998462; 23 (online=23, available=22) available servers
2014-08-14 16:20:46,101 DEBUG org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED event for b739cf1ef14dd664ae873bf5b16e7a35
2014-08-14 16:20:46,101 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=SYSTEM.TABLE,,1385477659546.b739cf1ef14dd664ae873bf5b16e7a35. state=CLOSED, ts=1408026046099, server=hdfs-ix22.se-ix.delta.prod,60020,1408022967725
2014-08-14 16:20:46,101 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x74407b3bc4a090b Creating (or updating) unassigned node for b739cf1ef14dd664ae873bf5b16e7a35 with OFFLINE state
2014-08-14 16:20:46,107 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=M_ZK_REGION_OFFLINE, server=nn-ix01.se-ix.delta.prod,60000,1408024947592, region=b739cf1ef14dd664ae873bf5b16e7a35
2014-08-14 16:20:46,107 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan for SYSTEM.TABLE,,1385477659546.b739cf1ef14dd664ae873bf5b16e7a35. destination server is hdfs-ix14.se-ix.delta.prod,60020,1408022998462
2014-08-14 16:20:46,107 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Using pre-existing plan for region SYSTEM.TABLE,,1385477659546.b739cf1ef14dd664ae873bf5b16e7a35.; plan=hri=SYSTEM.TABLE,,1385477659546.b739cf1ef14dd664ae873bf5b16e7a35., src=, dest=hdfs-ix14.se-ix.delta.prod,60020,1408022998462
2014-08-14 16:20:46,107 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Assigning region SYSTEM.TABLE,,1385477659546.b739cf1ef14dd664ae873bf5b16e7a35. to hdfs-ix14.se-ix.delta.prod,60020,1408022998462
2014-08-14 16:20:46,114 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENING, server=hdfs-ix14.se-ix.delta.prod,60020,1408022998462, region=b739cf1ef14dd664ae873bf5b16e7a35
2014-08-14 16:20:46,134 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_FAILED_OPEN, server=hdfs-ix14.se-ix.delta.prod,60020,1408022998462, region=b739cf1ef14dd664ae873bf5b16e7a35
2014-08-14 16:20:46,134 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan for SYSTEM.TABLE,,1385477659546.b739cf1ef14dd664ae873bf5b16e7a35. destination server is hdfs-ix14.se-ix.delta.prod,60020,1408022998462
2014-08-14 16:20:46,134 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: No previous transition plan was found (or we are ignoring an existing plan) for SYSTEM.TABLE,,1385477659546.b739cf1ef14dd664ae873bf5b16e7a35. so generated a random one; hri=SYSTEM.TABLE,,1385477659546.b739cf1ef14dd664ae873bf5b16e7a35., src=, dest=hdfs-ix08.se-ix.delta.prod,60020,1408022983438; 23 (online=23, available=22) available servers
2014-08-14 16:20:46,134 DEBUG org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED event for b739cf1ef14dd664ae873bf5b16e7a35
2014-08-14 16:20:46,134 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=SYSTEM.TABLE,,1385477659546.b739cf1ef14dd664ae873bf5b16e7a35. state=CLOSED, ts=1408026046129, server=hdfs-ix14.se-ix.delta.prod,60020,1408022998462
2014-08-14 16:20:46,134 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x74407b3bc4a090b Creating (or updating) unassigned node for b739cf1ef14dd664ae873bf5b16e7a35 with OFFLINE state
2014-08-14 16:20:46,139 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=M_ZK_REGION_OFFLINE, server=nn-ix01.se-ix.delta.prod,60000,1408024947592, region=b739cf1ef14dd664ae873bf5b16e7a35
2014-08-14 16:20:46,139 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan for SYSTEM.TABLE,,1385477659546.b739cf1ef14dd664ae873bf5b16e7a35. destination server is hdfs-ix08.se-ix.delta.prod,60020,1408022983438
2014-08-14 16:20:46,139 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Using pre-existing plan for region SYSTEM.TABLE,,1385477659546.b739cf1ef14dd664ae873bf5b16e7a35.; plan=hri=SYSTEM.TABLE,,1385477659546.b739cf1ef14dd664ae873bf5b16e7a35., src=, dest=hdfs-ix08.se-ix.delta.prod,60020,1408022983438
2014-08-14 16:20:46,139 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Assigning region SYSTEM.TABLE,,1385477659546.b739cf1ef14dd664ae873bf5b16e7a35. to hdfs-ix08.se-ix.delta.prod,60020,1408022983438
2014-08-14 16:20:46,146 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENING, server=hdfs-ix08.se-ix.delta.prod,60020,1408022983438, region=b739cf1ef14dd664ae873bf5b16e7a35
2014-08-14 16:20:46,169 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_FAILED_OPEN, server=hdfs-ix08.se-ix.delta.prod,60020,1408022983438, region=b739cf1ef14dd664ae873bf5b16e7a35
2014-08-14 16:20:46,169 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan for SYSTEM.TABLE,,1385477659546.b739cf1ef14dd664ae873bf5b16e7a35. destination server is hdfs-ix08.se-ix.delta.prod,60020,1408022983438
2014-08-14 16:20:46,169 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: No previous transition plan was found (or we are ignoring an existing plan) for SYSTEM.TABLE,,1385477659546.b739cf1ef14dd664ae873bf5b16e7a35. so generated a random one; hri=SYSTEM.TABLE,,1385477659546.b739cf1ef14dd664ae873bf5b16e7a35., src=, dest=hdfs-ix12.se-ix.delta.prod,60020,1408022994075; 23 (online=23, available=22) available servers
