We're running hbase 0.90.3. For some unknown reason, we now can't disable
one table because its first region can't be unassigned.
The log message looks like following and it repeats endlessly:
2011-07-25 13:27:23,745 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition
timed out:
HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
state=PENDING_CLOSE, ts=1311571458790
2011-07-25 13:27:23,745 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Region has been
PENDING_CLOSE for too long, running forced unassign again on
region=HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
2011-07-25 13:27:23,746 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
region HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
(offlining)
2011-07-25 13:27:23,746 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Attempting to unassign
region HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
which is already pending close but forcing an additional close
2011-07-25 13:27:23,748 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Server serverName=
test28.xxx.com,60020,1309749632386, load=(requests=0, regions=237,
usedHeap=11121, maxHeap=15983) returned
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: Received close for
HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa. but we are
not serving it for aaa8d345f5b7b6a69b786fe6d14ed9fa
2011-07-25 13:28:53,508 DEBUG org.apache.hadoop.hbase.master.HMaster: Not
running balancer because 1 region(s) in transition:
{aaa8d345f5b7b6a69b786fe6d14ed9fa=HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
state=PENDING_CLOSE, ts=1311571643746}
2011-07-25 13:28:55,308 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor:
Scanned 5424 catalog row(s) and gc'd 0 unreferenced parent region(s)
2011-07-25 13:30:17,565 INFO org.apache.zookeeper.ZooKeeper: Initiating
client connection, connectString=192.168.201.7:2181,192.168.201.6:2181,
192.168.201.5:2181 sessionTimeout=60000 watcher=hconnection
2011-07-25 13:30:17,566 INFO org.apache.zookeeper.ClientCnxn: Opening socket
connection to server /192.168.201.7:2181
2011-07-25 13:30:17,568 INFO org.apache.zookeeper.ClientCnxn: Socket
connection established to test07.xxx.com/192.168.201.7:2181, initiating
session
2011-07-25 13:30:17,572 INFO org.apache.zookeeper.ClientCnxn: Session
establishment complete on server test07.xxx.com/192.168.201.7:2181,
sessionid = 0x3314bcb3ae38799, negotiated timeout = 60000
2011-07-25 13:30:17,583 DEBUG org.apache.hadoop.hbase.client.MetaScanner:
Scanning .META. starting at row=HistoryNoticeInc,,00000000000000 for
max=2147483647 rows
2011-07-25 13:30:17,627 INFO
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Closed zookeeper sessionid=0x3314bcb3ae38799
2011-07-25 13:30:17,637 INFO org.apache.zookeeper.ZooKeeper: Session:
0x3314bcb3ae38799 closed
2011-07-25 13:30:17,637 INFO org.apache.zookeeper.ClientCnxn: EventThread
shut down
2011-07-25 13:30:33,745 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition
timed out:
HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
state=PENDING_CLOSE, ts=1311571643746
2011-07-25 13:30:33,745 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Region has been
PENDING_CLOSE for too long, running forced unassign again on
region=HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
The problem also blocks the region balancer.
Can I fix it without restarting the whole hbase cluster?
Thanks,
Yi
