Today I test Hbase verison 0.90.3
I seems like some bugs
1, if node exists and node state is RS_ZK_REGION_CLOSED
We should call ClosedRegionHandle to close region. In that case the reigon
has closed by region server
case PENDING_CLOSE:
LOG.info("Region has been PENDING_CLOSE for too " +
"long, running forced unassign again on region=" +
regionInfo.getRegionNameAsString());
try {
// If the server got the RPC, it will transition the node
// to CLOSING, so only do something here if no node exists
if (!ZKUtil.watchAndCheckExists(watcher,
ZKAssign.getNodeName(watcher,
regionInfo.getEncodedName()))) {
// Queue running of an unassign -- do actual unassign
// outside of the regionsInTransition lock.
unassigns.add(regionInfo);
}
else
{
//It need to process for node
state is RS_ZK_REGION_CLOSED
}
} catch (NoNodeException e) {
LOG.debug("Node no longer existed so not forcing another " +
"unassignment");
} catch (KeeperException e) {
LOG.warn("Unexpected ZK exception timing out a region " +
"close", e);
}
2, other It seems like zk message losing or there is some bugs for disabling
table when region server is splitting region.
above is the logs. I need to keep digging.
Shell logs:
hbase(main):003:0*
hbase(main):004:0* disable 'ufdr'
11/05/08 18:20:40 DEBUG zookeeper.ZKUtil: hconnection-0x2fcd58283c0037 Set
watcher on existing znode /hbase/root-region-server
11/05/08 18:20:40 DEBUG zookeeper.ZKUtil: hconnection-0x2fcd58283c0037
Retrieved 15 byte(s) of data from znode /hbase/root-region-server and set
watcher; C4C5.site:60020
11/05/08 18:53:19 DEBUG zookeeper.ZKUtil: hconnection-0x2fcd58283c0037
Retrieved 9 byte(s) of data from znode /hbase/table/ufdr; data=DISABLING
ERROR: org.apache.hadoop.hbase.RegionException: Retries exhausted, it took too
long to wait for the table ufdr to be disabled.
Region server logs:
2011-05-08 17:42:44,862 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Instantiated
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
2011-05-08 17:42:45,468 DEBUG org.apache.hadoop.hbase.regionserver.Store:
loaded
hdfs://C4C1:9000/hbase/ufdr/4418fb197685a21f77e151e401cf8b66/value/3442771824694350714.8e9a3b05abe1c3a692999cf5e8dfd9dd,
isReference=true, isBulkLoadResult=false, seqid=627830, majorCompaction=false
2011-05-08 17:42:45,471 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Onlined
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.;
next sequenceid=627831
2011-05-08 17:42:45,471 DEBUG
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction requested
for
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
because Region has references on open; priority=9, compaction queue size=40
2011-05-08 17:42:45,476 INFO org.apache.hadoop.hbase.catalog.MetaEditor: Added
daughter
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
in region .META.,,1, serverInfo=C4C4.site,60020,1304820199467
2011-05-08 17:42:45,476 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Region split, META
updated, and report to master.
Parent=ufdr,2011050812#8613817306227#0516,1304845660567.8e9a3b05abe1c3a692999cf5e8dfd9dd.,
new regions:
ufdr,2011050812#8613817306227#0516,1304847764729.5e4bca85c33fa6605ffc9a5c2eb94e62.,
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66..
Split took 0sec
2011-05-08 17:44:25,731 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Received close region:
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
2011-05-08 17:48:11,066 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
2011-05-08 17:48:11,067 INFO org.apache.hadoop.hbase.regionserver.Store:
Started compaction of 1 file(s) in cf=value, hasReferences=true, into
hdfs://C4C1:9000/hbase/ufdr/4418fb197685a21f77e151e401cf8b66/.tmp,
seqid=627830, totalSize=892.2m
2011-05-08 17:48:11,067 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Compacting
hdfs://C4C1:9000/hbase/ufdr/4418fb197685a21f77e151e401cf8b66/value/3442771824694350714.8e9a3b05abe1c3a692999cf5e8dfd9dd-hdfs://C4C1:9000/hbase/ufdr/8e9a3b05abe1c3a692999cf5e8dfd9dd/value/3442771824694350714-top,
keycount=2843998, bloomtype=NONE, size=892.2m
2011-05-08 17:48:32,340 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed compaction of 1 file(s), new
file=hdfs://C4C1:9000/hbase/ufdr/4418fb197685a21f77e151e401cf8b66/value/4640085674257062204,
size=446.2m; total size for store is 446.2m
2011-05-08 17:48:32,340 INFO org.apache.hadoop.hbase.regionserver.HRegion:
completed compaction on region
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
after 21sec
2011-05-08 17:52:46,626 DEBUG
org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing
close of
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
2011-05-08 17:52:46,627 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
regionserver:60020-0x12fcd58280c0003-0x12fcd58280c0003 Creating unassigned node
for 4418fb197685a21f77e151e401cf8b66 in a CLOSING state
2011-05-08 17:52:46,986 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Closing
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.:
disabling compactions & flushes
2011-05-08 17:52:46,986 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Updates disabled for region
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
2011-05-08 17:52:46,986 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
2011-05-08 17:52:46,986 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
regionserver:60020-0x12fcd58280c0003-0x12fcd58280c0003 Attempting to transition
node 4418fb197685a21f77e151e401cf8b66 from RS_ZK_REGION_CLOSING to
RS_ZK_REGION_CLOSED
2011-05-08 17:52:47,052 DEBUG org.apache.hadoop.hbase.zookeeper.ZKUtil:
regionserver:60020-0x12fcd58280c0003-0x12fcd58280c0003 Retrieved 125 byte(s) of
data from znode /hbase/unassigned/4418fb197685a21f77e151e401cf8b66;
data=region=ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.,
server=C4C4.site,60020,1304820199467, state=RS_ZK_REGION_CLOSING
2011-05-08 17:52:47,143 DEBUG
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher:
regionserver:60020-0x12fcd58280c0003-0x12fcd58280c0003 Received ZooKeeper
Event, type=NodeDataChanged, state=SyncConnected,
path=/hbase/unassigned/4418fb197685a21f77e151e401cf8b66
2011-05-08 17:52:47,143 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
regionserver:60020-0x12fcd58280c0003-0x12fcd58280c0003 Successfully
transitioned node 4418fb197685a21f77e151e401cf8b66 from RS_ZK_REGION_CLOSING to
RS_ZK_REGION_CLOSED
2011-05-08 17:52:47,143 DEBUG
org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
2011-05-08 17:52:47,696 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Received close region:
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
2011-05-08 17:52:47,696 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: Received close for region
we are not serving; 4418fb197685a21f77e151e401cf8b66
2011-05-08 17:52:47,814 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Received close region:
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
2011-05-08 17:52:47,814 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: Received close for region
we are not serving; 4418fb197685a21f77e151e401cf8b66
2011-05-08 17:52:48,167 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Received close region:
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
2011-05-08 17:52:48,167 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: Received close for region
we are not serving; 4418fb197685a21f77e151e401cf8b66
Hmaster logs:
2011-05-08 17:42:45,514 INFO org.apache.hadoop.hbase.master.ServerManager:
Received REGION_SPLIT:
ufdr,2011050812#8613817306227#0516,1304845660567.8e9a3b05abe1c3a692999cf5e8dfd9dd.:
Daughters;
ufdr,2011050812#8613817306227#0516,1304847764729.5e4bca85c33fa6605ffc9a5c2eb94e62.,
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
from C4C4.site,60020,1304820199467
2011-05-08 17:43:37,599 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Starting unassignment of region
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
(offlining)
2011-05-08 17:43:45,525 INFO org.apache.hadoop.hbase.master.ServerManager:
Received REGION_SPLIT:
ufdr,2011050812#8613817306227#0516,1304845660567.8e9a3b05abe1c3a692999cf5e8dfd9dd.:
Daughters;
ufdr,2011050812#8613817306227#0516,1304847764729.5e4bca85c33fa6605ffc9a5c2eb94e62.,
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
from C4C4.site,60020,1304820199467
2011-05-08 17:44:25,745 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Sent CLOSE to serverName=C4C4.site,60020,1304820199467, load=(requests=0,
regions=123, usedHeap=4097, maxHeap=8175) for region
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
2011-05-08 17:44:45,530 INFO org.apache.hadoop.hbase.master.ServerManager:
Received REGION_SPLIT:
ufdr,2011050812#8613817306227#0516,1304845660567.8e9a3b05abe1c3a692999cf5e8dfd9dd.:
Daughters;
ufdr,2011050812#8613817306227#0516,1304847764729.5e4bca85c33fa6605ffc9a5c2eb94e62.,
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
from C4C4.site,60020,1304820199467
2011-05-08 17:45:45,542 INFO org.apache.hadoop.hbase.master.ServerManager:
Received REGION_SPLIT:
ufdr,2011050812#8613817306227#0516,1304845660567.8e9a3b05abe1c3a692999cf5e8dfd9dd.:
Daughters;
ufdr,2011050812#8613817306227#0516,1304847764729.5e4bca85c33fa6605ffc9a5c2eb94e62.,
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
from C4C4.site,60020,1304820199467
2011-05-08 17:46:45,303 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting 4418fb197685a21f77e151e401cf8b66 on
serverName=C4C4.site,60020,1304820199467, load=(requests=0, regions=123,
usedHeap=4097, maxHeap=8175)
2011-05-08 17:46:45,538 INFO org.apache.hadoop.hbase.master.ServerManager:
Received REGION_SPLIT:
ufdr,2011050812#8613817306227#0516,1304845660567.8e9a3b05abe1c3a692999cf5e8dfd9dd.:
Daughters;
ufdr,2011050812#8613817306227#0516,1304847764729.5e4bca85c33fa6605ffc9a5c2eb94e62.,
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
from C4C4.site,60020,1304820199467
2011-05-08 17:47:45,548 INFO org.apache.hadoop.hbase.master.ServerManager:
Received REGION_SPLIT:
ufdr,2011050812#8613817306227#0516,1304845660567.8e9a3b05abe1c3a692999cf5e8dfd9dd.:
Daughters;
ufdr,2011050812#8613817306227#0516,1304847764729.5e4bca85c33fa6605ffc9a5c2eb94e62.,
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
from C4C4.site,60020,1304820199467
2011-05-08 17:48:45,545 INFO org.apache.hadoop.hbase.master.ServerManager:
Received REGION_SPLIT:
ufdr,2011050812#8613817306227#0516,1304845660567.8e9a3b05abe1c3a692999cf5e8dfd9dd.:
Daughters;
ufdr,2011050812#8613817306227#0516,1304847764729.5e4bca85c33fa6605ffc9a5c2eb94e62.,
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
from C4C4.site,60020,1304820199467
2011-05-08 17:49:46,108 INFO org.apache.hadoop.hbase.master.ServerManager:
Received REGION_SPLIT:
ufdr,2011050812#8613817306227#0516,1304845660567.8e9a3b05abe1c3a692999cf5e8dfd9dd.:
Daughters;
ufdr,2011050812#8613817306227#0516,1304847764729.5e4bca85c33fa6605ffc9a5c2eb94e62.,
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
from C4C4.site,60020,1304820199467
2011-05-08 17:50:46,105 INFO org.apache.hadoop.hbase.master.ServerManager:
Received REGION_SPLIT:
ufdr,2011050812#8613817306227#0516,1304845660567.8e9a3b05abe1c3a692999cf5e8dfd9dd.:
Daughters;
ufdr,2011050812#8613817306227#0516,1304847764729.5e4bca85c33fa6605ffc9a5c2eb94e62.,
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
from C4C4.site,60020,1304820199467
2011-05-08 17:51:46,117 INFO org.apache.hadoop.hbase.master.ServerManager:
Received REGION_SPLIT:
ufdr,2011050812#8613817306227#0516,1304845660567.8e9a3b05abe1c3a692999cf5e8dfd9dd.:
Daughters;
ufdr,2011050812#8613817306227#0516,1304847764729.5e4bca85c33fa6605ffc9a5c2eb94e62.,
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
from C4C4.site,60020,1304820199467
2011-05-08 17:52:46,112 INFO org.apache.hadoop.hbase.master.ServerManager:
Received REGION_SPLIT:
ufdr,2011050812#8613817306227#0516,1304845660567.8e9a3b05abe1c3a692999cf5e8dfd9dd.:
Daughters;
ufdr,2011050812#8613817306227#0516,1304847764729.5e4bca85c33fa6605ffc9a5c2eb94e62.,
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
from C4C4.site,60020,1304820199467
2011-05-08 17:52:47,309 DEBUG org.apache.hadoop.hbase.zookeeper.ZKUtil:
master:60000-0x22fcd582836003d Retrieved 125 byte(s) of data from znode
/hbase/unassigned/4418fb197685a21f77e151e401cf8b66 and set watcher;
region=ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.,
server=C4C4.site,60020,1304820199467, state=RS_ZK_REGION_CLOSED
2011-05-08 17:52:47,388 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Handling new unassigned node:
/hbase/unassigned/4418fb197685a21f77e151e401cf8b66
(region=ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.,
server=C4C4.site,60020,1304820199467, state=RS_ZK_REGION_CLOSED)
2011-05-08 17:52:47,388 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Handling transition=RS_ZK_REGION_CLOSED, server=C4C4.site,60020,1304820199467,
region=4418fb197685a21f77e151e401cf8b66
2011-05-08 17:52:47,388 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Received CLOSED for region 4418fb197685a21f77e151e401cf8b66 from server
C4C4.site,60020,1304820199467 but region was in the state null and not in
expected PENDING_CLOSE or CLOSING states
2011-05-08 17:52:47,397 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting 4418fb197685a21f77e151e401cf8b66 on
serverName=C4C4.site,60020,1304820199467, load=(requests=0, regions=123,
usedHeap=4097, maxHeap=8175)
2011-05-08 17:52:47,398 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Starting unassignment of region
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
(offlining)
2011-05-08 17:52:47,400 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Starting unassignment of region
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
(offlining)
2011-05-08 17:52:47,501 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting 4418fb197685a21f77e151e401cf8b66 on
serverName=C4C4.site,60020,1304820199467, load=(requests=0, regions=123,
usedHeap=4097, maxHeap=8175)
2011-05-08 17:52:47,501 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting 4418fb197685a21f77e151e401cf8b66 on
serverName=C4C4.site,60020,1304820199467, load=(requests=0, regions=123,
usedHeap=4097, maxHeap=8175)
2011-05-08 17:52:47,702 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Attempting to unassign region
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
but it is already in transition (PENDING_CLOSE)
2011-05-08 17:52:47,704 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Server serverName=C4C4.site,60020,1304820199467, load=(requests=0, regions=123,
usedHeap=4097, maxHeap=8175) returned
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: Received close for
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
but we are not serving it for 4418fb197685a21f77e151e401cf8b66
2011-05-08 17:52:47,816 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting 4418fb197685a21f77e151e401cf8b66 on
serverName=C4C4.site,60020,1304820199467, load=(requests=0, regions=123,
usedHeap=4097, maxHeap=8175)
2011-05-08 17:52:47,819 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Starting unassignment of region
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
(offlining)
2011-05-08 17:52:47,820 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting 4418fb197685a21f77e151e401cf8b66 on
serverName=C4C4.site,60020,1304820199467, load=(requests=0, regions=123,
usedHeap=4097, maxHeap=8175)
2011-05-08 17:52:47,820 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting 4418fb197685a21f77e151e401cf8b66 on
serverName=C4C4.site,60020,1304820199467, load=(requests=0, regions=123,
usedHeap=4097, maxHeap=8175)
2011-05-08 17:52:47,820 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Starting unassignment of region
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
(offlining)
2011-05-08 17:52:47,820 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Starting unassignment of region
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
(offlining)
2011-05-08 17:52:47,821 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting 4418fb197685a21f77e151e401cf8b66 on
serverName=C4C4.site,60020,1304820199467, load=(requests=0, regions=123,
usedHeap=4097, maxHeap=8175)
2011-05-08 17:52:47,821 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Server serverName=C4C4.site,60020,1304820199467, load=(requests=0, regions=123,
usedHeap=4097, maxHeap=8175) returned
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: Received close for
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
but we are not serving it for 4418fb197685a21f77e151e401cf8b66
2011-05-08 17:52:47,821 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting 4418fb197685a21f77e151e401cf8b66 on
serverName=C4C4.site,60020,1304820199467, load=(requests=0, regions=123,
usedHeap=4097, maxHeap=8175)
2011-05-08 17:52:47,822 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting 4418fb197685a21f77e151e401cf8b66 on
serverName=C4C4.site,60020,1304820199467, load=(requests=0, regions=123,
usedHeap=4097, maxHeap=8175)
2011-05-08 17:52:48,172 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Starting unassignment of region
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
(offlining)
2011-05-08 17:52:48,173 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Starting unassignment of region
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
(offlining)
2011-05-08 17:52:48,174 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Attempting to unassign region
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
but it is already in transition (PENDING_CLOSE)
2011-05-08 17:52:48,174 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Starting unassignment of region
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
(offlining)
2011-05-08 17:52:48,175 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Server serverName=C4C4.site,60020,1304820199467, load=(requests=0, regions=123,
usedHeap=4097, maxHeap=8175) returned
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: Received close for
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
but we are not serving it for 4418fb197685a21f77e151e401cf8b66
2011-05-08 17:52:48,175 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Starting unassignment of region
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
(offlining)
2011-05-08 17:52:48,176 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Starting unassignment of region
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
(offlining)
2011-05-08 17:52:48,176 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Attempting to unassign region
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
but it is already in transition (PENDING_CLOSE)
2011-05-08 17:52:48,177 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Attempting to unassign region
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
but it is already in transition (PENDING_CLOSE)
2011-05-08 17:52:48,178 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Attempting to unassign region
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
but it is already in transition (PENDING_CLOSE)
2011-05-08 17:52:48,178 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Attempting to unassign region
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
but it is already in transition (PENDING_CLOSE)
2011-05-08 17:52:48,178 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Attempting to unassign region
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
but it is already in transition (PENDING_CLOSE)
2011-05-08 17:52:48,690 INFO org.apache.hadoop.hbase.catalog.MetaEditor:
Deleted daughter reference
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.,
qualifier=splitB, from parent
ufdr,2011050812#8613817306227#0516,1304845660567.8e9a3b05abe1c3a692999cf5e8dfd9dd.
2011-05-08 17:55:51,787 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Regions in transition timed out:
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
state=PENDING_CLOSE, ts=1304848368173
2011-05-08 17:55:51,787 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Region has been PENDING_CLOSE for too long, running forced unassign again on
region=ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
2011-05-08 17:55:51,787 DEBUG org.apache.hadoop.hbase.zookeeper.ZKUtil:
master:60000-0x22fcd582836003d Set watcher on existing znode
/hbase/unassigned/4418fb197685a21f77e151e401cf8b66
2011-05-08 17:56:01,774 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Regions in transition timed out:
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
state=PENDING_CLOSE, ts=1304848368173
2011-05-08 17:56:01,774 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Region has been PENDING_CLOSE for too long, running forced unassign again on
region=ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
2011-05-08 17:56:01,775 DEBUG org.apache.hadoop.hbase.zookeeper.ZKUtil:
master:60000-0x22fcd582836003d Set watcher on existing znode
/hbase/unassigned/4418fb197685a21f77e151e401cf8b66
2011-05-08 17:56:11,774 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Regions in transition timed out:
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
state=PENDING_CLOSE, ts=1304848368173
2011-05-08 17:56:11,774 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Region has been PENDING_CLOSE for too long, running forced unassign again on
region=ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
2011-05-08 17:56:11,775 DEBUG org.apache.hadoop.hbase.zookeeper.ZKUtil:
master:60000-0x22fcd582836003d Set watcher on existing znode
/hbase/unassigned/4418fb197685a21f77e151e401cf8b66
2011-05-08 17:56:21,775 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Regions in transition timed out:
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
state=PENDING_CLOSE, ts=1304848368173
2011-05-08 17:5
