On Sat, May 14, 2011 at 7:08 PM, Gaojinchao <[email protected]> wrote:
> I try to make a patch and process it.
OK. What are you trying to do in the patch? (Turns out my issue was
different to what you see).
> I want to send a zk message again when close region is timeout.
> I try to reproduce and verify it. But it is difficulty.
Yes, this stuff is tricky. Its not easy making a test. Have you seen
TestZKBasedOpenCloseRegion and TestMasterFailover. These do messing
about with zk state. Perhaps they help?
> Do you give me some help and review it.
>
Did you change code in the below? If so, can you send a patch only?
Or better, attach it to a JIRA? Patch is good because it is the
difference between what is hbase and what you have added. Otherwise,
it takes a bit of work figuring what you have added.
Thank you Gao,
St.Ack
> Thanks.
>
> case PENDING_CLOSE:
> LOG.info("Region has been PENDING_CLOSE for too " +
> "long, running forced unassign again on region=" +
> regionInfo.getRegionNameAsString());
> try {
> // If the server got the RPC, it will transition the node
> // to CLOSING, so only do something here if no node exists
> if (!ZKUtil.watchAndCheckExists(watcher,
> ZKAssign.getNodeName(watcher,
> regionInfo.getEncodedName()))) {
> // Queue running of an unassign -- do actual unassign
> // outside of the regionsInTransition lock.
> unassigns.add(regionInfo);
> }
> else
> {
>
> // I add some code
> try {
> String node = ZKAssign.getNodeName(watcher,
> regionInfo.getEncodedName());
> Stat stat = new Stat();
> RegionTransitionData data =
> ZKAssign.getDataNoWatch(watcher,
> node, stat);
> if (data == null) {
> LOG.warn("Data is null, node " + node + " no
> longer exists");
> break;
> }
> if (data.getEventType() !=
> EventType.RS_ZK_REGION_CLOSED) {
> LOG.debug("Region has transitioned to CLOEING,
> allowing " +
> "watched event handlers to process");
> break;
> }
>
> //In this case, Region server has ClOSED
> data = new RegionTransitionData(
> EventType.RS_ZK_REGION_CLOSED,
> regionInfo.getRegionName(),
> master.getServerName());
> if (ZKUtil.setData(watcher, node, data.getBytes(),
> stat.getVersion()+1 )) {
>
> // Node is now closed, let's trigger another close
> handler
> LOG.info("Try to finish closed region=" +
> regionInfo.getRegionNameAsString() + "again" );
> }
>
> } catch (KeeperException ke) {
> LOG.error("Unexpected ZK exception timing out
> PENDING_CLOSE region",
> ke);
> break;
> }
>
> }
> } catch (NoNodeException e) {
> LOG.debug("Node no longer existed so not forcing another "
> +
> "unassignment");
> } catch (KeeperException e) {
> LOG.warn("Unexpected ZK exception timing out a region " +
> "close", e);
> }
> break;
>
> -----邮件原件-----
> 发件人: [email protected] [mailto:[email protected]] 代表 Stack
> 发送时间: 2011年5月10日 1:38
> 收件人: [email protected]
> 主题: Re: Table can't disable
>
> I'm looking into this Gao. Something similar seems to have happened
> here over the w/e. Let me look at our systems first and then I'l be
> back to you on this below. Thanks,
> St.Ack
>
> On Sun, May 8, 2011 at 4:58 AM, Gaojinchao <[email protected]> wrote:
>> Today I test Hbase verison 0.90.3
>> I seems like some bugs
>> 1, if node exists and node state is RS_ZK_REGION_CLOSED
>> We should call ClosedRegionHandle to close region. In that case the reigon
>> has closed by region server
>>
>> case PENDING_CLOSE:
>> LOG.info("Region has been PENDING_CLOSE for too " +
>> "long, running forced unassign again on region=" +
>> regionInfo.getRegionNameAsString());
>> try {
>> // If the server got the RPC, it will transition the node
>> // to CLOSING, so only do something here if no node exists
>> if (!ZKUtil.watchAndCheckExists(watcher,
>> ZKAssign.getNodeName(watcher,
>> regionInfo.getEncodedName()))) {
>> // Queue running of an unassign -- do actual unassign
>> // outside of the regionsInTransition lock.
>> unassigns.add(regionInfo);
>> }
>> else
>> {
>> //It need to process for node
>> state is RS_ZK_REGION_CLOSED
>> }
>> } catch (NoNodeException e) {
>> LOG.debug("Node no longer existed so not forcing another
>> " +
>> "unassignment");
>> } catch (KeeperException e) {
>> LOG.warn("Unexpected ZK exception timing out a region " +
>> "close", e);
>> }
>> 2, other It seems like zk message losing or there is some bugs for disabling
>> table when region server is splitting region.
>> above is the logs. I need to keep digging.
>>
>> Shell logs:
>> hbase(main):003:0*
>> hbase(main):004:0* disable 'ufdr'
>> 11/05/08 18:20:40 DEBUG zookeeper.ZKUtil: hconnection-0x2fcd58283c0037 Set
>> watcher on existing znode /hbase/root-region-server
>> 11/05/08 18:20:40 DEBUG zookeeper.ZKUtil: hconnection-0x2fcd58283c0037
>> Retrieved 15 byte(s) of data from znode /hbase/root-region-server and set
>> watcher; C4C5.site:60020
>> 11/05/08 18:53:19 DEBUG zookeeper.ZKUtil: hconnection-0x2fcd58283c0037
>> Retrieved 9 byte(s) of data from znode /hbase/table/ufdr; data=DISABLING
>>
>> ERROR: org.apache.hadoop.hbase.RegionException: Retries exhausted, it took
>> too long to wait for the table ufdr to be disabled.
>>
>>
>> Region server logs:
>>
>> 2011-05-08 17:42:44,862 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
>> Instantiated
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> 2011-05-08 17:42:45,468 DEBUG org.apache.hadoop.hbase.regionserver.Store:
>> loaded
>> hdfs://C4C1:9000/hbase/ufdr/4418fb197685a21f77e151e401cf8b66/value/3442771824694350714.8e9a3b05abe1c3a692999cf5e8dfd9dd,
>> isReference=true, isBulkLoadResult=false, seqid=627830,
>> majorCompaction=false
>> 2011-05-08 17:42:45,471 INFO org.apache.hadoop.hbase.regionserver.HRegion:
>> Onlined
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.;
>> next sequenceid=627831
>> 2011-05-08 17:42:45,471 DEBUG
>> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction
>> requested for
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> because Region has references on open; priority=9, compaction queue size=40
>> 2011-05-08 17:42:45,476 INFO org.apache.hadoop.hbase.catalog.MetaEditor:
>> Added daughter
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> in region .META.,,1, serverInfo=C4C4.site,60020,1304820199467
>> 2011-05-08 17:42:45,476 INFO
>> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Region split, META
>> updated, and report to master.
>> Parent=ufdr,2011050812#8613817306227#0516,1304845660567.8e9a3b05abe1c3a692999cf5e8dfd9dd.,
>> new regions:
>> ufdr,2011050812#8613817306227#0516,1304847764729.5e4bca85c33fa6605ffc9a5c2eb94e62.,
>>
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66..
>> Split took 0sec
>> 2011-05-08 17:44:25,731 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Received close region:
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> 2011-05-08 17:48:11,066 INFO org.apache.hadoop.hbase.regionserver.HRegion:
>> Starting compaction on region
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> 2011-05-08 17:48:11,067 INFO org.apache.hadoop.hbase.regionserver.Store:
>> Started compaction of 1 file(s) in cf=value, hasReferences=true, into
>> hdfs://C4C1:9000/hbase/ufdr/4418fb197685a21f77e151e401cf8b66/.tmp,
>> seqid=627830, totalSize=892.2m
>> 2011-05-08 17:48:11,067 DEBUG org.apache.hadoop.hbase.regionserver.Store:
>> Compacting
>> hdfs://C4C1:9000/hbase/ufdr/4418fb197685a21f77e151e401cf8b66/value/3442771824694350714.8e9a3b05abe1c3a692999cf5e8dfd9dd-hdfs://C4C1:9000/hbase/ufdr/8e9a3b05abe1c3a692999cf5e8dfd9dd/value/3442771824694350714-top,
>> keycount=2843998, bloomtype=NONE, size=892.2m
>> 2011-05-08 17:48:32,340 INFO org.apache.hadoop.hbase.regionserver.Store:
>> Completed compaction of 1 file(s), new
>> file=hdfs://C4C1:9000/hbase/ufdr/4418fb197685a21f77e151e401cf8b66/value/4640085674257062204,
>> size=446.2m; total size for store is 446.2m
>> 2011-05-08 17:48:32,340 INFO org.apache.hadoop.hbase.regionserver.HRegion:
>> completed compaction on region
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> after 21sec
>> 2011-05-08 17:52:46,626 DEBUG
>> org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing
>> close of
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> 2011-05-08 17:52:46,627 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
>> regionserver:60020-0x12fcd58280c0003-0x12fcd58280c0003 Creating unassigned
>> node for 4418fb197685a21f77e151e401cf8b66 in a CLOSING state
>> 2011-05-08 17:52:46,986 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
>> Closing
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.:
>> disabling compactions & flushes
>> 2011-05-08 17:52:46,986 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
>> Updates disabled for region
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> 2011-05-08 17:52:46,986 INFO org.apache.hadoop.hbase.regionserver.HRegion:
>> Closed
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> 2011-05-08 17:52:46,986 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
>> regionserver:60020-0x12fcd58280c0003-0x12fcd58280c0003 Attempting to
>> transition node 4418fb197685a21f77e151e401cf8b66 from RS_ZK_REGION_CLOSING
>> to RS_ZK_REGION_CLOSED
>> 2011-05-08 17:52:47,052 DEBUG org.apache.hadoop.hbase.zookeeper.ZKUtil:
>> regionserver:60020-0x12fcd58280c0003-0x12fcd58280c0003 Retrieved 125 byte(s)
>> of data from znode /hbase/unassigned/4418fb197685a21f77e151e401cf8b66;
>> data=region=ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.,
>> server=C4C4.site,60020,1304820199467, state=RS_ZK_REGION_CLOSING
>> 2011-05-08 17:52:47,143 DEBUG
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher:
>> regionserver:60020-0x12fcd58280c0003-0x12fcd58280c0003 Received ZooKeeper
>> Event, type=NodeDataChanged, state=SyncConnected,
>> path=/hbase/unassigned/4418fb197685a21f77e151e401cf8b66
>> 2011-05-08 17:52:47,143 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
>> regionserver:60020-0x12fcd58280c0003-0x12fcd58280c0003 Successfully
>> transitioned node 4418fb197685a21f77e151e401cf8b66 from RS_ZK_REGION_CLOSING
>> to RS_ZK_REGION_CLOSED
>> 2011-05-08 17:52:47,143 DEBUG
>> org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed
>> region
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> 2011-05-08 17:52:47,696 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Received close region:
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> 2011-05-08 17:52:47,696 WARN
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Received close for
>> region we are not serving; 4418fb197685a21f77e151e401cf8b66
>> 2011-05-08 17:52:47,814 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Received close region:
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> 2011-05-08 17:52:47,814 WARN
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Received close for
>> region we are not serving; 4418fb197685a21f77e151e401cf8b66
>> 2011-05-08 17:52:48,167 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Received close region:
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> 2011-05-08 17:52:48,167 WARN
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Received close for
>> region we are not serving; 4418fb197685a21f77e151e401cf8b66
>>
>>
>> Hmaster logs:
>> 2011-05-08 17:42:45,514 INFO org.apache.hadoop.hbase.master.ServerManager:
>> Received REGION_SPLIT:
>> ufdr,2011050812#8613817306227#0516,1304845660567.8e9a3b05abe1c3a692999cf5e8dfd9dd.:
>> Daughters;
>> ufdr,2011050812#8613817306227#0516,1304847764729.5e4bca85c33fa6605ffc9a5c2eb94e62.,
>>
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> from C4C4.site,60020,1304820199467
>> 2011-05-08 17:43:37,599 DEBUG
>> org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
>> region
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> (offlining)
>> 2011-05-08 17:43:45,525 INFO org.apache.hadoop.hbase.master.ServerManager:
>> Received REGION_SPLIT:
>> ufdr,2011050812#8613817306227#0516,1304845660567.8e9a3b05abe1c3a692999cf5e8dfd9dd.:
>> Daughters;
>> ufdr,2011050812#8613817306227#0516,1304847764729.5e4bca85c33fa6605ffc9a5c2eb94e62.,
>>
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> from C4C4.site,60020,1304820199467
>> 2011-05-08 17:44:25,745 DEBUG
>> org.apache.hadoop.hbase.master.AssignmentManager: Sent CLOSE to
>> serverName=C4C4.site,60020,1304820199467, load=(requests=0, regions=123,
>> usedHeap=4097, maxHeap=8175) for region
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> 2011-05-08 17:44:45,530 INFO org.apache.hadoop.hbase.master.ServerManager:
>> Received REGION_SPLIT:
>> ufdr,2011050812#8613817306227#0516,1304845660567.8e9a3b05abe1c3a692999cf5e8dfd9dd.:
>> Daughters;
>> ufdr,2011050812#8613817306227#0516,1304847764729.5e4bca85c33fa6605ffc9a5c2eb94e62.,
>>
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> from C4C4.site,60020,1304820199467
>> 2011-05-08 17:45:45,542 INFO org.apache.hadoop.hbase.master.ServerManager:
>> Received REGION_SPLIT:
>> ufdr,2011050812#8613817306227#0516,1304845660567.8e9a3b05abe1c3a692999cf5e8dfd9dd.:
>> Daughters;
>> ufdr,2011050812#8613817306227#0516,1304847764729.5e4bca85c33fa6605ffc9a5c2eb94e62.,
>>
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> from C4C4.site,60020,1304820199467
>> 2011-05-08 17:46:45,303 WARN
>> org.apache.hadoop.hbase.master.AssignmentManager: Overwriting
>> 4418fb197685a21f77e151e401cf8b66 on
>> serverName=C4C4.site,60020,1304820199467, load=(requests=0, regions=123,
>> usedHeap=4097, maxHeap=8175)
>> 2011-05-08 17:46:45,538 INFO org.apache.hadoop.hbase.master.ServerManager:
>> Received REGION_SPLIT:
>> ufdr,2011050812#8613817306227#0516,1304845660567.8e9a3b05abe1c3a692999cf5e8dfd9dd.:
>> Daughters;
>> ufdr,2011050812#8613817306227#0516,1304847764729.5e4bca85c33fa6605ffc9a5c2eb94e62.,
>>
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> from C4C4.site,60020,1304820199467
>> 2011-05-08 17:47:45,548 INFO org.apache.hadoop.hbase.master.ServerManager:
>> Received REGION_SPLIT:
>> ufdr,2011050812#8613817306227#0516,1304845660567.8e9a3b05abe1c3a692999cf5e8dfd9dd.:
>> Daughters;
>> ufdr,2011050812#8613817306227#0516,1304847764729.5e4bca85c33fa6605ffc9a5c2eb94e62.,
>>
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> from C4C4.site,60020,1304820199467
>> 2011-05-08 17:48:45,545 INFO org.apache.hadoop.hbase.master.ServerManager:
>> Received REGION_SPLIT:
>> ufdr,2011050812#8613817306227#0516,1304845660567.8e9a3b05abe1c3a692999cf5e8dfd9dd.:
>> Daughters;
>> ufdr,2011050812#8613817306227#0516,1304847764729.5e4bca85c33fa6605ffc9a5c2eb94e62.,
>>
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> from C4C4.site,60020,1304820199467
>> 2011-05-08 17:49:46,108 INFO org.apache.hadoop.hbase.master.ServerManager:
>> Received REGION_SPLIT:
>> ufdr,2011050812#8613817306227#0516,1304845660567.8e9a3b05abe1c3a692999cf5e8dfd9dd.:
>> Daughters;
>> ufdr,2011050812#8613817306227#0516,1304847764729.5e4bca85c33fa6605ffc9a5c2eb94e62.,
>>
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> from C4C4.site,60020,1304820199467
>> 2011-05-08 17:50:46,105 INFO org.apache.hadoop.hbase.master.ServerManager:
>> Received REGION_SPLIT:
>> ufdr,2011050812#8613817306227#0516,1304845660567.8e9a3b05abe1c3a692999cf5e8dfd9dd.:
>> Daughters;
>> ufdr,2011050812#8613817306227#0516,1304847764729.5e4bca85c33fa6605ffc9a5c2eb94e62.,
>>
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> from C4C4.site,60020,1304820199467
>> 2011-05-08 17:51:46,117 INFO org.apache.hadoop.hbase.master.ServerManager:
>> Received REGION_SPLIT:
>> ufdr,2011050812#8613817306227#0516,1304845660567.8e9a3b05abe1c3a692999cf5e8dfd9dd.:
>> Daughters;
>> ufdr,2011050812#8613817306227#0516,1304847764729.5e4bca85c33fa6605ffc9a5c2eb94e62.,
>>
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> from C4C4.site,60020,1304820199467
>> 2011-05-08 17:52:46,112 INFO org.apache.hadoop.hbase.master.ServerManager:
>> Received REGION_SPLIT:
>> ufdr,2011050812#8613817306227#0516,1304845660567.8e9a3b05abe1c3a692999cf5e8dfd9dd.:
>> Daughters;
>> ufdr,2011050812#8613817306227#0516,1304847764729.5e4bca85c33fa6605ffc9a5c2eb94e62.,
>>
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> from C4C4.site,60020,1304820199467
>> 2011-05-08 17:52:47,309 DEBUG org.apache.hadoop.hbase.zookeeper.ZKUtil:
>> master:60000-0x22fcd582836003d Retrieved 125 byte(s) of data from znode
>> /hbase/unassigned/4418fb197685a21f77e151e401cf8b66 and set watcher;
>> region=ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.,
>> server=C4C4.site,60020,1304820199467, state=RS_ZK_REGION_CLOSED
>> 2011-05-08 17:52:47,388 DEBUG
>> org.apache.hadoop.hbase.master.AssignmentManager: Handling new unassigned
>> node: /hbase/unassigned/4418fb197685a21f77e151e401cf8b66
>> (region=ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.,
>> server=C4C4.site,60020,1304820199467, state=RS_ZK_REGION_CLOSED)
>> 2011-05-08 17:52:47,388 DEBUG
>> org.apache.hadoop.hbase.master.AssignmentManager: Handling
>> transition=RS_ZK_REGION_CLOSED, server=C4C4.site,60020,1304820199467,
>> region=4418fb197685a21f77e151e401cf8b66
>> 2011-05-08 17:52:47,388 WARN
>> org.apache.hadoop.hbase.master.AssignmentManager: Received CLOSED for region
>> 4418fb197685a21f77e151e401cf8b66 from server C4C4.site,60020,1304820199467
>> but region was in the state null and not in expected PENDING_CLOSE or
>> CLOSING states
>> 2011-05-08 17:52:47,397 WARN
>> org.apache.hadoop.hbase.master.AssignmentManager: Overwriting
>> 4418fb197685a21f77e151e401cf8b66 on
>> serverName=C4C4.site,60020,1304820199467, load=(requests=0, regions=123,
>> usedHeap=4097, maxHeap=8175)
>> 2011-05-08 17:52:47,398 DEBUG
>> org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
>> region
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> (offlining)
>> 2011-05-08 17:52:47,400 DEBUG
>> org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
>> region
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> (offlining)
>> 2011-05-08 17:52:47,501 WARN
>> org.apache.hadoop.hbase.master.AssignmentManager: Overwriting
>> 4418fb197685a21f77e151e401cf8b66 on
>> serverName=C4C4.site,60020,1304820199467, load=(requests=0, regions=123,
>> usedHeap=4097, maxHeap=8175)
>> 2011-05-08 17:52:47,501 WARN
>> org.apache.hadoop.hbase.master.AssignmentManager: Overwriting
>> 4418fb197685a21f77e151e401cf8b66 on
>> serverName=C4C4.site,60020,1304820199467, load=(requests=0, regions=123,
>> usedHeap=4097, maxHeap=8175)
>> 2011-05-08 17:52:47,702 DEBUG
>> org.apache.hadoop.hbase.master.AssignmentManager: Attempting to unassign
>> region
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> but it is already in transition (PENDING_CLOSE)
>> 2011-05-08 17:52:47,704 DEBUG
>> org.apache.hadoop.hbase.master.AssignmentManager: Server
>> serverName=C4C4.site,60020,1304820199467, load=(requests=0, regions=123,
>> usedHeap=4097, maxHeap=8175) returned
>> org.apache.hadoop.hbase.NotServingRegionException:
>> org.apache.hadoop.hbase.NotServingRegionException: Received close for
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> but we are not serving it for 4418fb197685a21f77e151e401cf8b66
>> 2011-05-08 17:52:47,816 WARN
>> org.apache.hadoop.hbase.master.AssignmentManager: Overwriting
>> 4418fb197685a21f77e151e401cf8b66 on
>> serverName=C4C4.site,60020,1304820199467, load=(requests=0, regions=123,
>> usedHeap=4097, maxHeap=8175)
>> 2011-05-08 17:52:47,819 DEBUG
>> org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
>> region
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> (offlining)
>> 2011-05-08 17:52:47,820 WARN
>> org.apache.hadoop.hbase.master.AssignmentManager: Overwriting
>> 4418fb197685a21f77e151e401cf8b66 on
>> serverName=C4C4.site,60020,1304820199467, load=(requests=0, regions=123,
>> usedHeap=4097, maxHeap=8175)
>> 2011-05-08 17:52:47,820 WARN
>> org.apache.hadoop.hbase.master.AssignmentManager: Overwriting
>> 4418fb197685a21f77e151e401cf8b66 on
>> serverName=C4C4.site,60020,1304820199467, load=(requests=0, regions=123,
>> usedHeap=4097, maxHeap=8175)
>> 2011-05-08 17:52:47,820 DEBUG
>> org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
>> region
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> (offlining)
>> 2011-05-08 17:52:47,820 DEBUG
>> org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
>> region
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> (offlining)
>> 2011-05-08 17:52:47,821 WARN
>> org.apache.hadoop.hbase.master.AssignmentManager: Overwriting
>> 4418fb197685a21f77e151e401cf8b66 on
>> serverName=C4C4.site,60020,1304820199467, load=(requests=0, regions=123,
>> usedHeap=4097, maxHeap=8175)
>> 2011-05-08 17:52:47,821 DEBUG
>> org.apache.hadoop.hbase.master.AssignmentManager: Server
>> serverName=C4C4.site,60020,1304820199467, load=(requests=0, regions=123,
>> usedHeap=4097, maxHeap=8175) returned
>> org.apache.hadoop.hbase.NotServingRegionException:
>> org.apache.hadoop.hbase.NotServingRegionException: Received close for
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> but we are not serving it for 4418fb197685a21f77e151e401cf8b66
>> 2011-05-08 17:52:47,821 WARN
>> org.apache.hadoop.hbase.master.AssignmentManager: Overwriting
>> 4418fb197685a21f77e151e401cf8b66 on
>> serverName=C4C4.site,60020,1304820199467, load=(requests=0, regions=123,
>> usedHeap=4097, maxHeap=8175)
>> 2011-05-08 17:52:47,822 WARN
>> org.apache.hadoop.hbase.master.AssignmentManager: Overwriting
>> 4418fb197685a21f77e151e401cf8b66 on
>> serverName=C4C4.site,60020,1304820199467, load=(requests=0, regions=123,
>> usedHeap=4097, maxHeap=8175)
>> 2011-05-08 17:52:48,172 DEBUG
>> org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
>> region
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> (offlining)
>> 2011-05-08 17:52:48,173 DEBUG
>> org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
>> region
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> (offlining)
>> 2011-05-08 17:52:48,174 DEBUG
>> org.apache.hadoop.hbase.master.AssignmentManager: Attempting to unassign
>> region
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> but it is already in transition (PENDING_CLOSE)
>> 2011-05-08 17:52:48,174 DEBUG
>> org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
>> region
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> (offlining)
>> 2011-05-08 17:52:48,175 DEBUG
>> org.apache.hadoop.hbase.master.AssignmentManager: Server
>> serverName=C4C4.site,60020,1304820199467, load=(requests=0, regions=123,
>> usedHeap=4097, maxHeap=8175) returned
>> org.apache.hadoop.hbase.NotServingRegionException:
>> org.apache.hadoop.hbase.NotServingRegionException: Received close for
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> but we are not serving it for 4418fb197685a21f77e151e401cf8b66
>> 2011-05-08 17:52:48,175 DEBUG
>> org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
>> region
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> (offlining)
>> 2011-05-08 17:52:48,176 DEBUG
>> org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
>> region
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> (offlining)
>> 2011-05-08 17:52:48,176 DEBUG
>> org.apache.hadoop.hbase.master.AssignmentManager: Attempting to unassign
>> region
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> but it is already in transition (PENDING_CLOSE)
>> 2011-05-08 17:52:48,177 DEBUG
>> org.apache.hadoop.hbase.master.AssignmentManager: Attempting to unassign
>> region
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> but it is already in transition (PENDING_CLOSE)
>> 2011-05-08 17:52:48,178 DEBUG
>> org.apache.hadoop.hbase.master.AssignmentManager: Attempting to unassign
>> region
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> but it is already in transition (PENDING_CLOSE)
>> 2011-05-08 17:52:48,178 DEBUG
>> org.apache.hadoop.hbase.master.AssignmentManager: Attempting to unassign
>> region
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> but it is already in transition (PENDING_CLOSE)
>> 2011-05-08 17:52:48,178 DEBUG
>> org.apache.hadoop.hbase.master.AssignmentManager: Attempting to unassign
>> region
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> but it is already in transition (PENDING_CLOSE)
>> 2011-05-08 17:52:48,690 INFO org.apache.hadoop.hbase.catalog.MetaEditor:
>> Deleted daughter reference
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.,
>> qualifier=splitB, from parent
>> ufdr,2011050812#8613817306227#0516,1304845660567.8e9a3b05abe1c3a692999cf5e8dfd9dd.
>> 2011-05-08 17:55:51,787 INFO
>> org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition
>> timed out:
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> state=PENDING_CLOSE, ts=1304848368173
>> 2011-05-08 17:55:51,787 INFO
>> org.apache.hadoop.hbase.master.AssignmentManager: Region has been
>> PENDING_CLOSE for too long, running forced unassign again on
>> region=ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> 2011-05-08 17:55:51,787 DEBUG org.apache.hadoop.hbase.zookeeper.ZKUtil:
>> master:60000-0x22fcd582836003d Set watcher on existing znode
>> /hbase/unassigned/4418fb197685a21f77e151e401cf8b66
>> 2011-05-08 17:56:01,774 INFO
>> org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition
>> timed out:
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> state=PENDING_CLOSE, ts=1304848368173
>> 2011-05-08 17:56:01,774 INFO
>> org.apache.hadoop.hbase.master.AssignmentManager: Region has been
>> PENDING_CLOSE for too long, running forced unassign again on
>> region=ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> 2011-05-08 17:56:01,775 DEBUG org.apache.hadoop.hbase.zookeeper.ZKUtil:
>> master:60000-0x22fcd582836003d Set watcher on existing znode
>> /hbase/unassigned/4418fb197685a21f77e151e401cf8b66
>> 2011-05-08 17:56:11,774 INFO
>> org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition
>> timed out:
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> state=PENDING_CLOSE, ts=1304848368173
>> 2011-05-08 17:56:11,774 INFO
>> org.apache.hadoop.hbase.master.AssignmentManager: Region has been
>> PENDING_CLOSE for too long, running forced unassign again on
>> region=ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> 2011-05-08 17:56:11,775 DEBUG org.apache.hadoop.hbase.zookeeper.ZKUtil:
>> master:60000-0x22fcd582836003d Set watcher on existing znode
>> /hbase/unassigned/4418fb197685a21f77e151e401cf8b66
>> 2011-05-08 17:56:21,775 INFO
>> org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition
>> timed out:
>> ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> state=PENDING_CLOSE, ts=1304848368173
>> 2011-05-08 17:5
>>
>>
>
