[
https://issues.apache.org/jira/browse/HBASE-4872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13157333#comment-13157333
]
Ted Yu commented on HBASE-4872:
-------------------------------
This seems similar to HBASE-4729
Maybe we should introduce interlock between region movement and region
splitting.
> Balancer goes to move a region but its being split so we fail offlining
> because NodeExists and it crashes master
> ----------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-4872
> URL: https://issues.apache.org/jira/browse/HBASE-4872
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: stack
> Priority: Critical
> Fix For: 0.92.0
>
>
> So, offlining should deal with there being an existing znode in SPLITTING of
> SPLIT state. Here is log around the master crash.
> {code}
> // BALANCER RUNS
> 2011-11-24 07:41:18,686 INFO org.apache.hadoop.hbase.master.HMaster: balance
> hri=TestTable,0936288217,1322120470779.f681f38f85f6aad594f24f90559da051.,
> src=sv4r9s38,7003,1322110570576, dest=sv4r31s44,7003,1322110570645
> 2011-11-24 07:41:18,686 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
> region TestTable,0936288217,1322120470779.f681f38f85f6aad594f24f90559da051.
> (offlining)
> 2011-11-24 07:41:18,686 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:7001-0x133d3ee451d0000 Creating unassigned node for
> f681f38f85f6aad594f24f90559da051 in a CLOSING state
> 2011-11-24 07:41:18,702 ERROR
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/unassigned/f681f38f85f6aad594f24f90559da051 already exists and this is
> not a retry
> 2011-11-24 07:41:18,702 FATAL org.apache.hadoop.hbase.master.HMaster: Master
> server abort: loaded coprocessors are: []
> 2011-11-24 07:41:18,704 FATAL org.apache.hadoop.hbase.master.HMaster:
> Unexpected ZK exception creating node CLOSING
> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
> NodeExists for /hbase/unassigned/f681f38f85f6aad594f24f90559da051
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:110)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:637)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:459)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:441)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndWatch(ZKUtil.java:839)
> at
> org.apache.hadoop.hbase.zookeeper.ZKAssign.createNodeClosing(ZKAssign.java:568)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1766)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1732)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.balance(AssignmentManager.java:2763)
> at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:871)
> at org.apache.hadoop.hbase.master.HMaster$1.chore(HMaster.java:735)
> at org.apache.hadoop.hbase.Chore.run(Chore.java:67)
> at java.lang.Thread.run(Thread.java:662)
> 2011-11-24 07:41:18,706 INFO org.apache.hadoop.hbase.master.HMaster: Aborting
> // HERE THE SPLIT SHOWS UP ON MASTER
> 2011-11-24 07:41:19,310 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_SPLIT, server=sv4r9s38,7003,1322110570576,
> region=f681f38f85f6aad594f24f90559da051
> 2011-11-24 07:41:19,311 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region
> f681f38f85f6aad594f24f90559da051 from server sv4r9s38,7003,1322110570576 but
> region was not first in SPLITTING state; continuing
> 2011-11-24 07:41:19,311 DEBUG
> org.apache.hadoop.hbase.master.handler.SplitRegionHandler: Handling SPLIT
> event for f681f38f85f6aad594f24f90559da051; deleting node
> 2011-11-24 07:41:19,311 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:7001-0x133d3ee451d0000 Deleting existing unassigned node for
> f681f38f85f6aad594f24f90559da051 that is in expected state RS_ZK_REGION_SPLIT
> 2011-11-24 07:41:19,401 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:7001-0x133d3ee451d0000 Successfully deleted unassigned node for region
> f681f38f85f6aad594f24f90559da051 in expected state RS_ZK_REGION_SPLIT
> 2011-11-24 07:41:19,401 INFO
> org.apache.hadoop.hbase.master.handler.SplitRegionHandler: Handled SPLIT
> report);
> parent=TestTable,0936288217,1322120470779.f681f38f85f6aad594f24f90559da051.
> daughter
> a=TestTable,0936288217,1322120475927.98808642d25058c930fc9e5974f3715d.daughter
> b=TestTable,0936327449,1322120475927.00ba03057ddc6d3868fb6ef349cb9a2c.
> {code}
> Here is regionserver side of things at time of split:
> {code}
> 2011-11-24 07:41:15,927 DEBUG
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Split requested for
> TestTable,0936288217,1322120470779.f681f38f85f6aad594f24f90559da051..
> compaction_queue=(1:0), split_queue=1
> 2011-11-24 07:41:15,928 INFO
> org.apache.hadoop.hbase.regionserver.SplitTransaction: Starting split of
> region TestTable,0936288217,1322120470779.f681f38f85f6aad594f24f90559da051.
> 2011-11-24 07:41:15,928 DEBUG
> org.apache.hadoop.hbase.regionserver.SplitTransaction:
> regionserver:7003-0x133d3ee451d000c Creating ephemeral node for
> f681f38f85f6aad594f24f90559da051 in SPLITTING state
> 2011-11-24 07:41:15,945 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> regionserver:7003-0x133d3ee451d000c Attempting to transition node
> f681f38f85f6aad594f24f90559da051 from RS_ZK_REGION_SPLITTING to
> RS_ZK_REGION_SPLITTING
> 2011-11-24 07:41:15,953 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> regionserver:7003-0x133d3ee451d000c Successfully transitioned node
> f681f38f85f6aad594f24f90559da051 from RS_ZK_REGION_SPLITTING to
> RS_ZK_REGION_SPLITTING
> ...
> 2011-11-24 07:41:19,401 WARN org.apache.hadoop.hbase.zookeeper.ZKAssign:
> regionserver:7003-0x133d3ee451d000c Attempt to transition the unassigned node
> for f681f38f85f6aad594f24f90559da051 from RS_ZK_REGION_SPLIT to
> RS_ZK_REGION_SPLIT failed, the node existed and was in the expected state but
> then when setting data it no longer existed
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
