[
https://issues.apache.org/jira/browse/HBASE-4792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150948#comment-13150948
]
Jean-Daniel Cryans commented on HBASE-4792:
-------------------------------------------
It mirrors what's done RS-side, there's no limit there either. We could fix
that too if we want.
The reason there's no sleep is because the RS is sleeping and we don't want to
hit the same race another time.
> SplitRegionHandler doesn't care if it deletes the znode or not, leaves the
> parent region stuck offline
> ------------------------------------------------------------------------------------------------------
>
> Key: HBASE-4792
> URL: https://issues.apache.org/jira/browse/HBASE-4792
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Priority: Critical
> Fix For: 0.92.0, 0.94.0
>
> Attachments: HBASE-4792-0.92.patch
>
>
> Saw this on a little test cluster, really easy to trigger.
> First the master log:
> {quote}
> 2011-11-15 22:28:57,900 DEBUG
> org.apache.hadoop.hbase.master.handler.SplitRegionHandler: Handling SPLIT
> event for e5be6551c8584a6a1065466e520faf4e; deleting node
> 2011-11-15 22:28:57,900 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:62003-0x132f043bbde08c1 Deleting existing unassigned node for
> e5be6551c8584a6a1065466e520faf4e that is in expected state RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:57,975 WARN org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:62003-0x132f043bbde08c1 Attempting to delete unassigned node in
> RS_ZK_REGION_SPLIT state but after verifying state, we got a version mismatch
> 2011-11-15 22:28:57,975 INFO
> org.apache.hadoop.hbase.master.handler.SplitRegionHandler: Handled SPLIT
> report);
> parent=TestTable,0001355346,1321396080924.e5be6551c8584a6a1065466e520faf4e.
> daughter
> a=TestTable,0001355346,1321396132414.df9b549eb594a1f8728608a2a431224a.daughter
> b=TestTable,0001368082,1321396132414.de861596db4337dc341138f26b9c8bc2.
> ...
> 2011-11-15 22:28:58,052 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_SPLIT, server=sv4r28s44,62023,1321395865619,
> region=e5be6551c8584a6a1065466e520faf4e
> 2011-11-15 22:28:58,052 WARN
> org.apache.hadoop.hbase.master.AssignmentManager: Region
> e5be6551c8584a6a1065466e520faf4e not found on server
> sv4r28s44,62023,1321395865619; failed processing
> 2011-11-15 22:28:58,052 WARN
> org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region
> e5be6551c8584a6a1065466e520faf4e from server sv4r28s44,62023,1321395865619
> but it doesn't exist anymore, probably already processed its split
> (repeated forever)
> {quote}
> The master processes the split but when it calls ZKAssign.deleteNode it
> doesn't check the boolean that's returned. In this case it was false. So for
> the master the split was completed, but for the region server it's another
> story:
> {quote}
> 2011-11-15 22:28:57,661 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> regionserver:62023-0x132f043bbde08d3 Attempting to transition node
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLITTING to
> RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:57,775 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> regionserver:62023-0x132f043bbde08d3 Successfully transitioned node
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLITTING to
> RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:57,775 INFO
> org.apache.hadoop.hbase.regionserver.SplitTransaction: Still waiting on the
> master to process the split for e5be6551c8584a6a1065466e520faf4e
> 2011-11-15 22:28:57,876 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> regionserver:62023-0x132f043bbde08d3 Attempting to transition node
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLIT to RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:57,967 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> regionserver:62023-0x132f043bbde08d3 Successfully transitioned node
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLIT to RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:58,067 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> regionserver:62023-0x132f043bbde08d3 Attempting to transition node
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLIT to RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:58,108 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> regionserver:62023-0x132f043bbde08d3 Successfully transitioned node
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLIT to RS_ZK_REGION_SPLIT
> (printed forever)
> {quote}
> Since the znode isn't really deleted, it thinks the master just haven't got
> to process its region thus waits which leaves the region *unavailable*.
> We need to just retry the delete master-side ASAP since the RS will wait
> 100ms between retries.
> At the same time, it would be nice if ZKAssign.deleteNode always printed out
> the name of the region in its messages because it took me a while to see that
> the delete didn't take affect while looking at a grep.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
