[
https://issues.apache.org/jira/browse/HADOOP-9608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670925#comment-13670925
]
Todd Lipcon commented on HADOOP-9608:
-------------------------------------
Right, the full sequence of events as I understand them was:
- NNA is active, NNB is standby.
- NNB's hardware is getting retired, so it is removed, and a new node NNC
brought up.
- Configs updated with NNC's address, but NNA's ZKFC is not modified
- Manual failover occurred to NNC
- Some time later, NNC fails, and the automatic failover does not succeed
because NNA's ZKFC still has NNB's address.
> ZKFC should abort if it sees an unrecognized NN become active
> -------------------------------------------------------------
>
> Key: HADOOP-9608
> URL: https://issues.apache.org/jira/browse/HADOOP-9608
> Project: Hadoop Common
> Issue Type: Bug
> Components: ha
> Affects Versions: 3.0.0
> Reporter: Todd Lipcon
>
> We recently had an issue where one NameNode and ZKFC was updated to a new
> configuration/IP address but the ZKFC on the other node was not rebooted.
> Then, next time a failover occurred, the second ZKFC was not able to become
> active because the data in the ActiveBreadCrumb didn't match the data in its
> own configuration:
> {code}
> org.apache.hadoop.ha.ActiveStandbyElector: Exception handling the winning of
> election
> java.lang.IllegalArgumentException: Unable to determine service address for
> namenode 'XXXX'
> {code}
> To prevent this from happening, whenever the ZKFC sees a new NN become
> active, it should check that it's properly able to instantiate a
> ServiceTarget for it, and if not, abort (since this ZKFC wouldn't be able to
> handle a failover successfully)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
