[
https://issues.apache.org/jira/browse/HADOOP-10251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14583083#comment-14583083
]
Vinayakumar B commented on HADOOP-10251:
----------------------------------------
bq. Answer 2.when start only one ZKFC and NN ，the NN can be staying in ACTIVE
for long time.
Yes, fine.
Its because, patch from this issue, is not merged properly to your code.
In your source code, {{becomeActive()}} doesn't have line {{serviceState =
HAServiceState.ACTIVE;}}.
{code}private synchronized void becomeActive() throws ServiceFailedException {
LOG.info("Trying to make " + localTarget + " active...");
try
{
HAServiceProtocolHelper.transitionToActive(localTarget.getProxy( conf,
FailoverController.getRpcTimeoutToNewActive(conf)), createReqInfo());
String msg = "Successfully transitioned " + localTarget + " to active state";
LOG.info(msg);
recordActiveAttempt(new ActiveAttemptRecord(true, msg));
} catch (Throwable t) {
String msg = "Couldn't make " + localTarget + " active";
LOG.fatal(msg, t);
recordActiveAttempt(new ActiveAttemptRecord(false, msg + "\n" +
StringUtils.stringifyException(t)));
if (t instanceof ServiceFailedException) {
throw (ServiceFailedException)t; }
else {
throw new ServiceFailedException("Couldn't transition to active", t);
}
/*
TODO:
we need to make sure that if we get fenced and then quickly restarted,
none of these calls will retry across the restart boundary
perhaps the solution is that, whenever the nn starts, it gets a unique
ID, and when we start becoming active, we record it, and then any future
calls use the same ID
*/
}
}{code}
So if the previous state of NameNode is not STANDBY, then it will stay for long
time. But if its trasitioned from STANDBY, it will continously switch.
Add {{serviceState = HAServiceState.ACTIVE;}} in {{becomeActive()}} after
{{LOG.info(msg);}}, everything will be fine.
> Both NameNodes could be in STANDBY State if SNN network is unstable
> -------------------------------------------------------------------
>
> Key: HADOOP-10251
> URL: https://issues.apache.org/jira/browse/HADOOP-10251
> Project: Hadoop Common
> Issue Type: Bug
> Components: ha
> Affects Versions: 2.2.0
> Reporter: Vinayakumar B
> Assignee: Vinayakumar B
> Priority: Critical
> Fix For: 2.5.0
>
> Attachments: HADOOP-10251.patch, HADOOP-10251.patch,
> HADOOP-10251.patch, HADOOP-10251.patch, HADOOP-10251.patch
>
>
> Following corner scenario happened in one of our cluster.
> 1. NN1 was Active and NN2 was Standby
> 2. NN2 machine's network was slow
> 3. NN1 got shutdown.
> 4. NN2 ZKFC got the notification and trying to check for old active for
> fencing. (This took little more time, again due to slow network)
> 5. In between, NN1 got restarted by our automatic monitoring, and ZKFC made
> it Active.
> 6. Now NN2 ZKFC got Old Active as NN1 and it did graceful fencing of NN1 to
> STANBY.
> 7. Before writing ActiveBreadCrumb to ZK, NN2 ZKFC got session timeout and
> got shutdown before making NN2 Active.
> *Now cluster having both NameNodes as STANDBY.*
> NN1 ZKFC still thinks that its nameNode is in Active state.
> NN2 ZKFC waiting for election.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
