[
https://issues.apache.org/jira/browse/HADOOP-8217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242908#comment-13242908
]
Bikas Saha commented on HADOOP-8217:
------------------------------------
Ah. The confusion was caused by
bq. 4(failure case): if NN1.transitionToStandby() times out or fails, the
non-graceful fencing is initiated (same as in existing HA code for the last
several months)
It seemed like non-graceful fencing existed in HA code for several months. You
were referring to fencing methods.
I think the piece that was missing from the solution was
bq. 4(failure case): if NN1.transitionToStandby() times out or fails, the
non-graceful fencing is initiated
I think this is what confused me (and perhaps Hari too) into thinking that NN1
would behave badly. On HDFS-2185 I have commented on ZKFC state diagram missing
the arcs for transitionToActive/Standby() failing. It looks like ZKFC does
takes specific action there. Its just missing from the transition diagram
posted on that jira.
In this case, the problem is happening because FC2 is calling
NN1.transitionToStandby() and then FC1 is calling NN1.transitionToActive().
I would like to question the value of FC2 calling NN1.transitionToStandby() in
general. FC1 on NN1 is supposed to call NN1.transitionToStandby() because thats
is FC1's responsibility upon losing the leader lock.
Secondly, based on the recent work done to add breadcrumbs to the
ActiveStandbyElector, FC2 is going to fence NN1 if NN1 has not gracefully given
up the lock, which is clearly the case here. So the problem is already solved
unless I am mistaken.
> Edge case split-brain race in ZK-based auto-failover
> ----------------------------------------------------
>
> Key: HADOOP-8217
> URL: https://issues.apache.org/jira/browse/HADOOP-8217
> Project: Hadoop Common
> Issue Type: Bug
> Components: auto-failover, ha
> Affects Versions: 0.24.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hadoop-8217-testcase.txt
>
>
> As discussed in HADOOP-8206, the current design for automatic failover has
> the following race:
> - ZKFC1 gets active lock
> - ZKFC1 is about to send transitionToActive() and machine freezes (eg GC
> pause + swapping)
> - ZKFC1 loses its ZK lock, ZKFC2 gets ZK lock
> - ZKFC2 calls transitionToStandby on NN1, and transitions NN2 to active
> - ZKFC1 wakes up from pause, calls transitionToActive(), now we have a bad
> situation
> This is rare, since it requires ZKFC1 to freeze longer than its ZK session
> timeout, but worth fixing, since the results can be disastrous.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
