[
https://issues.apache.org/jira/browse/MESOS-3280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14909536#comment-14909536
]
Neil Conway commented on MESOS-3280:
------------------------------------
To followup on the bug in the auto-initialization code (first bullet above),
there's a race condition between the log recovery (auto-initialization)
protocol and the coordinator election protocol:
* to elect the coordinator, we try to pass an implicit promise (note that
there's no retry mechanism)
* to recover the log, we do a two-phase broadcast (see RecoverProtocolProcess),
where each node goes from EMPTY => STARTING => VOTING
* if a node in EMPTY or STARTING state receives a promise request, it silently
ignores it
Moreover, AFAICS there is no synchronization between starting the log recovery
protocol and doing coordinator election: coordinator election happens as soon
as we detect we're the Zk leader (RegistrarProcess::recover(), which calls
LogWriterProcess::start(), which tries to be elected as the coordinator),
whereas log recovery/auto-init actually starts earlier (in main() in
master/main.cpp). We wait on the `recovering` promise *locally* before starting
coordinator election at the Zk leader, but that doesn't mean that log recovery
has finished at a quorum of other nodes.
I'll attach a patch with a test case that makes the race condition more likely
by having 2 of the 3 nodes sleep before transitioning from STARTING => VOTING.
I'll also attach a log of an execution that shows the problem; note that you
need to annotate the replicated log code with a bunch of extra LOG()s to see
when messages are ignored (this could also be improved).
There's a few different ways we can fix the problem: e.g., by adding a retry to
the coordinator election protocol, or by ensuring we have a quorum of VOTING
nodes before trying to elect a coordinator (the latter approach seems like it
would be quite racy, though). I'll propose a fix shortly.
Note that there's another possible problem here: depending on the order in
which messages in the log recovery protocol are observed, a node might actually
transition from EMPTY => STARTING => RECOVERING, at which point it will do the
catchup protocol. Per talking with [~jieyu], this seems unexpected, and may be
problematic. I haven't found a reproducible test case yet, but I'll followup
with Jie.
> Master fails to access replicated log after network partition
> -------------------------------------------------------------
>
> Key: MESOS-3280
> URL: https://issues.apache.org/jira/browse/MESOS-3280
> Project: Mesos
> Issue Type: Bug
> Components: master, replicated log
> Affects Versions: 0.23.0
> Environment: Zookeeper version 3.4.5--1
> Reporter: Bernd Mathiske
> Assignee: Neil Conway
> Labels: mesosphere
> Attachments: rep-log-startup-race-test-1.patch
>
>
> In a 5 node cluster with 3 masters and 2 slaves, and ZK on each node, when a
> network partition is forced, all the masters apparently lose access to their
> replicated log. The leading master halts. Unknown reasons, but presumably
> related to replicated log access. The others fail to recover from the
> replicated log. Unknown reasons. This could have to do with ZK setup, but it
> might also be a Mesos bug.
> This was observed in a Chronos test drive scenario described in detail here:
> https://github.com/mesos/chronos/issues/511
> With setup instructions here:
> https://github.com/mesos/chronos/issues/508
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
