Closing the loop - what's the status on this? Can one of you open a JIRA and provide a patch for this?



Flavio Junqueira wrote:
Hi Henry, Apologies for the the delay. Your observation sounds right to me. Here is how I'm reading it; let me know if it makes sense.

If everyone votes for 3 in the second round and 3 has crashed, then in countVotes we will remove all votes to 3 and there will be no vote left. In such a case, there will be no winner as a result of the call to countVotes and lookForLeader won't change the current vote ( This is a situation in which we are stuck.

Does it sound reasonable to add an "else" to the "if" statement of to reset the vote? This modification would implementing resetting the vote when countVotes returns no winner, which should happen only when the replica itself votes for a dead leader.


On Oct 28, 2009, at 7:44 AM, Henry Robinson wrote:

[ Sending this direct since the Apache mailserver is rejecting my e-mails at the moment ]

As I understand it, 1 and 2 receive a vote for 3 in the first round, which causes them to vote for 3 in the second round. So in the second round, all votes cast are for 3. But 3 has died, so all votes for it are discounted. 1 and 2 continue to vote for 3 ad infinitum, never resetting their vote.

Does this sound plausible, or am I missing something?


On Tue, Oct 27, 2009 at 3:48 PM, Flavio Junqueira <> wrote: Hi Henry, I don't understand how 1 and 2 do not end up electing 2 in your situation. If they exclude 3 in countVotes, then countVotes will end up returning 2 and not 3, assuming there is a vote for 2. What am I missing?

The problem with QuorumPeer you're pointing at was also an issue with the FLE tests, and I couldn't see an easy way around it other than timing out and restarting leader election.


On Oct 27, 2009, at 6:35 AM, Henry Robinson wrote:

I've been working on adding a TCPResponderThread to the leader election
process so that if a deployment needs to be TCP only, it can be and still
use all election types. Testing this has exposed what might be a race
condition in the leader election code that prevents a leader from being

Here's the behaviour I see in LETest occasionally. With three nodes (reduced from 30 for ease of debugging), node 3 gets elected before either node 1 or node 2 finish their election (there is one round where each node that 3 has the highest id, and then 3 completes its second round by receiving votes for
itself from 1 and 2, but 1 and 2 do not receive votes from 3).

Now 3 is killed by the test harness. 1 and 2 are still voting for it, but
every time they try, the vote tally excludes 3 since it hasn't been heard
from. They then spin round the voting process, unable to reset their vote. I
expect that the heartbeat mechanism in a running QuorumPeer takes care of
this when the leader is lost, but the associated QuorumPeers aren't running.

If this is the case, then there is a simple fix to reset the nodes vote to
themselves if they are voting for a node that hasn't been heard from. I
don't know why using TCP instead of UDP for the responder thread is
exacerbating this (and we can't rule out my introducing a bug :)); but as
it's a race condition the different timings associated with waiting on a TCP
socket might just be enough to expose the issue.

Can someone verify this might be possible / figure out what I missed?


Reply via email to