[kudu-CR] WIP: KUDU-1365. Add leader pre-elections

[David Ribeiro Alves (Code Review)]

David Ribeiro Alves has posted comments on this change.

Change subject: WIP: KUDU-1365. Add leader pre-elections
..


Patch Set 2:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/4694/2/src/kudu/consensus/raft_consensus.cc
File src/kudu/consensus/raft_consensus.cc:

Line 1908: StartElection(NORMAL_ELECTION, reason);
> Yea, that looks possible, but I don't think it causes problems except for a
Just afraid that this interleaving will not be so unlikely with pre-elections. 
Since they require no IO, aren't they likely to go faster than regular 
elections? Maybe I'm missing a serialization point somewhere though.

Well we can always do nothing and see if this manifests itself in real 
scenarios. One thing we might consider if this becomes pathological (and I 
haven't fully thought this through), is to only do pre-elections when there 
isn't a real election going on. Since pre-elections are mostly for the sake of 
the partitioned/slow node that now can make sure that an election is called for 
when it comes back, pre-elections would still be helpful here, wheres if there 
just an election race where multiple replicas are trying to elect themselves 
pre-elections aren't helpful (maybe even worse?).


-- 
To view, visit http://gerrit.cloudera.org:8080/4694
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ifcfabd8c9ffe31f17ab768542a046426f656db43
Gerrit-PatchSet: 2
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon 
Gerrit-Reviewer: David Ribeiro Alves 
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy 
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon 
Gerrit-HasComments: Yes

[kudu-CR] WIP: KUDU-1365. Add leader pre-elections

[Todd Lipcon (Code Review)]

Todd Lipcon has posted comments on this change.

Change subject: WIP: KUDU-1365. Add leader pre-elections
..


Patch Set 2:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/4694/2/src/kudu/consensus/raft_consensus.cc
File src/kudu/consensus/raft_consensus.cc:

Line 1908: StartElection(NORMAL_ELECTION, reason);
> Just afraid that this interleaving will not be so unlikely with pre-electio
There's not a guaranteed serialization point, but there is some kind of 
"softer" thing that helps make this less likely: the peers that are voting will 
be holding some lock while writing their "real election" vote to disk, and that 
would make the pre-election respond with "too busy". So, the pre-election that 
happens while a real election was in progress is quite likely to get denied due 
to hitting this code path.

+1 on trying this out with various stress tests and address it if we see this 
being a problem. Thanks for calling it out, though, I hadn't considered it.


-- 
To view, visit http://gerrit.cloudera.org:8080/4694
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ifcfabd8c9ffe31f17ab768542a046426f656db43
Gerrit-PatchSet: 2
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon 
Gerrit-Reviewer: David Ribeiro Alves 
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy 
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon 
Gerrit-HasComments: Yes

[kudu-CR] WIP: KUDU-1365. Add leader pre-elections

[Todd Lipcon (Code Review)]

Todd Lipcon has posted comments on this change.

Change subject: WIP: KUDU-1365. Add leader pre-elections
..


Patch Set 2:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/4694/2/src/kudu/consensus/raft_consensus.cc
File src/kudu/consensus/raft_consensus.cc:

Line 1908: StartElection(NORMAL_ELECTION, reason);
> is there an interleaving in which we're already started a real election (bu
Yea, that looks possible, but I don't think it causes problems except for a 
potential extra election in a relatively rare case. If the pre-election for N+1 
completes before we heard from N, then it'll bump its term to N+1 and call the 
real election for N+1. Then if N completes successfully, it'll just ignore the 
result coming from the prior term in DoElectionCallback.

Any suggestion for how we should handle this case differently? I think we 
already had a similar kind of case where it was possible to start an election 
for term N, not hear back, and then start one for N+1 under load, but given we 
do the backoff thing, eventually we should be spacing them out enough to 
succeed.


-- 
To view, visit http://gerrit.cloudera.org:8080/4694
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ifcfabd8c9ffe31f17ab768542a046426f656db43
Gerrit-PatchSet: 2
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon 
Gerrit-Reviewer: David Ribeiro Alves 
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy 
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon 
Gerrit-HasComments: Yes

[kudu-CR] WIP: KUDU-1365. Add leader pre-elections

[David Ribeiro Alves (Code Review)]

David Ribeiro Alves has posted comments on this change.

Change subject: WIP: KUDU-1365. Add leader pre-elections
..


Patch Set 2:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/4694/2/src/kudu/consensus/raft_consensus.cc
File src/kudu/consensus/raft_consensus.cc:

Line 1908: StartElection(NORMAL_ELECTION, reason);
is there an interleaving in which we're already started a real election (but 
haven't won it yet) when we get here? Something like:

1. Term N (follower): send a real election for term N
2. Election callback expires again
3. Term N (follower): send a pre-election for term N+1
4. Pre-election callback from term N+1 completes, even though we are still 
waiting on the result of the real election.


-- 
To view, visit http://gerrit.cloudera.org:8080/4694
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ifcfabd8c9ffe31f17ab768542a046426f656db43
Gerrit-PatchSet: 2
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon 
Gerrit-Reviewer: David Ribeiro Alves 
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy 
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon 
Gerrit-HasComments: Yes

[kudu-CR] WIP: KUDU-1365. Add leader pre-elections

[Todd Lipcon (Code Review)]

Todd Lipcon has posted comments on this change.

Change subject: WIP: KUDU-1365. Add leader pre-elections
..


Patch Set 1:

One more TODO item here is that LeaderElectionExpBackoff needs to use a 
different count for figuring out how many leader elections have failed, since 
we are no longer bumping through terms during failed leader elections in many 
cases.

-- 
To view, visit http://gerrit.cloudera.org:8080/4694
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ifcfabd8c9ffe31f17ab768542a046426f656db43
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon 
Gerrit-Reviewer: David Ribeiro Alves 
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy 
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon 
Gerrit-HasComments: No

[kudu-CR] WIP: KUDU-1365. Add leader pre-elections

[Todd Lipcon (Code Review)]

Todd Lipcon has posted comments on this change.

Change subject: WIP: KUDU-1365. Add leader pre-elections
..


Patch Set 1:

submitted an itest x1000 loop here: 
http://dist-test.cloudera.org//job?job_id=todd.147621.11040

-- 
To view, visit http://gerrit.cloudera.org:8080/4694
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ifcfabd8c9ffe31f17ab768542a046426f656db43
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon 
Gerrit-Reviewer: David Ribeiro Alves 
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy 
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon 
Gerrit-HasComments: No

[kudu-CR] WIP: KUDU-1365. Add leader pre-elections

[Todd Lipcon (Code Review)]

Hello David Ribeiro Alves, Mike Percy,

I'd like you to do a code review.  Please visit

http://gerrit.cloudera.org:8080/4694

to review the following change.

Change subject: WIP: KUDU-1365. Add leader pre-elections
..

WIP: KUDU-1365. Add leader pre-elections

This implements the "pre-election" extension to the Raft algorithm.
The idea is that, before calling a leader election, a candidate first
sends a pre-election vote request to all voters. The voters respond
as they would have in a real vote, except they don't actually record
their vote.

WIP because this needs tests, etc.

Change-Id: Ifcfabd8c9ffe31f17ab768542a046426f656db43
---
M src/kudu/consensus/consensus.h
M src/kudu/consensus/consensus.proto
M src/kudu/consensus/leader_election-test.cc
M src/kudu/consensus/leader_election.cc
M src/kudu/consensus/leader_election.h
M src/kudu/consensus/raft_consensus.cc
M src/kudu/consensus/raft_consensus.h
M src/kudu/integration-tests/raft_consensus-itest.cc
8 files changed, 152 insertions(+), 72 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/94/4694/1
-- 
To view, visit http://gerrit.cloudera.org:8080/4694
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ifcfabd8c9ffe31f17ab768542a046426f656db43
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon 
Gerrit-Reviewer: David Ribeiro Alves 
Gerrit-Reviewer: Mike Percy