[
https://issues.apache.org/jira/browse/KUDU-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adar Dembo updated KUDU-2149:
-----------------------------
Status: In Review (was: In Progress)
> New failure detector implementation can lead to election stacking
> -----------------------------------------------------------------
>
> Key: KUDU-2149
> URL: https://issues.apache.org/jira/browse/KUDU-2149
> Project: Kudu
> Issue Type: Bug
> Components: consensus
> Affects Versions: 1.5.0
> Reporter: Adar Dembo
> Assignee: Adar Dembo
> Priority: Critical
>
> A new failure detector (FD) implementation was merged in commit 21b0f3d and
> is part of Kudu 1.5. One of the key changes is that the detection logic runs
> on a reactor thread rather than on a dedicated per-replica thread. But,
> because reactor threads are shared, the election started in the event of a
> failure must be thunked to the Raft thread pool (starting an election means
> casting a vote, which generally means performing IO, which is verboten on a
> reactor thread).
> By thunking, the FD immediately rearms; the previous implementation did not
> do this. If there's a lot of outstanding IO (i.e. during an election storm
> across thousands of tablets), it's possible for the FD to fire again while
> the first election task is still waiting to cast its vote. The new election
> task will try to acquire the consensus lock and block on it (it's held by the
> first election task). And so on. When the original IO finally completes, all
> of the follow-on elections will get unblocked at the same time.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
