Boyi Dai created ZOOKEEPER-5131:
-----------------------------------
Summary: LeaderElectionSupport fails on a withdrawn offer and
leaves its own offer behind
Key: ZOOKEEPER-5131
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-5131
Project: ZooKeeper
Issue Type: Bug
Components: recipes
Affects Versions: 3.9.7
Reporter: Boyi Dai
{{LeaderElectionSupport.start()}} creates an ephemeral sequential offer, then
{{determineElectionStatus()}} lists the offers under the root node and reads
each one with a separate {{getData}} call in {{toLeaderOffers()}}. If a
competing offer is withdrawn between the {{getChildren}} and the {{getData}}
(its owner calls {{stop()}} or its session ends), {{getData}} fails with
{{NoNode}} and {{start()}} moves to FAILED. That is an ordinary race, not a
real failure.
The failed election also keeps its own offer in ZooKeeper. It no longer watches
anything, but later contenders see it first and wait behind it, so no
participant may become leader until the failed instance calls {{stop()}} or its
session ends. Retrying {{start()}} on the same instance creates a second offer
and overwrites the field that held the first, so {{stop()}} can no longer
remove the first one.
This was found through code analysis, and I reproduced both parts with tests
that delete a competing offer between the listing and the read, and that make
the election fail after its offer was created.
*Steps to reproduce:*
# Create a competing offer {{n_0}} under the election root.
# Start {{LeaderElectionSupport}}; it creates {{n_1}} and lists {{[n_0, n_1]}}.
# Delete {{n_0}} before the election reads it.
# The election moves to FAILED, and {{n_1}} is left in ZooKeeper.
*Expected:* a withdrawn competing offer is skipped, and an election that does
fail removes its own offer.
*Proposed fix:*
* Skip an offer that disappears while the offers are read (this also keeps
{{getLeaderHostName()}} from failing in the same race).
* When the election fails, remove its own offer before sending the failed event.
* If our own offer was removed by someone else, report a failure instead of
stopping without any event.
* Run a watch-triggered election under the same lock as {{start()}} and
{{stop()}}, so a concurrent {{stop()}} can't be followed by a failed event.
I have a patch with tests and will open a PR.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)