Boyi Dai created ZOOKEEPER-5131:
-----------------------------------

             Summary: LeaderElectionSupport fails on a withdrawn offer and 
leaves its own offer behind
                 Key: ZOOKEEPER-5131
                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-5131
             Project: ZooKeeper
          Issue Type: Bug
          Components: recipes
    Affects Versions: 3.9.7
            Reporter: Boyi Dai


{{LeaderElectionSupport.start()}} creates an ephemeral sequential offer, then 
{{determineElectionStatus()}} lists the offers under the root node and reads 
each one with a separate {{getData}} call in {{toLeaderOffers()}}. If a 
competing offer is withdrawn between the {{getChildren}} and the {{getData}} 
(its owner calls {{stop()}} or its session ends), {{getData}} fails with 
{{NoNode}} and {{start()}} moves to FAILED. That is an ordinary race, not a 
real failure.

The failed election also keeps its own offer in ZooKeeper. It no longer watches 
anything, but later contenders see it first and wait behind it, so no 
participant may become leader until the failed instance calls {{stop()}} or its 
session ends. Retrying {{start()}} on the same instance creates a second offer 
and overwrites the field that held the first, so {{stop()}} can no longer 
remove the first one.

This was found through code analysis, and I reproduced both parts with tests 
that delete a competing offer between the listing and the read, and that make 
the election fail after its offer was created.

*Steps to reproduce:*
# Create a competing offer {{n_0}} under the election root.
# Start {{LeaderElectionSupport}}; it creates {{n_1}} and lists {{[n_0, n_1]}}.
# Delete {{n_0}} before the election reads it.
# The election moves to FAILED, and {{n_1}} is left in ZooKeeper.

*Expected:* a withdrawn competing offer is skipped, and an election that does 
fail removes its own offer.

*Proposed fix:*
* Skip an offer that disappears while the offers are read (this also keeps 
{{getLeaderHostName()}} from failing in the same race).
* When the election fails, remove its own offer before sending the failed event.
* If our own offer was removed by someone else, report a failure instead of 
stopping without any event.
* Run a watch-triggered election under the same lock as {{start()}} and 
{{stop()}}, so a concurrent {{stop()}} can't be followed by a failed event.

I have a patch with tests and will open a PR.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to