[
https://issues.apache.org/jira/browse/RATIS-2657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz-wo Sze updated RATIS-2657:
------------------------------
Component/s: Leader
(was: server)
> Leader re-initiates InstallSnapshot against a not-ready follower with no
> backoff
> --------------------------------------------------------------------------------
>
> Key: RATIS-2657
> URL: https://issues.apache.org/jira/browse/RATIS-2657
> Project: Ratis
> Issue Type: Bug
> Components: Leader
> Affects Versions: 3.2.2
> Reporter: Anony Mouse
> Priority: Minor
>
> h3. Summary
> When a follower answers {{ServerNotReadyException}}, the leader re-initiates
> the snapshot send immediately and repeatedly, with no backoff. Over a
> 15-second window in one measured run this produced thousands of rejected
> attempts for what should have been a couple of install events.
> h3. Measurements
> From one reviewed run (5-node group, one follower whose division was
> unavailable - see RATIS-2656, which is how we produced a reliably not-ready
> follower):
> * *4,876* {{ServerNotReadyException}} traces on the follower - {{is not in
> [STARTING, RUNNING]: current state is CLOSED}}
> * *130* leader-side {{send snapshot}} re-initiations
> * *2,216* {{Decrease nextIndex}} bounces
> for approximately *2* real install events. An earlier, independent
> observation of the same behavior during an ordinary follower reboot recorded
> roughly 400 attempts in 15.6 seconds, and a reviewer on a slower machine
> reproduced that at about four times the rate.
> h3. Why it may be worth addressing on its own
> The situation converges once the follower becomes ready, so this is not a
> liveness defect at 5 nodes - the cluster keeps serving throughout. But the
> retry loop spends leader CPU and network precisely when a follower is least
> able to absorb it, it makes logs on both sides hard to read during exactly
> the incident an operator is investigating, and the cost presumably scales
> with group size and snapshot size. A bounded retry or a short backoff on
> {{ServerNotReadyException}} would be cheap.
> Note that we produced the not-ready follower via RATIS-2656, but the behavior
> here is independent of it: any follower that is slow to reach {{RUNNING}} - a
> cold start, a slow disk, a long reload - should attract the same treatment.
> h3. Context
> Found by a Jepsen-style harness for Ratis:
> https://github.com/hooji/ratis-jepsen - {{jobs/07-snapshot-churn/}} for the
> first observation, {{reviews/08-membership-churn/}} for the quantified run
> and the corrected interpretation, with run stores preserved in-repo.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
