Anony Mouse created RATIS-2657:
----------------------------------
Summary: Leader re-initiates InstallSnapshot against a not-ready
follower with no backoff
Key: RATIS-2657
URL: https://issues.apache.org/jira/browse/RATIS-2657
Project: Ratis
Issue Type: Bug
Components: server
Affects Versions: 3.2.2
Reporter: Anony Mouse
h3. Summary
When a follower answers {{ServerNotReadyException}}, the leader re-initiates
the snapshot send immediately and repeatedly, with no backoff. Over a 15-second
window in one measured run this produced thousands of rejected attempts for
what should have been a couple of install events.
h3. Measurements
>From one reviewed run (5-node group, one follower whose division was
>unavailable - see RATIS-2656, which is how we produced a reliably not-ready
>follower):
* *4,876* {{ServerNotReadyException}} traces on the follower - {{is not in
[STARTING, RUNNING]: current state is CLOSED}}
* *130* leader-side {{send snapshot}} re-initiations
* *2,216* {{Decrease nextIndex}} bounces
for approximately *2* real install events. An earlier, independent observation
of the same behavior during an ordinary follower reboot recorded roughly 400
attempts in 15.6 seconds, and a reviewer on a slower machine reproduced that at
about four times the rate.
h3. Why it may be worth addressing on its own
The situation converges once the follower becomes ready, so this is not a
liveness defect at 5 nodes - the cluster keeps serving throughout. But the
retry loop spends leader CPU and network precisely when a follower is least
able to absorb it, it makes logs on both sides hard to read during exactly the
incident an operator is investigating, and the cost presumably scales with
group size and snapshot size. A bounded retry or a short backoff on
{{ServerNotReadyException}} would be cheap.
Note that we produced the not-ready follower via RATIS-2656, but the behavior
here is independent of it: any follower that is slow to reach {{RUNNING}} - a
cold start, a slow disk, a long reload - should attract the same treatment.
h3. Context
Found by a Jepsen-style harness for Ratis:
https://github.com/hooji/ratis-jepsen - {{jobs/07-snapshot-churn/}} for the
first observation, {{reviews/08-membership-churn/}} for the quantified run and
the corrected interpretation, with run stores preserved in-repo.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
