Siyao Meng created HDDS-17127:
---------------------------------
Summary: SCM retries checkpoint download from the leader without
backoff
Key: HDDS-17127
URL: https://issues.apache.org/jira/browse/HDDS-17127
Project: Apache Ozone
Issue Type: Sub-task
Components: SCM HA
Reporter: Siyao Meng
When {{SCMStateMachine#notifyInstallSnapshotFromLeader}} fails to download the
checkpoint, the future completes with null and Ratis sends the next install
snapshot notification right away. There is no backoff and no limit, so a
persistent failure turns into a tight loop. In the HDDS-16924 reproduction, two
followers made ~12,000 download attempts in about 2 minutes, each logging two
stack traces, and the leader logged an "Error streaming SCM DB checkpoint"
stack trace for every one.
*Proposal:* back off between failed download attempts (e.g. exponential with a
cap), keep one download in flight per follower, and rate-limit the error logs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
