Siyao Meng created HDDS-17316:
---------------------------------
Summary: Cap the wait between retries when loading the initial
secret key from SCM
Key: HDDS-17316
URL: https://issues.apache.org/jira/browse/HDDS-17316
Project: Apache Ozone
Issue Type: Improvement
Components: Security
Reporter: Siyao Meng
Assignee: Siyao Meng
At startup, OM and Datanode load the current secret key from SCM in
{{DefaultSecretKeySignerClient.loadInitialSecretKey}}. While SCM has not
initialized secret keys yet ({{SECRET_KEY_NOT_INITIALIZED}}), it retries up to
100 times with exponential backoff that restarts every 10 retries. The base
wait goes 2s, 4s, ..., 1024s, randomized by +/-50%, so a single wait can be up
to about 25 minutes and the total about 5.7 hours. The code comment ("1 2 4 8
... 512", "around 200min") understates both.
So once SCM finishes initializing its keys, an OM or Datanode can stay stuck
for many more minutes before trying again.
Proposed fix: keep the exponential backoff but cap the base wait at 16s (2s,
4s, 8s, 16s, 16s, ...), so a single wait is at most 24s and the total for 100
retries is around 26 minutes.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
