Siyao Meng created HDDS-17318:
---------------------------------
Summary: SCM client retries non-retriable server errors, safemode
wait ignores its timeout
Key: HDDS-17318
URL: https://issues.apache.org/jira/browse/HDDS-17318
Project: Apache Ozone
Issue Type: Improvement
Components: SCM Client
Reporter: Siyao Meng
Assignee: Siyao Meng
*SCM client retries errors that will not change.* {{SCMHAUtils.getRetryAction}}
fails over and retries any exception that is not in its known lists, including
an ordinary error returned by SCM. With the defaults
({{hdds.scmclient.max.retry.timeout=10m}},
{{hdds.scmclient.failover.retry.interval=2s}}) that is 300 attempts, about 10
minutes. For example, {{ozone admin containerbalancer stop}} when the balancer
is not running, or {{containerbalancer start}} with an invalid option, prints
"Retrying" for 10 minutes before showing the error.
SCM already converts the errors worth retrying (not leader, leader not ready,
Ratis reconfiguration, and so on) to {{ServerNotLeaderException}},
{{RetriableWithFailOverException}} or {{RetriableWithNoFailoverException}}, and
Ratis timeouts arrive as {{SCMException}}, which already fails without retry.
So the client can fail immediately on any other exception returned by SCM.
Connection failures and timeouts on the client side are still retried.
*{{ozone admin safemode wait -t}} does not honor the timeout.* The deadline is
only checked between calls to SCM, and each call can take the full SCM client
retry budget. Its "SCM is not available (yet?)" branch only catches
{{InterruptedException}}, so an error from SCM ends the command instead of
retrying until the deadline.
Proposed fix: fail fast on non-retriable exceptions returned by SCM, and run
the {{inSafeMode}} call with the remaining time as its deadline, retrying on
errors until the timeout.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
