Siyao Meng created HDDS-17321:
---------------------------------
Summary: Bound client retry and timeout waits to avoid long silent
hangs
Key: HDDS-17321
URL: https://issues.apache.org/jira/browse/HDDS-17321
Project: Apache Ozone
Issue Type: Epic
Components: Ozone Client
Reporter: Siyao Meng
Assignee: Siyao Meng
Several Ozone clients wait far longer than needed, often with nothing printed,
because a retry wait grows without bound, a timeout is retried many times on
the same server, an error that will not change is retried, or a timeout setting
is read in the wrong unit. For example, a CLI command started before a single
OM is ready keeps sleeping long after OM is up, and an unreachable OM or SCM
costs about 15 minutes per connection attempt.
This epic groups fixes that keep these waits bounded and in line with the
configured values, without changing behavior on the normal path.
h3. Fixes
* HDDS-17302 OM client retry wait time grows without bound with a single OM
* HDDS-17310 OM and SCM clients retry a connect timeout on the same server for
about 15 minutes
* HDDS-17312 gRPC client read and EC write timeouts are truncated to whole
seconds
* HDDS-17314 Make hdds.heartbeat.recon.interval control the Datanode to Recon
heartbeat interval
* HDDS-17316 Cap the wait between retries when loading the initial secret key
from SCM
* HDDS-17318 SCM client retries non-retriable server errors, safemode wait
ignores its timeout
* HDDS-17320 SCM client does not recognize RetriableWithNoFailoverException
returned by SCM
h3. Candidates
* OM "leader not ready" wait is not capped, and its counter is shared by all
threads using one client
* Datanode heartbeat client to SCM still retries a connect timeout 45 times
(related: HDDS-17269)
* OM admin client nests two retry loops and retries AccessControlException
* CA list fetch ({{HAUtils.getCAListWithRetry}}) retries forever
* Data path: a dead follower adds a watch timeout to every flush (related:
HDDS-10978), and reads retry a hung nearest replica first (related: HDDS-16419)
Related: HDDS-16382, HDDS-4409
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
