Siyao Meng created HDDS-17323:
---------------------------------
Summary: OM admin client nests two retry loops and retries
AccessControlException
Key: HDDS-17323
URL: https://issues.apache.org/jira/browse/HDDS-17323
Project: Apache Ozone
Issue Type: Bug
Components: Ozone Manager
Reporter: Siyao Meng
Assignee: Siyao Meng
{{OMAdminProtocolClientSideImpl.createProxyForSingleOM}} wraps a proxy that
already has a connection retry policy
({{ozone.om.admin.protocol.max.retries}}=20,
{{ozone.om.admin.protocol.wait.between.retries}}=1s) in a second {{RetryProxy}}
with a hardcoded 10 retries and 1s sleep. The outer policy retries every
exception, including {{AccessControlException}}. By calculation, for one peer
OM:
* connection refused: about 4 minutes (11 attempts of 21 connection tries each);
* connect timeout (host drops packets): about 2.8 hours (11 attempts of 46 ×
20s, from {{ipc.client.connect.max.retries.on.timeouts}} and
{{ipc.client.connect.timeout}});
* the OM accepts the connection but does not answer: about 2.75 hours (11
attempts of {{ozone.om.client.rpc.timeout}}, 15 minutes).
Callers include:
* {{OMFinalizeUpgradeRequestBase.preExecute}}, through
{{validatePeerOmVersionsBeforeFinalize}}, which checks the peer OMs one by one
on an OM RPC handler thread. A down peer OM can hold that handler for hours,
long after the finalize client gave up.
* {{OzoneManager.checkConfigBeforeBootstrap}}
* {{ozone admin om decommission}}, {{ozone admin om snapshot defrag}} and
{{ozone repair om compact}}
Proposed fix: drop the redundant outer retries (or at least fail on
{{AccessControlException}}), and bound the peer version check, e.g. no connect
timeout retries and a shorter RPC timeout.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
