[
https://issues.apache.org/jira/browse/HDDS-17323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17323:
------------------------------
Epic Link: HDDS-17321
> OM admin client nests two retry loops and retries AccessControlException
> ------------------------------------------------------------------------
>
> Key: HDDS-17323
> URL: https://issues.apache.org/jira/browse/HDDS-17323
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Manager
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
>
> {{OMAdminProtocolClientSideImpl.createProxyForSingleOM}} wraps a proxy that
> already has a connection retry policy
> ({{ozone.om.admin.protocol.max.retries}}=20,
> {{ozone.om.admin.protocol.wait.between.retries}}=1s) in a second
> {{RetryProxy}} with a hardcoded 10 retries and 1s sleep. The outer policy
> retries every exception, including {{AccessControlException}}. By
> calculation, for one peer OM:
> * connection refused: about 4 minutes (11 attempts of 21 connection tries
> each);
> * connect timeout (host drops packets): about 2.8 hours (11 attempts of 46 ×
> 20s, from {{ipc.client.connect.max.retries.on.timeouts}} and
> {{ipc.client.connect.timeout}});
> * the OM accepts the connection but does not answer: about 2.75 hours (11
> attempts of {{ozone.om.client.rpc.timeout}}, 15 minutes).
> Callers include:
> * {{OMFinalizeUpgradeRequestBase.preExecute}}, through
> {{validatePeerOmVersionsBeforeFinalize}}, which checks the peer OMs one by
> one on an OM RPC handler thread. A down peer OM can hold that handler for
> hours, long after the finalize client gave up.
> * {{OzoneManager.checkConfigBeforeBootstrap}}
> * {{ozone admin om decommission}}, {{ozone admin om snapshot defrag}} and
> {{ozone repair om compact}}
> Proposed fix: drop the redundant outer retries (or at least fail on
> {{AccessControlException}}), and bound the peer version check, e.g. no
> connect timeout retries and a shorter RPC timeout.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
