ivandika3 opened a new pull request, #11238:
URL: https://github.com/apache/ozone/pull/11238
## What changes were proposed in this pull request?
We need reduce the following SCM client configuration so that OM read and
write critical path is not blocked for too long when SCM is unresponsive
- hdds.scmclient.rpc.timeout
- ipc.client.connect.timeout
- ipc.client.connect.max.retries.on.timeouts
However, OzoneManager instantiates quite a lot of SCM client and each of
them share a single configuration although they have different latency and
retry requirement
Critical paths (low latency and fail fast)
- scmContainerClient (used in read critical path)
- scmBlockClient (used in write critical path)
- scmTopologyClient is also part of the scmBlockClient
SCM security clients (tolerates high latency and unbounded timeout)
- scmSecurityClient
- secretKeyClient
OM delete client (tolerates high latency and timeout due to the large
payload)
- scmBlockClient
Additionally, they are all stored in the same ClientCache entry. ClientCache
is a Map<SocketFactory, Client> and the standard implementation SocketFactory
returned in NetUtils#getDefaultSocketFactory in StandardSocketFactory that has
the following hashCode
```java
@Override
public int hashCode() {
return this.getClass().hashCode();
}
```
So this means that even if we use different two configurations, only the
first configured client will be stored in the ClientCache and the subsequent
one will reuse the cache entry. Therefore, we cannot simply create a separate
configuration with different timeout.
The ClientCache#getClient also documented this decision
```
// Construct & cache client. The configuration is only used for timeout,
// and Clients have connection pools. So we can either (a) lose some
// connection pooling and leak sockets, or (b) use the same timeout for all
// configurations. Since the IPC is usually intended globally, not
// per-job, we choose (a).
```
Since we need two different timeout, it conflicts with the ClientCache
mechanism.
Therefore, we need another solution to split the two different clients into
two different SocketFactory.
In another note, the Hadoop client design decisions seems to be suited only
for user Hadoop client (15 minutes might make sense for a single Hadoop
client). However, Ozone decided to reuse the Hadoop client into our critical
path without revisiting whether the default timeout makes sense in this
context. So in the future, we need to check whether OM should use Hadoop RPC
client for its critical path or whether we need to implement a high performance
for OM and SCM.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-16382
## How was this patch tested?
UT.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
