[
https://issues.apache.org/jira/browse/HDFS-16283?focusedWorklogId=788968&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-788968
]
ASF GitHub Bot logged work on HDFS-16283:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 08/Jul/22 12:54
Start Date: 08/Jul/22 12:54
Worklog Time Spent: 10m
Work Description: ZanderXu commented on code in PR #4524:
URL: https://github.com/apache/hadoop/pull/4524#discussion_r916788569
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterClientProtocol.java:
##########
@@ -759,14 +764,49 @@ public boolean mkdirs(String src, FsPermission masked,
boolean createParent)
}
}
+ private Map<String, FederationNamespaceInfo> getAvailableNamespaces()
+ throws IOException {
+ Map<String, FederationNamespaceInfo> allAvailableNamespaces =
+ new HashMap<>();
+ namenodeResolver.getNamespaces().forEach(
+ k -> allAvailableNamespaces.put(k.getNameserviceId(), k));
+ return allAvailableNamespaces;
+ }
+
+ /**
+ * Try to get a list of FederationNamespaceInfo for renewLease RPC.
+ */
+ private List<FederationNamespaceInfo> getRewLeaseNSs(List<String> namespaces)
+ throws IOException {
+ if (namespaces == null || namespaces.isEmpty()) {
+ return new ArrayList<>(namenodeResolver.getNamespaces());
+ }
+ List<FederationNamespaceInfo> result = new ArrayList<>();
+ Map<String, FederationNamespaceInfo> allAvailableNamespaces =
+ getAvailableNamespaces();
+ for (String namespace : namespaces) {
+ if (!allAvailableNamespaces.containsKey(namespace)) {
+ return new ArrayList<>(namenodeResolver.getNamespaces());
Review Comment:
`namenodeResolver.getNamespaces()` is a hashSet, I want to a List so that we
can use `invokeSingle` method to forward this rpc when there is only one
namespace.
```
List<FederationNamespaceInfo> nss = getRenewLeaseNSs(namespaces);
if (nss.size() == 1) {
rpcClient.invokeSingle(nss.get(0).getNameserviceId(), method);
} else {
rpcClient.invokeConcurrent(nss, method, false, false);
}
```
Of course, `Set` can also achieve this goal.
Issue Time Tracking
-------------------
Worklog Id: (was: 788968)
Time Spent: 6h (was: 5h 50m)
> RBF: improve renewLease() to call only a specific NameNode rather than make
> fan-out calls
> -----------------------------------------------------------------------------------------
>
> Key: HDFS-16283
> URL: https://issues.apache.org/jira/browse/HDFS-16283
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf
> Reporter: Aihua Xu
> Assignee: Aihua Xu
> Priority: Major
> Labels: pull-request-available
> Attachments: RBF_ improve renewLease() to call only a specific
> NameNode rather than make fan-out calls.pdf
>
> Time Spent: 6h
> Remaining Estimate: 0h
>
> Currently renewLease() against a router will make fan-out to all the
> NameNodes. Since renewLease() call is so frequent and if one of the NameNodes
> are slow, then eventually the router queues are blocked by all renewLease()
> and cause router degradation.
> We will make a change in the client side to keep track of NameNode Id in
> additional to current fileId so routers understand which NameNodes the client
> is renewing lease against.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
