goiri commented on code in PR #5991:
URL: https://github.com/apache/hadoop/pull/5991#discussion_r1307691912
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MembershipNamenodeResolver.java:
##########
@@ -478,4 +478,37 @@ private List<MembershipState>
getRecentRegistrationForQuery(
public void setRouterId(String router) {
this.routerId = router;
}
+
+ /**
+ * Shuffle cache, to ensure that the current nn will not be accessed first
next time.
+ *
+ * @param nsId name service id
+ * @param namenode namenode contexts
+ */
+ @Override
+ public synchronized void shuffleCache(String nsId, FederationNamenodeContext
namenode) {
+ cacheNS.compute(Pair.of(nsId, false), (ns, namenodeContexts) -> {
+ if (namenodeContexts != null && namenodeContexts.size() > 1) {
Review Comment:
You can do early exit with the reverse:
```
if (namenodeContexts == null || namenodeContexts.size() <= 1) {
return namenodeContexts;
}
XXXX
```
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MembershipNamenodeResolver.java:
##########
@@ -478,4 +478,37 @@ private List<MembershipState>
getRecentRegistrationForQuery(
public void setRouterId(String router) {
this.routerId = router;
}
+
+ /**
+ * Shuffle cache, to ensure that the current nn will not be accessed first
next time.
+ *
+ * @param nsId name service id
+ * @param namenode namenode contexts
+ */
+ @Override
+ public synchronized void shuffleCache(String nsId, FederationNamenodeContext
namenode) {
+ cacheNS.compute(Pair.of(nsId, false), (ns, namenodeContexts) -> {
+ if (namenodeContexts != null && namenodeContexts.size() > 1) {
+ FederationNamenodeContext firstNamenodeContext =
namenodeContexts.get(0);
+ /*
+ * 1.If the first nn in the cache is active, the active nn priority
cannot be lowered.
+ * This happens when other threads have already updated the cache.
+ *
+ * 2.If the first nn in the cache at this time is not the nn
+ * that needs to be lowered in priority, there is no need to rotate.
+ * This happens when other threads have already rotated the cache.
+ */
+ if (!firstNamenodeContext.getState().equals(ACTIVE)
+ &&
firstNamenodeContext.getRpcAddress().equals(namenode.getRpcAddress())) {
+ List<FederationNamenodeContext> rotatedNnContexts = new
ArrayList<>(namenodeContexts);
+ Collections.rotate(rotatedNnContexts, -1);
+ return rotatedNnContexts;
+ } else {
+ return namenodeContexts;
Review Comment:
Same we can reverse the if and do early return.
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MembershipNamenodeResolver.java:
##########
@@ -478,4 +478,37 @@ private List<MembershipState>
getRecentRegistrationForQuery(
public void setRouterId(String router) {
this.routerId = router;
}
+
+ /**
+ * Shuffle cache, to ensure that the current nn will not be accessed first
next time.
+ *
+ * @param nsId name service id
+ * @param namenode namenode contexts
+ */
+ @Override
+ public synchronized void shuffleCache(String nsId, FederationNamenodeContext
namenode) {
+ cacheNS.compute(Pair.of(nsId, false), (ns, namenodeContexts) -> {
+ if (namenodeContexts != null && namenodeContexts.size() > 1) {
+ FederationNamenodeContext firstNamenodeContext =
namenodeContexts.get(0);
+ /*
+ * 1.If the first nn in the cache is active, the active nn priority
cannot be lowered.
+ * This happens when other threads have already updated the cache.
+ *
+ * 2.If the first nn in the cache at this time is not the nn
+ * that needs to be lowered in priority, there is no need to rotate.
+ * This happens when other threads have already rotated the cache.
+ */
+ if (!firstNamenodeContext.getState().equals(ACTIVE)
+ &&
firstNamenodeContext.getRpcAddress().equals(namenode.getRpcAddress())) {
Review Comment:
Indetation looks funny.
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterClientRejectOverload.java:
##########
@@ -357,6 +363,83 @@ public void testNoNamenodesAvailable() throws Exception{
assertEquals(originalRouter0Failures, rpcMetrics0.getProxyOpNoNamenodes());
}
+ /**
+ * When failover occurs, the router may record that the ns has no active
namenode.
+ * Only when the router updates the cache next time can the memory status be
updated,
+ * causing the router to report NoNamenodesAvailableException for a long
time.
+ */
+ @Test
+ public void testNoNamenodesAvailableLongTimeWhenNsFailover() throws
Exception {
+ setupCluster(false, true);
+ transitionClusterNSToStandby(cluster);
+ for (RouterContext routerContext : cluster.getRouters()) {
+ // Manually trigger the heartbeat
+ Collection<NamenodeHeartbeatService> heartbeatServices = routerContext
+ .getRouter().getNamenodeHeartbeatServices();
+ for (NamenodeHeartbeatService service : heartbeatServices) {
+ service.periodicInvoke();
+ }
+ // Update service cache
+ routerContext.getRouter().getStateStore().refreshCaches(true);
+ }
+ // Record the time after the router first updated the cache
+ long firstLoadTime = Time.now();
+ List<MiniRouterDFSCluster.NamenodeContext> namenodes =
cluster.getNamenodes();
+
+ // Make sure all namenodes are in standby state
+ for (MiniRouterDFSCluster.NamenodeContext namenodeContext : namenodes) {
+ assertEquals(STANDBY.ordinal(),
Review Comment:
Doesn't this fit in one line?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
