[
https://issues.apache.org/jira/browse/HDFS-13522?focusedWorklogId=779684&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-779684
]
ASF GitHub Bot logged work on HDFS-13522:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 08/Jun/22 22:06
Start Date: 08/Jun/22 22:06
Worklog Time Spent: 10m
Work Description: hchaverri commented on code in PR #4127:
URL: https://github.com/apache/hadoop/pull/4127#discussion_r892900410
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/resources/hdfs-rbf-default.xml:
##########
@@ -781,4 +781,20 @@
(delete the source path directly) and skip (skip both trash and
deletion).
</description>
</property>
+
+ <property>
+ <name>dfs.federation.router.observer.read.enable</name>
+ <value>false</value>
+ <description>
+ Enable observer read in router. This is value is used across all
nameservices
Review Comment:
There's an extra "is" on this sentence
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MembershipNamenodeResolver.java:
##########
@@ -371,24 +405,37 @@ private List<MembershipState>
getRecentRegistrationForQuery(
membershipStore.getNamenodeRegistrations(request);
List<MembershipState> memberships = response.getNamenodeMemberships();
- if (!addExpired || !addUnavailable) {
- Iterator<MembershipState> iterator = memberships.iterator();
- while (iterator.hasNext()) {
- MembershipState membership = iterator.next();
- if (membership.getState() == EXPIRED && !addExpired) {
- iterator.remove();
- } else if (membership.getState() == UNAVAILABLE && !addUnavailable) {
- iterator.remove();
- }
+ List<MembershipState> observerMemberships = new ArrayList<>();
+ Iterator<MembershipState> iterator = memberships.iterator();
+ while (iterator.hasNext()) {
+ MembershipState membership = iterator.next();
+ if (membership.getState() == EXPIRED && !addExpired) {
+ iterator.remove();
+ } else if (membership.getState() == UNAVAILABLE && !addUnavailable) {
+ iterator.remove();
+ } else if (membership.getState() == OBSERVER && observerRead) {
+ iterator.remove();
+ observerMemberships.add(membership);
}
}
- List<MembershipState> priorityList = new ArrayList<>();
- priorityList.addAll(memberships);
- Collections.sort(priorityList, new NamenodePriorityComparator());
+ if(!observerRead) {
+ Collections.sort(memberships, new NamenodePriorityComparator());
+ LOG.debug("Selected most recent NN {} for query", memberships);
+ return memberships;
+ } else {
+ List<MembershipState> ret = new ArrayList<>(
+ memberships.size() + observerMemberships.size());
+ Collections.sort(memberships, new NamenodePriorityComparator());
+ if(observerMemberships.size() > 1) {
+ Collections.shuffle(observerMemberships);
Review Comment:
Can you comment on why this needs to be shuffled? This method doc mentions
it should start with the most recently updated registration. Is that correct?
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/MiniRouterDFSCluster.java:
##########
@@ -1046,6 +1047,27 @@ public void switchToStandby(String nsId, String nnId) {
}
}
+ /**
+ * Switch a namenode in a nameservice to be the observer.
+ * @param nsId Nameservice identifier.
+ * @param nnId Namenode identifier.
+ */
+ public void switchToObserver(String nsId, String nnId) {
+ try {
+ int total = cluster.getNumNameNodes();
+ NameNodeInfo[] nns = cluster.getNameNodeInfos();
+ for (int i = 0; i < total; i++) {
+ NameNodeInfo nn = nns[i];
+ if (nn.getNameserviceId().equals(nsId) &&
+ nn.getNamenodeId().equals(nnId)) {
+ cluster.transitionToObserver(i);
+ }
Review Comment:
Add break; to stop iterating through the list
Issue Time Tracking
-------------------
Worklog Id: (was: 779684)
Time Spent: 10h (was: 9h 50m)
> RBF: Support observer node from Router-Based Federation
> -------------------------------------------------------
>
> Key: HDFS-13522
> URL: https://issues.apache.org/jira/browse/HDFS-13522
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: federation, namenode
> Reporter: Erik Krogen
> Assignee: Simbarashe Dzinamarira
> Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-13522.001.patch, HDFS-13522.002.patch,
> HDFS-13522_WIP.patch, RBF_ Observer support.pdf, Router+Observer RPC
> clogging.png, ShortTerm-Routers+Observer.png
>
> Time Spent: 10h
> Remaining Estimate: 0h
>
> Changes will need to occur to the router to support the new observer node.
> One such change will be to make the router understand the observer state,
> e.g. {{FederationNamenodeServiceState}}.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
