[
https://issues.apache.org/jira/browse/HDFS-17534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17853323#comment-17853323
]
ASF GitHub Bot commented on HDFS-17534:
---------------------------------------
goiri commented on code in PR #6861:
URL: https://github.com/apache/hadoop/pull/6861#discussion_r1631807404
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/order/DestinationOrder.java:
##########
@@ -29,11 +29,14 @@ public enum DestinationOrder {
LOCAL, // Local first
RANDOM, // Random order
HASH_ALL, // Follow consistent hashing
- SPACE; // Available space based order
+ SPACE, // Available space based order
+ LEADER_FOLLOWER; // try leader sub-cluster first, if failed, try followers
/** Approaches that write folders in all subclusters. */
public static final EnumSet<DestinationOrder> FOLDER_ALL = EnumSet.of(
HASH_ALL,
RANDOM,
- SPACE);
+ SPACE,
+ // leader-follower mode should make sure all directory exists in case of
switching
Review Comment:
I think we had some documentation somewhere describing the multiple cluster
approach.
Can we add it there too?
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/order/LeaderFollowerResolver.java:
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.federation.resolver.order;
+
+import org.apache.hadoop.hdfs.server.federation.resolver.PathLocation;
+import org.apache.hadoop.hdfs.server.federation.resolver.RemoteLocation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Set;
+
+public class LeaderFollowerResolver implements OrderedResolver {
+ protected static final Logger LOG =
+ LoggerFactory.getLogger(LeaderFollowerResolver.class);
+
+ @Override
+ public String getFirstNamespace(String path, PathLocation loc) {
+ // always return first destination
Review Comment:
This is the default no?
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/resolver/TestMultipleDestinationResolver.java:
##########
@@ -340,6 +350,13 @@ public void testReadOnly() throws IOException {
assertDest("subcluster1", dest12);
}
+ @Test
+ public void testLeaderFollower() throws IOException {
+ PathLocation dest0 =
+
resolver.getDestinationForPath("/leaderfollower/folder0/file0.txt");
Review Comment:
Too many spaces.
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/order/LeaderFollowerResolver.java:
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.federation.resolver.order;
+
+import org.apache.hadoop.hdfs.server.federation.resolver.PathLocation;
+import org.apache.hadoop.hdfs.server.federation.resolver.RemoteLocation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Set;
+
+public class LeaderFollowerResolver implements OrderedResolver {
Review Comment:
Javadoc with the overall idea.
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/order/DestinationOrder.java:
##########
@@ -29,11 +29,14 @@ public enum DestinationOrder {
LOCAL, // Local first
RANDOM, // Random order
HASH_ALL, // Follow consistent hashing
- SPACE; // Available space based order
+ SPACE, // Available space based order
+ LEADER_FOLLOWER; // try leader sub-cluster first, if failed, try followers
/** Approaches that write folders in all subclusters. */
public static final EnumSet<DestinationOrder> FOLDER_ALL = EnumSet.of(
HASH_ALL,
RANDOM,
- SPACE);
+ SPACE,
+ // leader-follower mode should make sure all directory exists in case of
switching
Review Comment:
[HDFSRouterFederation.md](https://github.com/apache/hadoop/blob/01d257d5aa94163244cd3f1149d5ba2cb9f1e6ff/hadoop-hdfs-project/hadoop-hdfs-rbf/src/site/markdown/HDFSRouterFederation.md?plain=1#L289)
> RBF: Support leader follower mode for multiple subclusters
> ----------------------------------------------------------
>
> Key: HDFS-17534
> URL: https://issues.apache.org/jira/browse/HDFS-17534
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
> Reporter: Yuanbo Liu
> Assignee: Yuanbo Liu
> Priority: Major
> Labels: pull-request-available
>
> Currently there are five modes in multiple subclusters like
> HASH, LOCAL, RANDOM, HASH_ALL,SPACE;
> Proposal a new mode called leader/follower mode. routers try to write to
> leader subcluster as many as possible. When routers read data, put leader
> subcluster into first rank.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
