[
https://issues.apache.org/jira/browse/HDFS-17302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17800255#comment-17800255
]
ASF GitHub Bot commented on HDFS-17302:
---------------------------------------
huangzhaobo99 commented on code in PR #6380:
URL: https://github.com/apache/hadoop/pull/6380#discussion_r1435977171
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/fairness/ProportionRouterRpcFairnessPolicyController.java:
##########
@@ -0,0 +1,76 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfs.server.federation.fairness;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.server.federation.router.FederationUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.util.Set;
+
+import static
org.apache.hadoop.hdfs.server.federation.fairness.RouterRpcFairnessConstants.CONCURRENT_NS;
+import static
org.apache.hadoop.hdfs.server.federation.router.RBFConfigKeys.DFS_ROUTER_FAIR_HANDLER_PROPORTION_DEFAULT;
+import static
org.apache.hadoop.hdfs.server.federation.router.RBFConfigKeys.DFS_ROUTER_FAIR_HANDLER_PROPORTION_KEY_PREFIX;
+import static
org.apache.hadoop.hdfs.server.federation.router.RBFConfigKeys.DFS_ROUTER_HANDLER_COUNT_DEFAULT;
+import static
org.apache.hadoop.hdfs.server.federation.router.RBFConfigKeys.DFS_ROUTER_HANDLER_COUNT_KEY;
+
+/**
+ * Proportion fairness policy extending {@link
AbstractRouterRpcFairnessPolicyController}
+ * and fetching proportion of handlers from configuration for all available
name services,
+ * based on the proportion and the total number of handlers, calculate the
handlers of all ns.
+ * The handlers count will not change for this controller.
+ */
+public class ProportionRouterRpcFairnessPolicyController extends
+ AbstractRouterRpcFairnessPolicyController{
+
+ private static final Logger LOG =
+
LoggerFactory.getLogger(ProportionRouterRpcFairnessPolicyController.class);
+
+ public ProportionRouterRpcFairnessPolicyController(Configuration conf){
+ init(conf);
+ }
+
+ @Override
+ public void init(Configuration conf) {
+ super.init(conf);
+ // Total handlers configured to process all incoming Rpc.
+ int handlerCount = conf.getInt(DFS_ROUTER_HANDLER_COUNT_KEY,
DFS_ROUTER_HANDLER_COUNT_DEFAULT);
+
+ LOG.info("Handlers available for fairness assignment {} ", handlerCount);
+
+ // Get all name services configured
+ Set<String> allConfiguredNS = FederationUtil.getAllConfiguredNS(conf);
+
+ // Insert the concurrent nameservice into the set to process together
+ allConfiguredNS.add(CONCURRENT_NS);
+ for (String nsId : allConfiguredNS) {
Review Comment:
From the allocation perspective, Basically consistent with the
StaticRouterPCFairnessPolicyController policy. Can you share on your ideas? Thx.
> RBF: ProportionRouterRpcFairnessPolicyController-Sharing and isolation.
> -----------------------------------------------------------------------
>
> Key: HDFS-17302
> URL: https://issues.apache.org/jira/browse/HDFS-17302
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: rbf
> Reporter: Jian Zhang
> Assignee: Jian Zhang
> Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-17302.001.patch, HDFS-17302.002.patch
>
>
> h2. Current shortcomings
> [HDFS-14090|https://issues.apache.org/jira/browse/HDFS-14090] provides a
> StaticRouterRpcFairnessPolicyController to support configuring different
> handlers for different ns. Using the StaticRouterRpcFairnessPolicyController
> allows the router to isolate different ns, and the ns with a higher load will
> not affect the router's access to the ns with a normal load. But the
> StaticRouterRpcFairnessPolicyController still falls short in many ways, such
> as:
> 1. *Configuration is inconvenient and error-prone*: When I use
> StaticRouterRpcFairnessPolicyController, I first need to know how many
> handlers the router has in total, then I have to know how many nameservices
> the router currently has, and then carefully calculate how many handlers to
> allocate to each ns so that the sum of handlers for all ns will not exceed
> the total handlers of the router, and I also need to consider how many
> handlers to allocate to each ns to achieve better performance. Therefore, I
> need to be very careful when configuring. Even if I configure only one more
> handler for a certain ns, the total number is more than the number of
> handlers owned by the router, which will also cause the router to fail to
> start. At this time, I had to investigate the reason why the router failed to
> start. After finding the reason, I had to reconsider the number of handlers
> for each ns.
> 2. *Extension ns is not supported*: During the running of the router, if a
> new ns is added to the cluster and a mount is added for the ns, but because
> no handler is allocated for the ns, the ns cannot be accessed through the
> router. We must reconfigure the number of handlers and then refresh the
> configuration. At this time, the router can access the ns normally. When we
> reconfigure the number of handlers, we have to face disadvantage 1:
> Configuration is inconvenient and error-prone.
> 3. *Waste handlers*: The main purpose of proposing
> RouterRpcFairnessPolicyController is to enable the router to access ns with
> normal load and not be affected by ns with higher load. First of all, not all
> ns have high loads; secondly, ns with high loads do not have high loads 24
> hours a day. It may be that only certain time periods, such as 0 to 8
> o'clock, have high loads, and other time periods have normal loads. Assume
> there are 2 ns, and each ns is allocated half of the number of handlers.
> Assume that ns1 has many requests from 0 to 14 o'clock, and almost no
> requests from 14 to 24 o'clock, ns2 has many requests from 12 to 24 o'clock,
> and almost no requests from 0 to 14 o'clock; when it is between 0 o'clock and
> 12 o'clock and between 14 o'clock and 24 o'clock, only one ns has more
> requests and the other ns has almost no requests, so we have wasted half of
> the number of handlers.
> 4. *Only isolation, no sharing*: The staticRouterRpcFairnessPolicyController
> does not support sharing, only isolation. I think isolation is just a means
> to improve the performance of router access to normal ns, not the purpose. It
> is impossible for all ns in the cluster to have high loads. On the contrary,
> in most scenarios, only a few ns in the cluster have high loads, and the
> loads of most other ns are normal. For ns with higher load and ns with normal
> load, we need to isolate their handlers so that the ns with higher load will
> not affect the performance of ns with lower load. However, for nameservices
> that are also under normal load, or are under higher load, we do not need to
> isolate them, these ns of the same nature can share the handlers of the
> router; The performance is better than assigning a fixed number of handlers
> to each ns, because each ns can use all the handlers of the router.
> h2. New features
> Based on the above staticRouterRpcFairnessPolicyController, there are
> deficiencies in usage and performance. I provide a new
> RouterRpcFairnessPolicyController:
> ProportionRouterRpcFairnessPolicyController (maybe with a better name) to
> solve the above major shortcomings.
> 1. *More user-friendly configuration* : Supports allocating handlers
> proportionally to each ns. For example, we can give ns1 a handler ratio of
> 0.2, then ns1 will use 0.2 of the total number of handlers on the router.
> Using this method, we do not need to confirm in advance how many handlers the
> router has.
> 2. *Sharing* :
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
