[
https://issues.apache.org/jira/browse/YARN-11524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17739739#comment-17739739
]
ASF GitHub Bot commented on YARN-11524:
---------------------------------------
slfan1989 commented on code in PR #5797:
URL: https://github.com/apache/hadoop/pull/5797#discussion_r1251305309
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/Federation.md:
##########
@@ -235,7 +235,112 @@ SQL-Server scripts are located in
**sbin/FederationStateStore/SQLServer/**.
|`yarn.federation.subcluster-resolver.class` |
`org.apache.hadoop.yarn.server.federation.resolver.DefaultSubClusterResolverImpl`
| The class used to resolve which subcluster a node belongs to, and which
subcluster(s) a rack belongs to. |
|`yarn.federation.machine-list` | `<path of machine-list file>` | Path of
machine-list file used by `SubClusterResolver`. Each line of the file is a node
with sub-cluster and rack information. Below is the example: <br/> <br/> node1,
subcluster1, rack1 <br/> node2, subcluster2, rack1 <br/> node3, subcluster3,
rack2 <br/> node4, subcluster3, rack2 |
-###ON RMs:
+**How to Configure policy-manager ?**
+
+- Router Policy
+
+ Router Policy defines the logic for determining the routing of an
application submission and determines the HomeSubCluster for the application.
+
+ - HashBasedRouterPolicy
+ - This policy selects a sub-cluster based on the hash of the job's queue
name. It is particularly useful when dealing with a large number of queues in a
system, providing a default behavior. Furthermore, it ensures that all jobs
belonging to the same queue are consistently mapped to the same sub-cluster,
which can improve locality and performance.
+ - LoadBasedRouterPolicy
+ - This is a simplified load-balancing policy implementation. The policy
utilizes binary weights (0/1 values) to enable or disable each sub-cluster. It
selects the sub-cluster with the least load to forward the application traffic,
ensuring optimal distribution.
+ - LocalityRouterPolicy
+ - This policy selects the sub-cluster based on the node specified by the
client for running its application. Follows these conditions:
+ - It succeeds if
+ - There are three AMContainerResourceRequests in the order NODE, RACK,
ANY
+ - Falls back to WeightedRandomRouterPolicy
+ - Null or empty AMContainerResourceRequests;
+ - One AMContainerResourceRequests and it has ANY as ResourceName;
+ - The node is in blacklisted SubClusters.
+ - It fails if
+ - The node does not exist and RelaxLocality is False;
+ - We have an invalid number (not 0, 1 or 3) resource requests
+ - RejectRouterPolicy
+ - This policy simply rejects all incoming requests.
+ - UniformRandomRouterPolicy
+ - This simple policy picks at uniform random among any of the currently
active sub-clusters. This policy is easy to use and good for testing.
+ - WeightedRandomRouterPolicy
+ - This policy implements a weighted random sample among currently active
sub-clusters.
+
+- AMRM Policy
+
+ AMRM Proxy defines the logic to split the resource request list received by
AM among RMs.
+
+ - BroadcastAMRMProxyPolicy
+ - This policy simply broadcasts each ResourceRequest to all the available
sub-clusters.
+ - HomeAMRMProxyPolicy
+ - This policy simply sends the ResourceRequest to the home sub-cluster.
+ - LocalityMulticastAMRMProxyPolicy
+ - Host localized ResourceRequests are always forwarded to the RM that owns
the corresponding node, based on the feedback of a SubClusterResolver
+ If the SubClusterResolver cannot resolve this node we default to
forwarding the ResourceRequest to the home sub-cluster.
+ - Rack localized ResourceRequests are forwarded to the RMs that owns the
corresponding rack. Note that in some deployments each rack could be
+ striped across multiple RMs. This policy respects that. If the
SubClusterResolver cannot resolve this rack we default to forwarding
+ the ResourceRequest to the home sub-cluster.
+ - ANY requests corresponding to node/rack local requests are forwarded
only to the set of RMs that owns the corresponding localized requests. The
number of
+ containers listed in each ANY is proportional to the number of localized
container requests (associated to this ANY via the same allocateRequestId).
+ - RejectAMRMProxyPolicy
+ - This policy simply rejects all requests. Useful to prevent apps from
accessing any sub-cluster.
+
+- Policy Manager
+
+ The PolicyManager is providing a combination of RouterPolicy and AMRMPolicy.
Currently, We offer six types of PolicyManagers:
+ - HashBroadcastPolicyManager
+ - Policy that routes applications via hashing of their queuename, and
broadcast resource requests. This picks a HashBasedRouterPolicy for the router
and a BroadcastAMRMProxyPolicy for the amrmproxy as they are designed to work
together.
+ - We configure to use this policy:
+ ```xml
+ <property>
+ <name>yarn.federation.policy-manager</name>
+
<value>org.apache.hadoop.yarn.server.federation.policies.manager.HashBroadcastPolicyManager</value>
+ </property>
+ ```
+ - HomePolicyManager
+ - Policy manager which uses the UniformRandomRouterPolicy for the Router
and HomeAMRMProxyPolicy as the AMRMProxy policy to find the RM.
+ - We configure to use this policy:
+ ```xml
+ <property>
+ <name>yarn.federation.policy-manager</name>
+
<value>org.apache.hadoop.yarn.server.federation.policies.manager.HomePolicyManager</value>
+ </property>
+ ```
+ - PriorityBroadcastPolicyManager
+ - Policy that allows operator to configure "weights" for routing. This
picks a PriorityRouterPolicy for the router and a BroadcastAMRMProxyPolicy for
the amrmproxy as they are designed to work together.
+ - We configure to use this policy:
+ ```xml
+ <property>
+ <name>yarn.federation.policy-manager</name>
+
<value>org.apache.hadoop.yarn.server.federation.policies.manager.PriorityBroadcastPolicyManager</value>
+ </property>
+ ```
+ - RejectAllPolicyManager
+ - This policy rejects all requests for both router and amrmproxy routing.
This picks a RejectRouterPolicy for the router and a RejectAMRMProxyPolicy for
the amrmproxy as they are designed to work together.
+ - We configure to use this policy:
+ ```xml
+ <property>
+ <name>yarn.federation.policy-manager</name>
+
<value>org.apache.hadoop.yarn.server.federation.policies.manager.RejectAllPolicyManager</value>
+ </property>
+ ```
+ - UniformBroadcastPolicyManager
+ - It combines the basic policies: UniformRandomRouterPolicy and
BroadcastAMRMProxyPolicy, which are designed to work together and "spread" the
load among sub-clusters uniformly. This simple policy might impose heavy load
on the RMs and return more containers than a job requested as all requests are
(replicated and) broadcasted.
+ - We configure to use this policy:
+ ```xml
+ <property>
+ <name>yarn.federation.policy-manager</name>
+
<value>org.apache.hadoop.yarn.server.federation.policies.manager.UniformBroadcastPolicyManager</value>
+ </property>
+ ```
+ - WeightedLocalityPolicyManager
+ - Policy that allows operator to configure "weights" for routing. This
picks a LocalityRouterPolicy for the router and a
LocalityMulticastAMRMProxyPolicy for the amrmproxy as they are designed to work
together.
+ - We configure to use this policy:
Review Comment:
Thank you for your suggestion! I will improve the documentation.
> Improve the Policy Description in Federation.md
> -----------------------------------------------
>
> Key: YARN-11524
> URL: https://issues.apache.org/jira/browse/YARN-11524
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: federation
> Affects Versions: 3.4.0
> Reporter: Shilun Fan
> Assignee: Shilun Fan
> Priority: Major
> Labels: pull-request-available
>
> The Federation.md file lacks an introduction to Policy. In this Jira ticket,
> we will enhance the content regarding Router Policy, AMRM Policy, and
> PolicyManager to help users gain a better understanding of these types of
> policies.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
