[
https://issues.apache.org/jira/browse/YARN-7708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16370783#comment-16370783
]
Young Chen edited comment on YARN-7708 at 2/21/18 12:24 AM:
------------------------------------------------------------
The rationale for this policy is to lower load on sub clusters that are
receiving an abnormal amount of jobs or are unable to keep up for performance
reasons. To alleviate job failures/excessive delays, this policy attempts to
redirect load away from highly loaded sub clusters by updating a
WeightedLocalityPolicyManager with modified weights.
This patch introduces the LoadBasedGlobalPolicy. It's configurable in the
following ways:
* MAX_EDIT: The maximum number of sub clusters this policy will throttle. The
top N clusters by pending load will have their policy weights scaled down.
* MIN_PENDING: The minimum number of pending applications before a sub cluster
weight qualifies for editing.
* MAX_PENDING: The maximum number of pending applications in the scalable
range. Sub clusters exceeding this will have their policy weights set to
MIN_WEIGHT
* MIN_WEIGHT: The minimum weight possible for a highly loaded sub cluster.
* SCALING: The scaling method that maps pending load to sub cluster policy
weight. Currently there are three scaling methods: quadratic, log, and linear.
was (Author: youchen):
This patch introduces the LoadBasedGlobalPolicy. It's configurable in the
following ways:
* MAX_EDIT: The maximum number of sub clusters this policy will throttle. The
top N clusters by pending load will have their policy weights scaled down.
* MIN_PENDING: The minimum number of pending applications before a sub cluster
weight qualifies for editing.
* MAX_PENDING: The maximum number of pending applications in the scalable
range. Sub clusters exceeding this will have their policy weights set to
MIN_WEIGHT
* MIN_WEIGHT: The minimum weight possible for a highly loaded sub cluster.
* SCALING: The scaling method that maps pending load to sub cluster policy
weight. Currently there are three scaling methods: quadratic, log, and linear.
> [GPG] Load based policy generator
> ---------------------------------
>
> Key: YARN-7708
> URL: https://issues.apache.org/jira/browse/YARN-7708
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Carlo Curino
> Assignee: Young Chen
> Priority: Major
> Attachments: YARN-7708-YARN-7402.01.cumulative.patch
>
>
> This policy reads load from the "pendingQueueLength" metrics and provides
> scaling into a set of weights that influence the AMRMProxy and Router
> behaviors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
