Tao Yang created YARN-11736:
-------------------------------
Summary: Enhance MultiNodeLookupPolicy to allow configuration of
extended comparators for better usability.
Key: YARN-11736
URL: https://issues.apache.org/jira/browse/YARN-11736
Project: Hadoop YARN
Issue Type: Improvement
Components: multi-node-placement
Reporter: Tao Yang
Assignee: Tao Yang
Currently when multi-nodes is enabled, there is only 1 implementation of
_MultiNodeLookupPolicy_ interface: {_}ResourceUsageMultiNodeLookupPolicy{_},
which will sort nodes by allocated resources in ascending order.
If cluster has nodes with different resource-spec, the resource utilization of
smaller nodes will be significantly high, while larger nodes will experience
low resource utilization. This may rise the hotspot risk and reduce the
scheduling effectiveness.
So I propose to add a new policy called _EnhancedMultiNodeLookupPolicy_ to meet
requirements from some complex scenarios, which should contains serveral
inherit comparators and can be extended later, and supports configuring
specified comparators for different policy instances.
{*}Implementation Details{*}:
1. _MultiNodeSorter#initPolicy_ will pass the policyConf which is cloned from
scheduler configuration and attached the name of current policy, so that we can
fetch the specified configuration for this policy inside the implementations of
MultiNodeLookupPolicy.
2. new implementation of MultiNodeLookupPolicy: EnhancedMultiNodeLookupPolicy
2.1) contains several inherit comparators and can be extendable later.
comparator keys: _ALLOCATED_RESOURCE / UNALLOCATED_RESOURCE /
DOMINANT_RESOURCE_RATIO / NODE_ID,_ order-directions: _ASC / DESC._
2.2) supports configuring specified comparators with
order-direction(ASC/DESC) for different policy instances via conf-key:
{+}{color:#172b4d}yarn.scheduler.capacity.multi-node-sorting.policy.<policy-name>.comparators{color}{+},
value format is
"<comparator_key_1>[:<order_direction_1>],<comparator_key_2>[:<order_direction_2>],...".
For example, "DOMINANT_ALLOCATED_RATIO,NODE_ID:DESC" means that for policy
test, nodes should be sorted by dominant-resource-ratio in ascending order, by
nodeID desc in descending order.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
