[
https://issues.apache.org/jira/browse/YARN-11736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17901819#comment-17901819
]
ASF GitHub Bot commented on YARN-11736:
---------------------------------------
TaoYang526 commented on PR #7121:
URL: https://github.com/apache/hadoop/pull/7121#issuecomment-2506890772
@zuston Thanks for the feedback.
> Oh, I will we encoutered the similar problems about multi node placement
about different resource spec. And I found some bugs about this features,
please refer to: https://zuston.vercel.app/publish/hadoop-yarn/
You are right, I have proposed to fix those bugs in YARN-9598, but it was in
dispute after some discussions, and part of it were merged into community in
YARN-11573 which you mentioned in your article.
FYI, when the scheduler found another node can place the pending request,
reserved container for this request can be unreserved before assigning, you can
see the details in RegularContainerAllocator#assignContainer.
>BTW, I think the node sorting policy could be extended by ourself, there is
no necessary to change the default policy.
This PR doesn't change the default policy, just add a new policy can be
configured to use.
> Enhance MultiNodeLookupPolicy to allow configuration of extended comparators
> for better usability.
> --------------------------------------------------------------------------------------------------
>
> Key: YARN-11736
> URL: https://issues.apache.org/jira/browse/YARN-11736
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: multi-node-placement
> Reporter: Tao Yang
> Assignee: Tao Yang
> Priority: Major
> Labels: pull-request-available
>
> Currently when multi-nodes is enabled, there is only 1 implementation of
> _MultiNodeLookupPolicy_ interface: {_}ResourceUsageMultiNodeLookupPolicy{_},
> which will sort nodes by allocated resources in ascending order.
> If cluster has nodes with different resource-spec, the resource utilization
> of smaller nodes will be significantly high, while larger nodes will
> experience low resource utilization. This may rise the hotspot risk and
> reduce the scheduling effectiveness.
> So I propose to add a new policy called _MultiComparatorPolicy_ to meet
> requirements from some complex scenarios, which should contains serveral
> inherit comparators and can be extended later, and supports configuring
> specified comparators for different policy instances.
>
> {*}Implementation Details{*}:
> 1. _MultiNodeSorter#initPolicy_ will pass the policyConf which is cloned from
> scheduler configuration and attached the name of current policy, so that we
> can fetch the specified configuration for this policy inside the
> implementations of MultiNodeLookupPolicy.
> 2. new implementation of {_}MultiNodeLookupPolicy{_}: _MultiComparatorPolicy_
> 2.1) contains several inherit comparators and can be extendable later.
> comparator keys: _ALLOCATED_RESOURCE / UNALLOCATED_RESOURCE /
> DOMINANT_RESOURCE_RATIO / NODE_ID,_ order-directions: _ASC / DESC._
> 2.2) supports configuring specified comparators with
> order-direction(ASC/DESC) for different policy instances via conf-key:
> {+}{color:#172b4d}yarn.scheduler.capacity.multi-node-sorting.policy.<policy-name>.comparators{color}{+},
> value format is
> "<comparator_key_1>[:<order_direction_1>],<comparator_key_2>[:<order_direction_2>],...".
> For example, "DOMINANT_ALLOCATED_RATIO,NODE_ID:DESC" means that for policy
> test, nodes should be sorted by dominant-resource-ratio in ascending order,
> by nodeID desc in descending order.
> 3. Refactor variable names in
> AbstractCSQueue/CSQueue/FiCaSchedulerApp/AppPlacementAllocator after
> supporting multiple policy instances with the same policy class.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
