[
https://issues.apache.org/jira/browse/YARN-7494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346404#comment-16346404
]
Tao Yang commented on YARN-7494:
--------------------------------
Thanks [~cheersyang] for your mention.
Some thoughts (parts are the same with those in my last comments) from my side:
# Sorting by nodeLookupPolicy for every allocation process is expensive. We
have planned to add new service to manage and periodically refresh
per-ordering-policy ordered list of nodes, scheduler can filter candidate nodes
from ordered node lists for app request and need no more sorting. So that we
can define cluster-level(or default) ordering policy to achieve better load
balance or other requirements and it's better for the performance of scheduler.
# This patch iterates all partition nodes to create new
PartitionBasedCandidateNodeSet instance for every schedule process in
CapacityScheduler#getCandidateNodeSet. I think we can keep a single instance to
avoid always creating it. Further more, we can replace it with ordered node
list if the plan is acceptable.
# This patch remains as it is to iterate all nodes and trigger the schedule
process for every node in CapacityScheduler#schedule. It's property for
scheduler before which dose allocation for single node. But for multiple nodes,
I think it's better to iterates all partitions to trigger the schedule process,
we can move multiNodePlacementEnabled check branch from
CapacityScheduler#getCandidateNodeSet to CapacityScheduler#schedule, do
different iteration and logic for different choose.
# CandidateNodeSet#getAllNodes returns Map<NodeId, N> type, and it seems no
need to find node by NodeId, perhaps we can change it to Set or List to support
getting ordered nodes.
Thanks.
> Add muti node lookup support for better placement
> -------------------------------------------------
>
> Key: YARN-7494
> URL: https://issues.apache.org/jira/browse/YARN-7494
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
> Reporter: Sunil G
> Assignee: Sunil G
> Priority: Major
> Attachments: YARN-7494.001.patch, YARN-7494.v0.patch,
> YARN-7494.v1.patch
>
>
> Instead of single node, for effectiveness we can consider a multi node lookup
> based on partition to start with.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
