[
https://issues.apache.org/jira/browse/YARN-7494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346727#comment-16346727
]
Weiwei Yang commented on YARN-7494:
-----------------------------------
Hi [~sunilg]
Some updates today, I took a deeper look at the patch. Besides the comments I
mentioned earlier, there is some more problems we need to address
1. Discussed with Wangda offline, we agree that to make the "multi-node-lookup"
be configurable per-app, per-queue and per-cluster(scheduler). Reason: this
will be essential for a production cluster, we need the capability to enable
this feature step by step, e.g first enable for 1 apps, then 10 apps, then 1
queue and eventually the entire cluster. And this is a change to the factory
class, won't be too much.
2.(My opinion) I am not in favor of the config name:
"yarn.capacity.scheduler.multi-node-placement-enabled", it does not seem to be
informative. If we are going to implement #1, can we configure it to be
something like,
// scheduler
yarn.capacity.sorting-nodes.policy.class
...DefaultSortingNodesPolicy (which returns a single node set)
...NodeUtilizationBasedSortingPolicy
// queue
yarn.capacity.queue.<name>.sorting-nodes.policy.class
NodeUtilizationBasedSortingPolicy
// app
ENV string
So default is
{{yarn.capacity.sorting-nodes.policy.class=DefaultSortingNodesPolicy}}, queue
and app can override this policy.
3. API level, I think we need a {{sorting nodes service}} like [~Tao Yang] and
I both mentioned, because you have to run some policy to sort nodes in some
interval right? AppPlacementAllocator should retrieve candidate nodes from this
service, not directly from a policy, a policy should be just a sorting
algorithm.
We can setup a meeting to discuss this if you are available. Thanks.
> Add muti node lookup support for better placement
> -------------------------------------------------
>
> Key: YARN-7494
> URL: https://issues.apache.org/jira/browse/YARN-7494
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
> Reporter: Sunil G
> Assignee: Sunil G
> Priority: Major
> Attachments: YARN-7494.001.patch, YARN-7494.v0.patch,
> YARN-7494.v1.patch
>
>
> Instead of single node, for effectiveness we can consider a multi node lookup
> based on partition to start with.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
