[
https://issues.apache.org/jira/browse/YARN-7494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16417590#comment-16417590
]
Sunil G commented on YARN-7494:
-------------------------------
Thanks [~cheersyang] and [~leftnoteasy]
bq.Maintaining a separate cache will require to update them which not seem to
be necessary to me.
This will help us in future as well. We depend on labelManager a lot and its
always to get node per partition. Such a cache, will help for metrics and for
other apis which we can do in a clean up jira.
bq.You'll get a lot of UT failures in FIFO path. A way to fix this is to do
this in {{CapacityScheduler#addApplicationAttempt}}, such as
Makes sense. I ll update in next patch.
bq.Suggest to load it from configuration file, such as
This will force such a config to be only at CS level. I dont see any pblm but
this will add to a complex per-policy config. I thought of making this even
more simpler. we are now configuring policy like
{{yarn.scheduler.capacity.multi-node-sorting.policy}} to a value like
"resource-based" or "org....customPolicy.class". Now such a complex name which
is fully qualified will make your proposal more complex. Hence we can opt for
[name=resource-based, timeout=1200] or something similar in nature. Thoughts?
bq.CS#getCandidateNodeSet, the get_node_nodes_from_give_partition operation is
very expensive
Yes. I though i used that in CS api as well. Infact i used only in
MultiNodeManager. I ll change that in next patch. Thank you.
I will wait for config comment from [~cheersyang] before uploading next patch.
> Add muti node lookup support for better placement
> -------------------------------------------------
>
> Key: YARN-7494
> URL: https://issues.apache.org/jira/browse/YARN-7494
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
> Reporter: Sunil G
> Assignee: Sunil G
> Priority: Major
> Attachments: YARN-7494.001.patch, YARN-7494.002.patch,
> YARN-7494.003.patch, YARN-7494.004.patch, YARN-7494.005.patch,
> YARN-7494.v0.patch, YARN-7494.v1.patch, multi-node-designProposal.png
>
>
> Instead of single node, for effectiveness we can consider a multi node lookup
> based on partition to start with.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
