[
https://issues.apache.org/jira/browse/YARN-7494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392740#comment-16392740
]
Weiwei Yang commented on YARN-7494:
-----------------------------------
Hi [~sunilg]/[~leftnoteasy]
Sorry I just get the time to review this. Thanks for the patch first of all.
Some review comments
Agree with [~leftnoteasy], can we put {{MultiNodeLookupPolicy}} into RMContext
instead of binding it to a specific scheduler. Regarding the support of per-app
policy, I think when I discussed with [~sunilg] we both agreed this will be
supported. From the code level, I have some different opinion.
Let me brief my idea based on [~sunilg]'s patch:
We can let {{MultiNodeSortingManager}} extend {{CompositeService}}, and when a
particular policy class implements interface {{MultiNodeLookupPolicy}}, it can
further choose to extend {{AbstractService}} (for policies do backend sorting)
or not to. And we can remove the SortingThread from the manager, and let the
policy handle the sorting fashion on their needs. Some pseudo code,
{code}
MultiNodeSortingManager extends CompositeService
// The policy key can be the name of the policy from the config
Map<String policyKey, MultiNodeLookupPolicy> policies
void registerPolicy(MultiNodeLookupPolicy policy) {
// register a policy in the manager
policies.add(policy);
policy.init();
// register as a service to the manager if necessary
addIfService(policy);
}
void unregisterPolicy(MultiNodeLookupPolicy policy) {
// unregister a policy from the manager
policies.remove(policy);
policy.stop();
}
void serviceStart() {
// start all registered policies (if they are services)
}
void serviceStop() {
// stop all
}
// Retrieve result by a partition and a policy key
Iterator<N> getPreferredNodeIterator(String partition, String policyKey) {
policies.get(policyKey).getPreferredNodeIterator(partition);
}
{code}
When we find an app with its own policy, the code can look like
{code}
// Read policy from scheduling env
MultiNodeLookupPolicy p = appSchedulingInfo.getSortingNodesPolicy();
if (multinodeMgr.isNotRegistered(p)) {
multinodeMgr.register(p);
}
// In app allocator code, it will be same for every implementation class
multinodeMgr.getPreferredNodeIterator(partition,
appSchedulingInfo.getSortingNodesPolicyKey());
{code}
Please take a look. Thanks.
> Add muti node lookup support for better placement
> -------------------------------------------------
>
> Key: YARN-7494
> URL: https://issues.apache.org/jira/browse/YARN-7494
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
> Reporter: Sunil G
> Assignee: Sunil G
> Priority: Major
> Attachments: YARN-7494.001.patch, YARN-7494.002.patch,
> YARN-7494.v0.patch, YARN-7494.v1.patch, multi-node-designProposal.png
>
>
> Instead of single node, for effectiveness we can consider a multi node lookup
> based on partition to start with.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
