[
https://issues.apache.org/jira/browse/YARN-7494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16415536#comment-16415536
]
Weiwei Yang commented on YARN-7494:
-----------------------------------
Hi [~sunilg]
I haven't read the new patch yet, probably tomorrow I can find time to do it.
But I would like to give you some quick feedback
{quote}
Hence a clear cached node per partition is now available in ClusterNodeTracker.
Now from MultiNodeSorter
{quote}
I understand about this cache, but is it more simpler to just do
{code:java}
getNodes(node ->partition.equals(node.getPartition()));
{code}
in ClusterNodeTracker? Maintaining a separate cache will require to update them
which not seem to be necessary to me.
{quote}
Since we are doing this from FiCaSchedulerApp, we can definitely assume its CS.
Correct?
{quote}
You'll get a lot of UT failures in FIFO path. A way to fix this is to do this
in {{CapacityScheduler#addApplicationAttempt}}, such as
{code}
FiCaSchedulerApp attempt = new FiCaSchedulerApp(applicationAttemptId,
application.getUser(), queue, queue.getAbstractUsersManager(),
rmContext, application.getPriority(), isAttemptRecovering,
activitiesManager);
// Set sorting nodes policy if it was not set in app env
// and global or queue level config exists.
setAppMultiNodeSortingPolicyIfNecessary(queue, attempt);
{code}
bq. Could this be more on per-policy level?
Yes, this is better to be a per-policy interval. Suggest to load it from
configuration file, such as
{{yarn.scheduler.capacity.multi-node-sorting.policy.ABC.sorting-task.interval.ms}}
where ABC is the name of a policy.
What do you think?
> Add muti node lookup support for better placement
> -------------------------------------------------
>
> Key: YARN-7494
> URL: https://issues.apache.org/jira/browse/YARN-7494
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
> Reporter: Sunil G
> Assignee: Sunil G
> Priority: Major
> Attachments: YARN-7494.001.patch, YARN-7494.002.patch,
> YARN-7494.003.patch, YARN-7494.004.patch, YARN-7494.005.patch,
> YARN-7494.v0.patch, YARN-7494.v1.patch, multi-node-designProposal.png
>
>
> Instead of single node, for effectiveness we can consider a multi node lookup
> based on partition to start with.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
