[
https://issues.apache.org/jira/browse/YARN-7494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16544759#comment-16544759
]
Weiwei Yang commented on YARN-7494:
-----------------------------------
Hi [~sunilg]
ApplicationSchedulingConfig
* Is "yarn.scheduler.capacity.multi-node-sorting.policy" necessary? We can use
"yarn.scheduler.capacity.root.multi-node-sorting.policy" to set global sorting
policy for CS right?
CapacitySchedulerConfiguration
* line 2197 - 2201: the check is not necessary, policyClassName cannot be null
as there is default value given.
* line 2203: looks like the sorting interval is a global setting, is it better
to set per-policy? E.g
"yarn.scheduler.capacity.multi-node-sorting.policy.<policyName>.sorting-task.interval.ms".
This could be a followup task if you agree with this suggestion.
MultiNodeLookupPolicy
* API: Iterator<N> getPreferredNodeIterator(Collection<N> nodes, String
partition); It looks like the the purpose of adding first argument (a
collection of nodes) is to support in-place sorting policy, however from the
API level, it is confusing that in \{{ResourceUsageBasedMultiNodeLookupPolicy}}
this argument is not used at all. For consistency, should we make sure the
iterator that \{{getPreferredNodeIterator}} returns only iterates over a
sub-set of the candidates collection?
LocalityAppPlacementAllocator
* line 81-85: can we add a debug message here to indicate what kind of policy
this app placement allocator used?
TestFifoScheduler/TestNMReconnect/TestQueueParsing/TestReservations/TestRMWebApp/TestUtils
* I commented on this earlier, these UT classes were just modified to add a
mock MultiNodeSortingManager to the context, I don't think this is necessary,
can we remove them?
Minor ones
* are the changes in ActivitiesLogger/ActivitiesManager also for this JIRA?
Seems they are for a different purpose, should we separate them to another
ticket ?
* There a lot of classes have import problems, such as unused imports and *
import, please take a look and fix them.
Thanks
> Add muti node lookup support for better placement
> -------------------------------------------------
>
> Key: YARN-7494
> URL: https://issues.apache.org/jira/browse/YARN-7494
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
> Reporter: Sunil Govindan
> Assignee: Sunil Govindan
> Priority: Major
> Attachments: YARN-7494.001.patch, YARN-7494.002.patch,
> YARN-7494.003.patch, YARN-7494.004.patch, YARN-7494.005.patch,
> YARN-7494.006.patch, YARN-7494.007.patch, YARN-7494.008.patch,
> YARN-7494.009.patch, YARN-7494.v0.patch, YARN-7494.v1.patch,
> multi-node-designProposal.png
>
>
> Instead of single node, for effectiveness we can consider a multi node lookup
> based on partition to start with.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
