[
https://issues.apache.org/jira/browse/YARN-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13451453#comment-13451453
]
Vinod Kumar Vavilapalli commented on YARN-2:
--------------------------------------------
Alright, the patch needs some minor upmerge to the latest trunk. Other than
that, a few comments: lots of text, but mostly minor issues:
- api.records.Resource should perhaps be not comparable anymore after these
changes?
ResourceComparator:
- It seems unnatural for the comparator to store {{clusterResource}}, is it
possible to just pass in {{clusterResource}} where-ever needed? I see
{{compare()}} -> {{getResourceAsValue()}} and {{divide()}} ->
{{getResourceAsValue()}} using this. For all the traces of divide(), I verified
that {{clusterResource}} can be passed all the way through. For {{compare()}}
method, see below.
- Haven't read the paper yet, but in the current impl, we may end up comparing
cpus to memory when comparing two resources. Is that intended? If not, even
compare() won't need to read {{clusterResource}}.
- {{ResourceComparator}} is doing much more than comparisons - may be call it
something like {{ResourceSchedulingCalculator}}?
- {{divide()}} seems to be completely unrelated to {{divideAndCeil()}} looking
at the implementation. One of the names need to change, not sure which.
- Not sure how to differentiate between {{ratio()}} and {{divide()}} either.
The implementation does differ a lot, I couldn't figure out which old
references of the division operator were replaced by {{divide()}} and which
ones by {{ratio()}}. Names can be made more explicit, if not that, at the least
the documentation.
Resources.java
- {{multiplyAndNormalizeUp()}} and {{multiplyAndNormalizeDown()}},
{{roundUp()}}, {{roundDown()}}, ratio(), divide(), divideAndCeil() and equals()
are unnecessary level of indirections to ResourceComparator and can be removed?
lessThan(), lessThanOrEqual(), greaterThan(), greaterThanOrEqual(), min(),
max() are useful though.
- createResource(int memory) constructor is going to be problem when other
resources/scheduling-algos come in. We should rename it, but okay doing it
later.
bq. Could we change the name of ResourceMemoryCpuComparator to something more
like DefaultMultiResourceComparator? I think
ResourceMemoryCpuNetworkBandwithDiskStorageGPUComparator is a bit long, but it
is the direction we are headed in.
I too agree. Given this is also going to be public (albeit admin facing)
configuration, I am +1 for something like MultiResourceComparator. Thoughts?
Problems worth considering in follow-up JIRAs:
- Num cores to be a float to allow a bit of over-subscription like it is
possible today without any concept of cores. Granted this can be simulated by
artificially increasing the number of cores in configuration, but we should
think of a more appropriate way.
- Some calculations like in LeafQueue have become very hard to read, we can
rewrite them to calculate one item per line :)
> Enhance CS to schedule accounting for both memory and cpu cores
> ---------------------------------------------------------------
>
> Key: YARN-2
> URL: https://issues.apache.org/jira/browse/YARN-2
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: capacityscheduler, scheduler
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Fix For: 2.0.2-alpha
>
> Attachments: MAPREDUCE-4327.patch, MAPREDUCE-4327.patch,
> MAPREDUCE-4327.patch, MAPREDUCE-4327-v2.patch, MAPREDUCE-4327-v3.patch,
> MAPREDUCE-4327-v4.patch, MAPREDUCE-4327-v5.patch, YARN-2-help.patch,
> YARN-2.patch, YARN-2.patch
>
>
> With YARN being a general purpose system, it would be useful for several
> applications (MPI et al) to specify not just memory but also CPU (cores) for
> their resource requirements. Thus, it would be useful to the
> CapacityScheduler to account for both.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
