[
https://issues.apache.org/jira/browse/YARN-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13539180#comment-13539180
]
Arun C Murthy commented on YARN-2:
----------------------------------
Ok, I finally got around to finishing this up.
W.r.t cores, on some more thinking, I'm inclined to go along with the concept
of integral 'virtual cores' instead of a float-precision 'cores' for the
following reasons:
* It provides a level on indirection to deal with heterogenous cores viz. much
more important for CPUs (as opposed to memory, disk b/w etc.). I've also added
a notion of physical-to-virtual cores translation
(yarn.nodemanager.vcores-pcores-ratio), per NodeManager, similar to
physical-to-virtual memory translation (yarn.nodemanager.vmem-pmem-ratio) that
we already have in place.
* It ensures we do minimal floating-point operations in the inner-most loop
which are very expensive (for e.g. we dropped usage of Math.ceil in
MAPREDUCE-1354 for JobTracker - Math.ceil is a JNI call). This is something
I've been very focussed on since the dawn, which explains the integral
implementations of divideAndCeil we've had since the beginning.
To make it clear, I've also renamed the apis to be (get,set)VirtualCores and
marked them evolving - in future we can add (get,set)Cores after we finalize
how we specify not just #cores, but also their capabilities (gigahertz?).
Thoughts?
> Enhance CS to schedule accounting for both memory and cpu cores
> ---------------------------------------------------------------
>
> Key: YARN-2
> URL: https://issues.apache.org/jira/browse/YARN-2
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: capacityscheduler, scheduler
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Fix For: 2.0.3-alpha
>
> Attachments: MAPREDUCE-4327.patch, MAPREDUCE-4327.patch,
> MAPREDUCE-4327.patch, MAPREDUCE-4327-v2.patch, MAPREDUCE-4327-v3.patch,
> MAPREDUCE-4327-v4.patch, MAPREDUCE-4327-v5.patch, YARN-2-help.patch,
> YARN-2.patch, YARN-2.patch, YARN-2.patch, YARN-2.patch
>
>
> With YARN being a general purpose system, it would be useful for several
> applications (MPI et al) to specify not just memory but also CPU (cores) for
> their resource requirements. Thus, it would be useful to the
> CapacityScheduler to account for both.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
