[ https://issues.apache.org/jira/browse/YARN-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13730301#comment-13730301 ]
Eli Collins edited comment on YARN-1024 at 8/6/13 3:08 AM: ----------------------------------------------------------- I agree we need to define the meaning of a virtual core unambiguously (otherwise we won't be able to support two different frameworks on the same cluster that may have differing ideas of what a vcore is). I also agree with Phil that there are essentially two high-level use cases: 1. Jobs that want to express how much CPU capacity the job needs. Real world example - a distcp job wants to express it needs 100 containers but only a fraction of a CPU for each since it will spend most of its time blocking on IO. 2. Services - ie long-lived frameworks (ie support 2-level scheduling) - that want to request cores on many machines on a cluster and want to express CPU-level parallelism and aggregate demand (because they will schedule fine-grain requests w/in their long-lived containers). Eg a framework should be able to ask for two containers on a host, each with one core, so it can get two containers that can execute in parallel on a full core. This is assuming we plan to support long-running services in Yarn (YARN-896), which is hopefully not controversial. Real world example is HBase which may want 2 guaranteed cores per host on a given set of hosts. Seems like there are two high-level approaches: 1. Get rid of vcores. If we define 1vcore=1pcore (1vcore=1vcpu for virtual environments) and support fractional cores (YARN-972) then services can ask for 1 or more vcores knowing they're getting real cores and jobs just ask for what fraction of a vcore they think they need. This is really abandoning the concept of a "virtual core" because it's actually expressing a physical requirement (like memory, we assume Yarn is not dramatically over-committing the host). We can handle heterogeneous CPUs via attributes (as discussed in other Yarn jiras) since most clusters in my experience don't have wildly different processors (eg 1 or 2 generations is common), and attributes are sufficient to express policies like "all my cores should have equal/comparable performance". 2. Keep going with vcores as a CPU unit of measurement. If we define 1vcore=1ECU (works 1:1 for virtual environments) then services need to understand the the power of a core so they can ask for that many vcores - essentially they are just undoing the virtualization. YARN would need to make sure two containers each with 1 pcores worth of vcores does in fact give you two cores (just like hypervisors schedule vcpus for the same VM on different pcores to ensure parallelism), but there would be no guarantee that two containers on the same host each w/ one vcore would run in parallel. Jobs that want fractional cores would just express 1vcore per container and work their way up based on the experience running on the cluster (or also undo the virtualization by calculating vcore/pcore if they know what fraction of a pcore they want). Heterogenous CPUs does not fall out naturally (still need attributes) since there's no guarantee you can describe the difference between two CPUs is roughly 1 or more vcore (eg 2.4 - vs 2.0 Ghz < 1ECU), however there's no need for fractional vcores. I think either is reasonable and can be made to work, though I think #1 is preferable because: - Some frameworks want to express containers in physical resources (this is consistent with how YARN handles memory) - You can support jobs that don't want a full core via fractional cores (or slightly over-committing cores) - You can support heterogeneous cores via attributes ("I want equivalent containers") - vcores are optional anyway (only used in DRF) and therefore only need to be expressed if you care about physical cores because you need to reserve them or say you want a fraction of one Either way I think "vcore" is the wrong name either way because in #1 1vcore=1pcore so there's no virtualization and in #2 1 vcore is not a virtualization of a core (10 vcores does not give me 10 levels of parallelism), it's _just a unit_ (like an ECU). was (Author: eli): I agree we need to define the meaning of a virtual core unambiguously (otherwise we won't be able to support two different frameworks on the same cluster that may have differing ideas of what a vcore is). I also agree with Phil that there are essentially two high-level use cases: 1. Jobs that want to express how much CPU capacity the job needs. Real world example - a distcp job wants to express it needs 100 containers but only a fraction of a CPU for each since it will spend most of its time blocking on IO. 2. Services - ie long-lived frameworks (ie support 2-level scheduling) - that want to request cores on many machines on a cluster and want to express CPU-level parallelism and aggregate demand (because they will schedule fine-grain requests w/in their long-lived containers). Eg a framework should be able to ask for two containers on a host, each with one core, so it can get two containers that can execute in parallel on a full core. This is assuming we plan to support long-running services in Yarn (YARN-896), which is hopefully not controversial. Real world example is HBase which may want 2 guaranteed cores per host on a given set of hosts. Seems like there are two high-level approaches: 1. Get rid of vcores. If we define 1vcore=1pcore (1vcore=1vcpu for virtual environments) and support fractional cores (YARN-972) then services can ask for 1 or more vcores knowing they're getting real cores and jobs just ask for what fraction of a vcore they think they need. This is really abandoning the concept of a "virtual core" because it's actually expressing a physical requirement (like memory, we assume Yarn is not dramatically over-committing the host). We can handle heterogeneous CPUs via attributes (as discussed in other Yarn jiras) since most clusters in my experience don't have wildly different processors (eg 1 or 2 generations is common), and attributes are sufficient to express policies like "all my cores should have equal/comparable performance". 2. Keep going with vcores as a CPU unit of measurement. If we define 1vcore=1ECU (works 1:1 for virtual environments) then services (#1) need to understand the the power of a core so they can ask for that many vcores - essentially they are just undoing the virtualization. YARN would need to make sure two containers each with 1 pcores worth of vcores does in fact give you two cores( just like hypervisors schedule vcpus for the same VM on different pcores to ensure parallelism), but there would be no guarantee that two containers on the same host each w/ one vcore would run in parallel. Jobs that want fractional cores would just express 1vcore per container and work they're way up based on the experience running on the cluster (or also undo the virtualization by calculating vcore/pcore if they know what fraction of a pcore they want). Heterogenous CPUs does not fall out naturally (still need attributes) since there's no guarantee you can describe the difference between two CPUs is roughly 1 or more vcore (eg 2.4 - vs 2.0 Ghz < 1ECU), however there's no need for fractional vcores. I think either is reasonable and can be made to work, though I think #1 is preferable because: - Some frameworks want to express containers in physical resources (this is consistent with how YARN handles memory) - You can support jobs that don't want a full core via fractional cores (or slightly over-committing cores) - You can support heterogeneous cores via attributes ("I want equivalent containers") - vcores are optional anyway (only used in DRF) and therefore only need to be expressed if you care about physical cores because you need to reserve them or say you want a fraction of one Either way I think "vcore" is the wrong name either way because in #1 1vcore=1pcore so there's no virtualization and in #2 1 vcore is not a virtualization of a core (10 vcores does not give me 10 levels of parallelism), it's _just a unit_ (like an ECU). > Define a virtual core unambigiously > ----------------------------------- > > Key: YARN-1024 > URL: https://issues.apache.org/jira/browse/YARN-1024 > Project: Hadoop YARN > Issue Type: Improvement > Reporter: Arun C Murthy > Assignee: Arun C Murthy > > We need to clearly define the meaning of a virtual core unambiguously so that > it's easy to migrate applications between clusters. > For e.g. here is Amazon EC2 definition of ECU: > http://aws.amazon.com/ec2/faqs/#What_is_an_EC2_Compute_Unit_and_why_did_you_introduce_it > Essentially we need to clearly define a YARN Virtual Core (YVC). > Equivalently, we can use ECU itself: *One EC2 Compute Unit provides the > equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor.* -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira