[
https://issues.apache.org/jira/browse/YARN-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13730301#comment-13730301
]
Eli Collins edited comment on YARN-1024 at 8/6/13 3:09 AM:
-----------------------------------------------------------
I agree we need to define the meaning of a virtual core unambiguously
(otherwise we won't be able to support two different frameworks on the same
cluster that may have differing ideas of what a vcore is). I also agree with
Phil that there are essentially two high-level use cases:
1. Jobs that want to express how much CPU capacity the job needs. Real world
example - a distcp job wants to express it needs 100 containers but only a
fraction of a CPU for each since it will spend most of its time blocking on IO.
2. Services - ie long-lived frameworks (ie support 2-level scheduling) - that
want to request cores on many machines on a cluster and want to express
CPU-level parallelism and aggregate demand (because they will schedule
fine-grain requests w/in their long-lived containers). Eg a framework should be
able to ask for two containers on a host, each with one core, so it can get two
containers that can execute in parallel on a full core. This is assuming we
plan to support long-running services in Yarn (YARN-896), which is hopefully
not controversial. Real world example is HBase which may want 2 guaranteed
cores per host on a given set of hosts.
Seems like there are two high-level approaches:
1. Get rid of vcores. If we define 1vcore=1pcore (1vcore=1vcpu for virtual
environments) and support fractional cores (YARN-972) then services can ask for
1 or more vcores knowing they're getting real cores and jobs just ask for what
fraction of a vcore they think they need. This is really abandoning the concept
of a "virtual core" because it's actually expressing a physical requirement
(like memory, we assume Yarn is not dramatically over-committing the host). We
can handle heterogeneous CPUs via attributes (as discussed in other Yarn jiras)
since most clusters in my experience don't have wildly different processors (eg
1 or 2 generations is common), and attributes are sufficient to express
policies like "all my cores should have equal/comparable performance".
2. Keep going with vcores as a CPU unit of measurement. If we define
1vcore=1ECU (works 1:1 for virtual environments) then services need to
understand the the power of a core so they can ask for that many vcores -
essentially they are just undoing the virtualization. YARN would need to make
sure two containers each with 1 pcores worth of vcores does in fact give you
two cores (just like hypervisors schedule vcpus for the same VM on different
pcores to ensure parallelism), but there would be no guarantee that two
containers on the same host each w/ one vcore would run in parallel. Jobs that
want fractional cores would just express 1vcore per container and work their
way up based on the experience running on the cluster (or also undo the
virtualization by calculating vcore/pcore if they know what fraction of a pcore
they want). Heterogenous CPUs does not fall out naturally (still need
attributes) since there's no guarantee you can describe the difference between
two CPUs is roughly 1 or more vcore (eg 2.4 - vs 2.0 Ghz < 1ECU), however
there's no need for fractional vcores.
I think either is reasonable and can be made to work, though I think #1 is
preferable because:
- Some frameworks want to express containers in physical resources (this is
consistent with how YARN handles memory)
- You can support jobs that don't want a full core via fractional cores (or
slightly over-committing cores)
- You can support heterogeneous cores via attributes ("I want equivalent
containers")
- vcores are optional anyway (only used in DRF) and therefore only need to be
expressed if you care about physical cores because you need to reserve them or
say you want a fraction of one
Either way I think "vcore" is the wrong name because in #1 1vcore=1pcore so
there's no virtualization and in #2 1 vcore is not a virtualization of a core
(10 vcores does not give me 10 levels of parallelism), it's _just a unit_ (like
an ECU).
was (Author: eli):
I agree we need to define the meaning of a virtual core unambiguously
(otherwise we won't be able to support two different frameworks on the same
cluster that may have differing ideas of what a vcore is). I also agree with
Phil that there are essentially two high-level use cases:
1. Jobs that want to express how much CPU capacity the job needs. Real world
example - a distcp job wants to express it needs 100 containers but only a
fraction of a CPU for each since it will spend most of its time blocking on IO.
2. Services - ie long-lived frameworks (ie support 2-level scheduling) - that
want to request cores on many machines on a cluster and want to express
CPU-level parallelism and aggregate demand (because they will schedule
fine-grain requests w/in their long-lived containers). Eg a framework should be
able to ask for two containers on a host, each with one core, so it can get two
containers that can execute in parallel on a full core. This is assuming we
plan to support long-running services in Yarn (YARN-896), which is hopefully
not controversial. Real world example is HBase which may want 2 guaranteed
cores per host on a given set of hosts.
Seems like there are two high-level approaches:
1. Get rid of vcores. If we define 1vcore=1pcore (1vcore=1vcpu for virtual
environments) and support fractional cores (YARN-972) then services can ask for
1 or more vcores knowing they're getting real cores and jobs just ask for what
fraction of a vcore they think they need. This is really abandoning the concept
of a "virtual core" because it's actually expressing a physical requirement
(like memory, we assume Yarn is not dramatically over-committing the host). We
can handle heterogeneous CPUs via attributes (as discussed in other Yarn jiras)
since most clusters in my experience don't have wildly different processors (eg
1 or 2 generations is common), and attributes are sufficient to express
policies like "all my cores should have equal/comparable performance".
2. Keep going with vcores as a CPU unit of measurement. If we define
1vcore=1ECU (works 1:1 for virtual environments) then services need to
understand the the power of a core so they can ask for that many vcores -
essentially they are just undoing the virtualization. YARN would need to make
sure two containers each with 1 pcores worth of vcores does in fact give you
two cores (just like hypervisors schedule vcpus for the same VM on different
pcores to ensure parallelism), but there would be no guarantee that two
containers on the same host each w/ one vcore would run in parallel. Jobs that
want fractional cores would just express 1vcore per container and work their
way up based on the experience running on the cluster (or also undo the
virtualization by calculating vcore/pcore if they know what fraction of a pcore
they want). Heterogenous CPUs does not fall out naturally (still need
attributes) since there's no guarantee you can describe the difference between
two CPUs is roughly 1 or more vcore (eg 2.4 - vs 2.0 Ghz < 1ECU), however
there's no need for fractional vcores.
I think either is reasonable and can be made to work, though I think #1 is
preferable because:
- Some frameworks want to express containers in physical resources (this is
consistent with how YARN handles memory)
- You can support jobs that don't want a full core via fractional cores (or
slightly over-committing cores)
- You can support heterogeneous cores via attributes ("I want equivalent
containers")
- vcores are optional anyway (only used in DRF) and therefore only need to be
expressed if you care about physical cores because you need to reserve them or
say you want a fraction of one
Either way I think "vcore" is the wrong name either way because in #1
1vcore=1pcore so there's no virtualization and in #2 1 vcore is not a
virtualization of a core (10 vcores does not give me 10 levels of parallelism),
it's _just a unit_ (like an ECU).
> Define a virtual core unambigiously
> -----------------------------------
>
> Key: YARN-1024
> URL: https://issues.apache.org/jira/browse/YARN-1024
> Project: Hadoop YARN
> Issue Type: Improvement
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
>
> We need to clearly define the meaning of a virtual core unambiguously so that
> it's easy to migrate applications between clusters.
> For e.g. here is Amazon EC2 definition of ECU:
> http://aws.amazon.com/ec2/faqs/#What_is_an_EC2_Compute_Unit_and_why_did_you_introduce_it
> Essentially we need to clearly define a YARN Virtual Core (YVC).
> Equivalently, we can use ECU itself: *One EC2 Compute Unit provides the
> equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor.*
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
