[
https://issues.apache.org/jira/browse/YARN-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096933#comment-16096933
]
Wangda Tan commented on YARN-3409:
----------------------------------
Just read above comments from [~sunilg]/[~grey]/[[email protected]]
regarding to resource v.s. label.
To me, this discussion started since design phase of YARN-796 (3 years ago).
Like [~grey] said, all the labels (including partition/constraints) are used
for filtering, and what how much resource can be consumed after container
allocated is determined by resource only. NM will not do any check for
partition/attribute from coming containers, and AM will not send such
partition/attribute to NM as part of ContainerLaunchContext as well.
As [~sunilg] mentioned, change of partition will not cause container killed by
NM, instead, since it change resource accounting (from use 10G of partition-x
to 10G of partition-y), so preemption could kick-in, but this is completely
decided by RM.
bq. The big advantage I see is that we don't have two duplicate sets of code.
If we put two things into Resource, we have to define type of each resource
vector (consumable/non-consumable/string-attribute/node-partition). Which could
be very unclear and inefficient for RM to handle them as well. From what I can
see, modern schedulers are mostly use separated concept of resource and
attributes. And this cannot avoid having separate implementation to handle
label/resource
> Add constraint node labels
> --------------------------
>
> Key: YARN-3409
> URL: https://issues.apache.org/jira/browse/YARN-3409
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, capacityscheduler, client
> Reporter: Wangda Tan
> Assignee: Naganarasimha G R
> Attachments: Constraint-Node-Labels-Requirements-Design-doc_v1.pdf,
> YARN-3409.WIP.001.patch
>
>
> Specify only one label for each node (IAW, partition a cluster) is a way to
> determinate how resources of a special set of nodes could be shared by a
> group of entities (like teams, departments, etc.). Partitions of a cluster
> has following characteristics:
> - Cluster divided to several disjoint sub clusters.
> - ACL/priority can apply on partition (Only market team / marke team has
> priority to use the partition).
> - Percentage of capacities can apply on partition (Market team has 40%
> minimum capacity and Dev team has 60% of minimum capacity of the partition).
> Constraints are orthogonal to partition, they’re describing attributes of
> node’s hardware/software just for affinity. Some example of constraints:
> - glibc version
> - JDK version
> - Type of CPU (x86_64/i686)
> - Type of OS (windows, linux, etc.)
> With this, application can be able to ask for resource has (glibc.version >=
> 2.20 && JDK.version >= 8u20 && x86_64).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
