[
https://issues.apache.org/jira/browse/YARN-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023067#comment-16023067
]
Lei Guo commented on YARN-3409:
-------------------------------
[~templedf], we also came from HPC background, and had discussions on similiar
approach last year with [~leftnoteasy]/[~kkaranasos]/others. For scheduling
purpose, there are always two phases: filtering and placement. The filtering
phase is to reduce the search scope of candidates; while the placement phase is
to find the best candidate and allocate resources. Both consumable resource and
non-consumable resource could be used in the filtering phase, and only the
consumable resource involved in the second phase.
For HPC world, we model everything as resource, so no logic duplication in
filtering phase, while the placement phase need some checking on whether the
resource requested is consumable.
For Yarn, so far the resource defined in YARN-3926 are consumable resource, and
partition label (close to host partition in HPC) is already introduced for
filtering phase, there is no obvious advantage of modeling constraints as
resource like in HPC world. So we concluded constraints could be modeled
similar to label instead of resource for better consistency, Basically, all
Labels (partition label or constraint label) are only for the filtering; and
all resource are consumable.
> Add constraint node labels
> --------------------------
>
> Key: YARN-3409
> URL: https://issues.apache.org/jira/browse/YARN-3409
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, capacityscheduler, client
> Reporter: Wangda Tan
> Assignee: Naganarasimha G R
> Attachments: Constraint-Node-Labels-Requirements-Design-doc_v1.pdf,
> YARN-3409.WIP.001.patch
>
>
> Specify only one label for each node (IAW, partition a cluster) is a way to
> determinate how resources of a special set of nodes could be shared by a
> group of entities (like teams, departments, etc.). Partitions of a cluster
> has following characteristics:
> - Cluster divided to several disjoint sub clusters.
> - ACL/priority can apply on partition (Only market team / marke team has
> priority to use the partition).
> - Percentage of capacities can apply on partition (Market team has 40%
> minimum capacity and Dev team has 60% of minimum capacity of the partition).
> Constraints are orthogonal to partition, they’re describing attributes of
> node’s hardware/software just for affinity. Some example of constraints:
> - glibc version
> - JDK version
> - Type of CPU (x86_64/i686)
> - Type of OS (windows, linux, etc.)
> With this, application can be able to ask for resource has (glibc.version >=
> 2.20 && JDK.version >= 8u20 && x86_64).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
