[
https://issues.apache.org/jira/browse/YARN-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15691048#comment-15691048
]
Naganarasimha G R commented on YARN-3409:
-----------------------------------------
Thanks [~bibinchundatt], [~varun_saxena] & [~rohithsharma], for taking the
discussion ahead.
bq. We should call current feature as NodeConstraints since we have label is
considered as cluster partition.Resource request as constraintsExpression
instead of constraintslabelexpression.
Actually how the way it has progressed is like there are 2 kinds of labels,
{{"Partition"}} where in its used to segregate the cluster & as well gurantee
the capacities, and other is {{"Constraints"}} which tries to capture the
attribute of the node. Agree that even if it sounds similar its different in
functionality but based on community feedback we can further decide on the
naming convention.
bq. Nodes with same Constraints could be in multiple partition,so are we
allowing resource to serve any labels satisfying constraints or only subset of
label resource pool satisfying constraints.
As Varun mentioned, it would be like for the nodes satisfying
partition(expression, multiple in case of ||), constraints expression would be
applied to further filter and do the allocation.
bq. At RM side is it required to have a constraints list?.We could consider all
constraints or attributes with which NM registers.For dynamic updation its
better that way rt?
As we have type for each constraint, we need to ensure that two NMs's dont give
constaints with same name but of different type. Hence we need to have a
superset in RM based on which evaluations can be done. Plans are there to
dynamically add the constraints and then register the NM-constraint mapping in
case the constriant doesnt exist.
bq. Any relaxation similar to node locality.If request doesnt satisfy
constraints will it ever serve request.If we are planning to support then
addition attribute in ResourceRequest.
Well currently we plan to make it strict enforcement for constraints and
relaxation, even if req we can have it when YARN-4902 supports
{code}
Delayed-OR [
- Give me default partition with constraint expr1,
(Wait for x mins)
- Any node in default partition.
]
{code}
bq. currently headroom is calculated per partition basis, how does head room
will be calculated once we support constraint label? What is the impact ? Does
it application wait for ever if required constraint label not found?
Yes as Varun mentioned Constraints will merely be used for node selection and
neither are capacities guranteed for it, and hence headroom will not be
considered. Headroom we can provide for a partition per application here it can
be an expression so it cannot be guaranteeing definite heardroom for
application as there can overlap of constraints across the resource requests.
As per current plan application will wait for ever if required constaint is not
available.
bq. So for constraint based Resource Request will it be always of type ANY .
Schedulers have node locality delay during allocation .Does node locality
matter for resource request with constraint label?
Yes similar to labels we initially plan to consider the constraint expression
present in ANY as final expression. As mentioned earlier nodes with matching
locality and constraints will be tried to picked if failed then nodes in the
same rack (of the specified nodes) satifying the constriant expression will be
picked else any nodes satisfying the constraint expression will be picked.
bq. Let us create a new branch for this work.
Yes [~varun_saxena] as per initial discussion with [~wangda], he also suggested
the same as the changes might not fit into smaller number of patches.
> Add constraint node labels
> --------------------------
>
> Key: YARN-3409
> URL: https://issues.apache.org/jira/browse/YARN-3409
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, capacityscheduler, client
> Reporter: Wangda Tan
> Assignee: Naganarasimha G R
> Attachments: Constraint-Node-Labels-Requirements-Design-doc_v1.pdf
>
>
> Specify only one label for each node (IAW, partition a cluster) is a way to
> determinate how resources of a special set of nodes could be shared by a
> group of entities (like teams, departments, etc.). Partitions of a cluster
> has following characteristics:
> - Cluster divided to several disjoint sub clusters.
> - ACL/priority can apply on partition (Only market team / marke team has
> priority to use the partition).
> - Percentage of capacities can apply on partition (Market team has 40%
> minimum capacity and Dev team has 60% of minimum capacity of the partition).
> Constraints are orthogonal to partition, they’re describing attributes of
> node’s hardware/software just for affinity. Some example of constraints:
> - glibc version
> - JDK version
> - Type of CPU (x86_64/i686)
> - Type of OS (windows, linux, etc.)
> With this, application can be able to ask for resource has (glibc.version >=
> 2.20 && JDK.version >= 8u20 && x86_64).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
