[
https://issues.apache.org/jira/browse/YARN-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15723947#comment-15723947
]
Wangda Tan commented on YARN-3409:
----------------------------------
[~Naganarasimha],
bq. ... this is one such example we had faced earlier, but based on scenario it
can vary, but if we have a custom type and define how to compare it would be
better to adapt anything in future. ...
The problem is the predefined type cannot solve the problem like r+version, or
version in format like (x.y.z-build). And customized type sounds too heavy for
the solution, I don't really think we should support such complex case.
To me we should only support common constraint use cases. In my mind, we don't
need explicit type (I will explain why), and two internal type will be enough:
1) String:
2) Long array: Any version could be converted to long array. And I found all
the long constraints in design doc should be addressed by resource profile. But
if we really need it, we can treat it as a long\[1\], or have a new internal
type long.
3) Other type: I don't really think we need any other type, for example, double
sounds not necessary, and types like DATE/BOOLEAN can be covered by string.
The reason you mentioned to type is valid but we can still get good performance
without predefine the type. For example, we can have a internal field in
RMNodeLabal to save its long array (or any other required internal type beyond
string). This could be done automatically based on regex matching when we add
the constraint. And we can do the same thing on node constraint expr. Once they
are done, if the expr is >=, and both of constraint pattern and node constraint
have pre-saved long array, we can just compare the arrays.
For the type compatibility issue, if both of them have different internal type
(like long array v long array), we can downgrade to string comparison. This is
not a perfect solution, but I think we cannot handle it well in the
predefined-type too, for example, how to avoid two versions with different
format, one is 1.8.3 and another one is 1.8.
bq. further it would be better to have for constraint Name too, which will be
helpful for Boolean type.
I think we should have constraint name, a constraint should have name and
value, the type is arguable per my above comment.
Thoughts?
> Add constraint node labels
> --------------------------
>
> Key: YARN-3409
> URL: https://issues.apache.org/jira/browse/YARN-3409
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, capacityscheduler, client
> Reporter: Wangda Tan
> Assignee: Naganarasimha G R
> Attachments: Constraint-Node-Labels-Requirements-Design-doc_v1.pdf,
> YARN-3409.WIP.001.patch
>
>
> Specify only one label for each node (IAW, partition a cluster) is a way to
> determinate how resources of a special set of nodes could be shared by a
> group of entities (like teams, departments, etc.). Partitions of a cluster
> has following characteristics:
> - Cluster divided to several disjoint sub clusters.
> - ACL/priority can apply on partition (Only market team / marke team has
> priority to use the partition).
> - Percentage of capacities can apply on partition (Market team has 40%
> minimum capacity and Dev team has 60% of minimum capacity of the partition).
> Constraints are orthogonal to partition, they’re describing attributes of
> node’s hardware/software just for affinity. Some example of constraints:
> - glibc version
> - JDK version
> - Type of CPU (x86_64/i686)
> - Type of OS (windows, linux, etc.)
> With this, application can be able to ask for resource has (glibc.version >=
> 2.20 && JDK.version >= 8u20 && x86_64).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
