[
https://issues.apache.org/jira/browse/YARN-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15724142#comment-15724142
]
Naganarasimha G R commented on YARN-3409:
-----------------------------------------
[~leftnoteasy]
bq. The problem is the predefined type cannot solve the problem like r+version,
or version in format like (x.y.z-build). And customized type sounds too heavy
for the solution.
bq. This is not a perfect solution, but I think we cannot handle it well in the
predefined-type too, for example, how to avoid two versions with different
format, one is 1.8.3 and another one is 1.8.
well its not one version type solving for all versions, what i tried to capture
was representation of one such version comparison. Version format is not going
to be so dynamic for a software that we add or remove dynamically some sections
to it. so if for a given software they want to parse string at the end or the
number of longs separated by a "." can be easily custom coded into a class.
bq. Long array: Any version could be converted to long array.
the above example {{x.y.z-build}} which you gave doesnt fit in long array right
? suppose we some how come with a pattern to match these stuff with a internal
structure, how many such patterns we need to support and how to verify patterns
against other patterns ?
bq. And I found all the long constraints in design doc should be addressed by
resource profile.
LONG types in the example are
{code}
NUM_OF_DISKS :=> run container on nodes which has more local disks. Spark kind
of tasks doing a lot of data processing would like to have more disks to
process the data parallelly,
NUM_OF_INTF :=> run container on nodes which has dual networks (generally
required for AM’s to be run in dual plane so that its accessible for clients),
MAX_MEM :=> run container on nodes which has more memory (so that there will be
chances for resizing the container even though the current allocation is Less ,
suitable for spark )
{code}
all these scenarios captured are of non consumeable resources types, hence
cannot be addressed as another resource type (so as to solve through resource
profiles).
bq. But if we really need it, we can treat it as a long\[1\], or have a new
internal type long.
how to capture internal type as long/ string or any other custom type ? if its
inherent logic and if it doesnt match what user intended then none of the
resource requests matches or lead to undesired allocations.
bq. Other type: I don't really think we need any other type, for example,
double sounds not necessary, and types like DATE/BOOLEAN can be covered by
string.
Agree that Boolean can be acheived with String itself and for DATE might not be
immediately required but consider for date different formats of parsing to be
supported. And also consider the scenario where in we want to schedule on the
node where load average for past (1/5/15) mins is less than a particular value ?
All these kinds non consumables resources are supported in one way or the other
in other schedulers
bq. For example, we can have a internal field in RMNodeLabal to save its long
array (or any other required internal type beyond string).
Based on regex identifying internal type is risky we may match wrongly than
what user intends to (if we do it based on some order), this would lead to
allocations happening in a undesired manner. Also how/who provides what is the
internal type/regex again there would be some kind of input which needs to be
taken for a constraint right ? If its predefined type then as mentioned it
would be done in some order which might lead to undesired allocations.
bq. For the type compatibility issue, if both of them have different internal
type (like long array v long array), we can downgrade to string comparison.
This is not a perfect solution, but I think we cannot handle it well in the
predefined-type too.
If we downgrade to string comarison for long and decimal values then obviously
there will be cases where in compare operations (like greater than , less than)
will fail.
And in my predefined type we do not allow constraint expression if it doesnt
match the constraint type and for the example (1.8 and 1.8.3), it needs to be
handled as custom version type rather a generic version type.
bq. And customized type sounds too heavy for the solution, I don't really think
we should support such complex case.
Well is it heavy in terms of implementation or usage ? As we keep string as the
default type in usage front its pretty simple and implementation and
maintainence front i dont see too much of a problem other than changes which is
required on the existing labels.
Most of the back end implementation would hardly require any huge change as it
needs to store another kind and we can make use most of the commonNodeLabels
Manager and store.
IMHO I would like it to be flexible enough for future scenarios too hence i
would still prefer for Constraint Type.
> Add constraint node labels
> --------------------------
>
> Key: YARN-3409
> URL: https://issues.apache.org/jira/browse/YARN-3409
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, capacityscheduler, client
> Reporter: Wangda Tan
> Assignee: Naganarasimha G R
> Attachments: Constraint-Node-Labels-Requirements-Design-doc_v1.pdf,
> YARN-3409.WIP.001.patch
>
>
> Specify only one label for each node (IAW, partition a cluster) is a way to
> determinate how resources of a special set of nodes could be shared by a
> group of entities (like teams, departments, etc.). Partitions of a cluster
> has following characteristics:
> - Cluster divided to several disjoint sub clusters.
> - ACL/priority can apply on partition (Only market team / marke team has
> priority to use the partition).
> - Percentage of capacities can apply on partition (Market team has 40%
> minimum capacity and Dev team has 60% of minimum capacity of the partition).
> Constraints are orthogonal to partition, they’re describing attributes of
> node’s hardware/software just for affinity. Some example of constraints:
> - glibc version
> - JDK version
> - Type of CPU (x86_64/i686)
> - Type of OS (windows, linux, etc.)
> With this, application can be able to ask for resource has (glibc.version >=
> 2.20 && JDK.version >= 8u20 && x86_64).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
