[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[ https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347096#comment-16347096 ] Hudson commented on YARN-6593: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13589 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13589/]) YARN-6593. [API] Introduce Placement Constraint object. (Konstantinos (arun suresh: rev 33a796d9b778bf7350e87a4e36ca30c925cf7036) * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/resource/PlacementConstraint.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ProtoUtils.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/resource/PlacementConstraintTransformations.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/resource/package-info.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/resource/package-info.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/pb/PlacementConstraintToProtoConverter.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/resource/TestPlacementConstraintTransformations.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/pb/package-info.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/test/java/org/apache/hadoop/yarn/api/resource/TestPlacementConstraints.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/pb/PlacementConstraintFromProtoConverter.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/resource/PlacementConstraints.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestPlacementConstraintPBConversion.java > [API] Introduce Placement Constraint object > --- > > Key: YARN-6593 > URL: https://issues.apache.org/jira/browse/YARN-6593 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos >Priority: Major > Fix For: YARN-6592 > > Attachments: YARN-6593.001.patch, YARN-6593.002.patch, > YARN-6593.003.patch, YARN-6593.004.patch, YARN-6593.005.patch, > YARN-6593.006.patch, YARN-6593.007.patch, YARN-6593.008.patch > > > Just removed Fixed version and moved it to target version as we set fix > version only after patch is committed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[ https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113584#comment-16113584 ] Konstantinos Karanasos commented on YARN-6593: -- bq. Committed to YARN-6592, thanks Konstantinos Karanasos and thanks reviews from Chris Douglas, Arun Suresh, Subru Krishnan, Jian He, Panagiotis Garefalakis, Sunil G. (hopefully I mentioned everyone), Great, thanks [~leftnoteasy]! And thanks everyone for the reviews! > [API] Introduce Placement Constraint object > --- > > Key: YARN-6593 > URL: https://issues.apache.org/jira/browse/YARN-6593 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Fix For: YARN-6592 > > Attachments: YARN-6593.001.patch, YARN-6593.002.patch, > YARN-6593.003.patch, YARN-6593.004.patch, YARN-6593.005.patch, > YARN-6593.006.patch, YARN-6593.007.patch, YARN-6593.008.patch > > > Just removed Fixed version and moved it to target version as we set fix > version only after patch is committed. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[ https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113489#comment-16113489 ] Wangda Tan commented on YARN-6593: -- [~kkaranasos], you were asking this question in good time, I'm committing it now. :) > [API] Introduce Placement Constraint object > --- > > Key: YARN-6593 > URL: https://issues.apache.org/jira/browse/YARN-6593 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-6593.001.patch, YARN-6593.002.patch, > YARN-6593.003.patch, YARN-6593.004.patch, YARN-6593.005.patch, > YARN-6593.006.patch, YARN-6593.007.patch, YARN-6593.008.patch > > > Just removed Fixed version and moved it to target version as we set fix > version only after patch is committed. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[ https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113486#comment-16113486 ] Konstantinos Karanasos commented on YARN-6593: -- I opened YARN-6942 for adding usage examples. [~leftnoteasy], it seems we are all good with this JIRA -- shall we go on and commit it? > [API] Introduce Placement Constraint object > --- > > Key: YARN-6593 > URL: https://issues.apache.org/jira/browse/YARN-6593 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-6593.001.patch, YARN-6593.002.patch, > YARN-6593.003.patch, YARN-6593.004.patch, YARN-6593.005.patch, > YARN-6593.006.patch, YARN-6593.007.patch, YARN-6593.008.patch > > > Just removed Fixed version and moved it to target version as we set fix > version only after patch is committed. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[ https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16110278#comment-16110278 ] Jian He commented on YARN-6593: --- Just commented on YARN-6594. > [API] Introduce Placement Constraint object > --- > > Key: YARN-6593 > URL: https://issues.apache.org/jira/browse/YARN-6593 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-6593.001.patch, YARN-6593.002.patch, > YARN-6593.003.patch, YARN-6593.004.patch, YARN-6593.005.patch, > YARN-6593.006.patch, YARN-6593.007.patch, YARN-6593.008.patch > > > Just removed Fixed version and moved it to target version as we set fix > version only after patch is committed. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[ https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16110248#comment-16110248 ] Jian He commented on YARN-6593: --- The use-case is to be able to select containers based on key and values, say, I want to find my containers with version=v1, and env= test, and name=web1 or name = web2. Yes, currently it's a set of values not a single value, I misspoke. We can still make it work by searching the entire string as Arun said. But that's not explicit. The AM, client, or even UI then needs to parse the string to extract the keys and values. I was thinking this change will affect this jira's implementation also, hence continue commenting here. If you think this anyways can be revisited. We can commit this first and work on it in YARN-6594. It doesn't matter to me. > [API] Introduce Placement Constraint object > --- > > Key: YARN-6593 > URL: https://issues.apache.org/jira/browse/YARN-6593 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-6593.001.patch, YARN-6593.002.patch, > YARN-6593.003.patch, YARN-6593.004.patch, YARN-6593.005.patch, > YARN-6593.006.patch, YARN-6593.007.patch, YARN-6593.008.patch > > > Just removed Fixed version and moved it to target version as we set fix > version only after patch is committed. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[ https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16110175#comment-16110175 ] Konstantinos Karanasos commented on YARN-6593: -- Regarding the class with examples, as you say, let's move examples to a separate JIRA. Especially given that placement constraints can be specified both at the container level (YARN-6594) and the application level (YARN-6595). It would be good to give all the possible ways to define constraints there. I would like to understand better the use cases that would require allocation tags to have values. But let's move the discussion to YARN-6594. BTW, to be precise, allocation tags in constraints are modeled as a set of values (not a single value) with key being null. We opted for this approach after discussions with [~leftnoteasy] and [~arun.sur...@gmail.com] in order to not need multiple constraint objects if we want to specify a list of tags in the constraint. PS: I am traveling today, thus the late responses. > [API] Introduce Placement Constraint object > --- > > Key: YARN-6593 > URL: https://issues.apache.org/jira/browse/YARN-6593 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-6593.001.patch, YARN-6593.002.patch, > YARN-6593.003.patch, YARN-6593.004.patch, YARN-6593.005.patch, > YARN-6593.006.patch, YARN-6593.007.patch, YARN-6593.008.patch > > > Just removed Fixed version and moved it to target version as we set fix > version only after patch is committed. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[ https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16110036#comment-16110036 ] Jian He commented on YARN-6593: --- bq. Examples are essential, but can that be part of a followup JIRA? Particularly since the implementation(s) may affect the API. sure, no problem with that. Actually, what's being done in YARN-6594 may also affect this jira's implementation: Regarding container tags, IMO, there are two types of tags. 1) Ones for scheduling decisions which is done by this umberlla jira. (This is modeled as single value in current patch ) 2) Ones for annotating metaInfo for container, which this umberlla jira originally didn't cover. Right now 1) is modeled as a single value. For 2), I expect it to be a key/value pair to support various use-case. Question is whether we should model both as key/value pairs for consistency. I checked with Wangda, we prefer it to be the same for consistency user experience. Btw, Kubernetes also models both as key/value pairs. > [API] Introduce Placement Constraint object > --- > > Key: YARN-6593 > URL: https://issues.apache.org/jira/browse/YARN-6593 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-6593.001.patch, YARN-6593.002.patch, > YARN-6593.003.patch, YARN-6593.004.patch, YARN-6593.005.patch, > YARN-6593.006.patch, YARN-6593.007.patch, YARN-6593.008.patch > > > Just removed Fixed version and moved it to target version as we set fix > version only after patch is committed. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[ https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16109976#comment-16109976 ] Chris Douglas commented on YARN-6593: - bq. The only thing remaining in this jira is the example class for how to use the APIs - whether it's worth to do or not ? Examples are essential, but can that be part of a followup JIRA? Particularly since the implementation(s) may affect the API. > [API] Introduce Placement Constraint object > --- > > Key: YARN-6593 > URL: https://issues.apache.org/jira/browse/YARN-6593 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-6593.001.patch, YARN-6593.002.patch, > YARN-6593.003.patch, YARN-6593.004.patch, YARN-6593.005.patch, > YARN-6593.006.patch, YARN-6593.007.patch, YARN-6593.008.patch > > > Just removed Fixed version and moved it to target version as we set fix > version only after patch is committed. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[ https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16109961#comment-16109961 ] Jian He commented on YARN-6593: --- Let's move the discussion there. The only thing remaining in this jira is the example class for how to use the APIs - whether it's worth to do or not ? > [API] Introduce Placement Constraint object > --- > > Key: YARN-6593 > URL: https://issues.apache.org/jira/browse/YARN-6593 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-6593.001.patch, YARN-6593.002.patch, > YARN-6593.003.patch, YARN-6593.004.patch, YARN-6593.005.patch, > YARN-6593.006.patch, YARN-6593.007.patch, YARN-6593.008.patch > > > Just removed Fixed version and moved it to target version as we set fix > version only after patch is committed. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[ https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16109921#comment-16109921 ] Arun Suresh commented on YARN-6593: --- bq. Actually, my comments should apply to YARN-6594 rather this jira. [~jianhe], I am guessing you are referring to the *AllocationTags* .. then yes, we should continue that discussion in YARN-6594, since this pertains just to the *PlacementConstraint* object. (Can you please cross-post it there as well ?) My opinion though is that the allocation tags should remain a list of string keys. I understand this might offload some work from the AM, but do you see it helping the RM/scheduler in making a scheduling decision ? If it is just to select a bunch of containers, one can still perform a search like: select containers where allocationtags contains "env=test" etc. Forcing the tags to be a map might even make the querying more complicated. > [API] Introduce Placement Constraint object > --- > > Key: YARN-6593 > URL: https://issues.apache.org/jira/browse/YARN-6593 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-6593.001.patch, YARN-6593.002.patch, > YARN-6593.003.patch, YARN-6593.004.patch, YARN-6593.005.patch, > YARN-6593.006.patch, YARN-6593.007.patch, YARN-6593.008.patch > > > Just removed Fixed version and moved it to target version as we set fix > version only after patch is committed. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[ https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16109872#comment-16109872 ] Jian He commented on YARN-6593: --- Actually, my comments should apply to YARN-6594 rather this jira. For example, I want to associate some meta info like "env=test" and "version=v2" info to the container and the AM can define the keys by itself. AM can then do select based on these key and values for the containers returned. To me, the tag is not just used for scheduling decisions. It's also for associating metainfo to the container, otherwise, the AM has to maintain and persist the mapping by itself. If YARN natively supports this, it'll be a lot easier for the AM. bq. I have some first examples in PlacementConstraints. Do you think we should add them in a different class? I was thinking having a class which lists all sorts scenarios and the API definitions, user can pretty much copy the semantics. Reading javadoc is one option but still not straightfoward. > [API] Introduce Placement Constraint object > --- > > Key: YARN-6593 > URL: https://issues.apache.org/jira/browse/YARN-6593 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-6593.001.patch, YARN-6593.002.patch, > YARN-6593.003.patch, YARN-6593.004.patch, YARN-6593.005.patch, > YARN-6593.006.patch, YARN-6593.007.patch, YARN-6593.008.patch > > > Just removed Fixed version and moved it to target version as we set fix > version only after patch is committed. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[
https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16109100#comment-16109100
]
Konstantinos Karanasos commented on YARN-6593:
--
Thanks for the feedback, [~jianhe].
bq. should we have an example class to demonstrate how to use the APIs for
different scenarios as mentioned in the document? I think that's useful for
users.
I have some first examples in {{PlacementConstraints}}. Do you think we should
add them in a different class?
bq. looks like the allocationTag is modeled as a single key, I think a
key/value pair will be more flexible ? Like I want to associate different
dimensions of informations to the container.
We talked a lot about this with [~leftnoteasy] and [~arun.sur...@gmail.com],
and decided it would be easier for the users to add the tags as a value with
null key, given that tags do not have values. Still we can add multiple tags to
a container. Can you give an example that we cannot support with the current
proposal?
> [API] Introduce Placement Constraint object
> ---
>
> Key: YARN-6593
> URL: https://issues.apache.org/jira/browse/YARN-6593
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Attachments: YARN-6593.001.patch, YARN-6593.002.patch,
> YARN-6593.003.patch, YARN-6593.004.patch, YARN-6593.005.patch,
> YARN-6593.006.patch, YARN-6593.007.patch, YARN-6593.008.patch
>
>
> Just removed Fixed version and moved it to target version as we set fix
> version only after patch is committed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[ https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16107663#comment-16107663 ] Jian He commented on YARN-6593: --- looks like the allocationTag is modeled as a single key, I think a key/value pair will be more flexible ? Like I want to associate different dimensions of informations to the container. > [API] Introduce Placement Constraint object > --- > > Key: YARN-6593 > URL: https://issues.apache.org/jira/browse/YARN-6593 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-6593.001.patch, YARN-6593.002.patch, > YARN-6593.003.patch, YARN-6593.004.patch, YARN-6593.005.patch, > YARN-6593.006.patch, YARN-6593.007.patch, YARN-6593.008.patch > > > Just removed Fixed version and moved it to target version as we set fix > version only after patch is committed. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[ https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16107632#comment-16107632 ] Jian He commented on YARN-6593: --- should we have an example class to demonstrate how to use the APIs? I think that's useful for users. > [API] Introduce Placement Constraint object > --- > > Key: YARN-6593 > URL: https://issues.apache.org/jira/browse/YARN-6593 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-6593.001.patch, YARN-6593.002.patch, > YARN-6593.003.patch, YARN-6593.004.patch, YARN-6593.005.patch, > YARN-6593.006.patch, YARN-6593.007.patch, YARN-6593.008.patch > > > Just removed Fixed version and moved it to target version as we set fix > version only after patch is committed. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[ https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105796#comment-16105796 ] Konstantinos Karanasos commented on YARN-6593: -- Sounds good, thanks [~leftnoteasy] and [~chris.douglas] for the reviews and the feedback! > [API] Introduce Placement Constraint object > --- > > Key: YARN-6593 > URL: https://issues.apache.org/jira/browse/YARN-6593 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-6593.001.patch, YARN-6593.002.patch, > YARN-6593.003.patch, YARN-6593.004.patch, YARN-6593.005.patch, > YARN-6593.006.patch, YARN-6593.007.patch, YARN-6593.008.patch > > > Just removed Fixed version and moved it to target version as we set fix > version only after patch is committed. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[ https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105689#comment-16105689 ] Wangda Tan commented on YARN-6593: -- Latest patch looks good, +1, I will commit it next Tue (give a couple of days for people to review). Thanks [~kkaranasos] > [API] Introduce Placement Constraint object > --- > > Key: YARN-6593 > URL: https://issues.apache.org/jira/browse/YARN-6593 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-6593.001.patch, YARN-6593.002.patch, > YARN-6593.003.patch, YARN-6593.004.patch, YARN-6593.005.patch, > YARN-6593.006.patch, YARN-6593.007.patch, YARN-6593.008.patch > > > Just removed Fixed version and moved it to target version as we set fix > version only after patch is committed. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[
https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105328#comment-16105328
]
Hadoop QA commented on YARN-6593:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
29s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
24s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 48m 13s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6593 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12879375/YARN-6593.008.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux dab341b1e7eb 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 369f731 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/16589/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U:
hadoop-yarn-project/hadoop-yarn |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/16589/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> [API] Introduce Placement Constraint object
>
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[
https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16104341#comment-16104341
]
Hadoop QA commented on YARN-6593:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
44s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
53s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 59s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 1 new + 24 unchanged - 0 fixed = 25 total (was 24) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
33s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
24s{color} | {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api
generated 1 new + 123 unchanged - 0 fixed = 124 total (was 123) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
33s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
27s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 60m 57s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6593 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12879279/YARN-6593.007.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux 76744de0f9fc 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 38c6fa5 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/16585/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
|
| javadoc |
https://builds.apache.org/job/PreCommit-YARN-Build/16585/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt
|
| Test
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[
https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16104264#comment-16104264
]
Konstantinos Karanasos commented on YARN-6593:
--
Thanks [~leftnoteasy] -- I addressed your comments in the latest patch.
Note: regarding the #NODE/RACK, I made them package protected in
{{PlacementConstraint}}, but kept them public in {{PlacementConstraints}}
because they are required for applications to define the scope of the
constraints.
> [API] Introduce Placement Constraint object
> ---
>
> Key: YARN-6593
> URL: https://issues.apache.org/jira/browse/YARN-6593
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Attachments: YARN-6593.001.patch, YARN-6593.002.patch,
> YARN-6593.003.patch, YARN-6593.004.patch, YARN-6593.005.patch,
> YARN-6593.006.patch, YARN-6593.007.patch
>
>
> Just removed Fixed version and moved it to target version as we set fix
> version only after patch is committed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[
https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16103963#comment-16103963
]
Wangda Tan commented on YARN-6593:
--
Thanks [~kkaranasos] for updating the patch, could you check the reported
Javadocs warning?
In addition to that, some comments for API annotations:
1) Since we're in the early phase of the feature, instead of marking
PlacementConstraint(s) to be public/evolving, I prefer to mark them to
public/unstable.
2) private/evolving is not proper, private should come unstable or empty (Just
like CapacitySchedulerConfiguration) always.
3) Do we really need to mark {{api/resource/package-info.java}} to public? It
could accidentally add {{@public}} classes in the future
4) A couple of {{static final}} fields in PlacementConstraint(s) should be
{{@private}}, for example: {{PlacementConstraints#NODE/RACK}}.
> [API] Introduce Placement Constraint object
> ---
>
> Key: YARN-6593
> URL: https://issues.apache.org/jira/browse/YARN-6593
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Attachments: YARN-6593.001.patch, YARN-6593.002.patch,
> YARN-6593.003.patch, YARN-6593.004.patch, YARN-6593.005.patch,
> YARN-6593.006.patch
>
>
> Just removed Fixed version and moved it to target version as we set fix
> version only after patch is committed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[
https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16102601#comment-16102601
]
Hadoop QA commented on YARN-6593:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
56s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 3m
54s{color} | {color:red} hadoop-yarn in trunk failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 9m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
30s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 53s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 5 new + 24 unchanged - 0 fixed = 29 total (was 24) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
38s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
25s{color} | {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api
generated 1 new + 123 unchanged - 0 fixed = 124 total (was 123) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
32s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
35s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 54m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6593 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12879089/YARN-6593.006.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux e141d37d973f 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 27a1a5f |
| Default Java | 1.8.0_131 |
| compile |
https://builds.apache.org/job/PreCommit-YARN-Build/16566/artifact/patchprocess/branch-compile-hadoop-yarn-project_hadoop-yarn.txt
|
| findbugs | v3.1.0-RC1 |
| checkstyle |
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[
https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100778#comment-16100778
]
Hadoop QA commented on YARN-6593:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
47s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 51s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 61 new + 24 unchanged - 0 fixed = 85 total (was 24) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
23s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
25s{color} | {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api
generated 2 new + 123 unchanged - 0 fixed = 125 total (was 123) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
33s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
25s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
33s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 51m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6593 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12878896/YARN-6593.005.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux d610b38f0c6b 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / f81a4ef |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/16543/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
|
| whitespace |
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[ https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16096506#comment-16096506 ] Wangda Tan commented on YARN-6593: -- [~kkaranasos], sounds good. Thanks > [API] Introduce Placement Constraint object > --- > > Key: YARN-6593 > URL: https://issues.apache.org/jira/browse/YARN-6593 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Fix For: 3.0.0-alpha3 > > Attachments: YARN-6593.001.patch, YARN-6593.002.patch, > YARN-6593.003.patch, YARN-6593.004.patch > > > This JIRA introduces an object for defining placement constraints. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[ https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095566#comment-16095566 ] Konstantinos Karanasos commented on YARN-6593: -- Thanks for the feedback, [~leftnoteasy] and [~chris.douglas]! I will address all your comments and upload a new version. bq. I'm not sure when you plan to use SingleConstraintTransformer/SpecializedConstraintTransformer. If you agree that they should not be user-facing, do you think we should move them to a separate JIRA while doing implementation? I agree they should not be user-facing -- I will fix the annotations. The transformations will, for example, be used at the scheduler side, if the scheduler wants to do something with specific cardinality or target constraints. I would suggest to keep it at this JIRA so that we have all these related changes clubbed together. It seems that the problem of updating constraints would require more discussion. Since this patch is already introducing a lot of new stuff, I would also suggest to push the update to future work, after we discuss what the exact semantics of updating constraints are. A first solution to the problem would be, like you say, to invalidate an existing scheduling request and send a new one. > [API] Introduce Placement Constraint object > --- > > Key: YARN-6593 > URL: https://issues.apache.org/jira/browse/YARN-6593 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Fix For: 3.0.0-alpha3 > > Attachments: YARN-6593.001.patch, YARN-6593.002.patch, > YARN-6593.003.patch, YARN-6593.004.patch > > > This JIRA introduces an object for defining placement constraints. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[ https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095440#comment-16095440 ] Wangda Tan commented on YARN-6593: -- bq. Do you have specific use cases in mind? Regarding to LRA, change of cardinality is one of the most important requirement, for example, increase #HTTP-instances for load balancing. bq. I don't have a clear definition of validity across placement requests ... All valid concerns to me. I don't think we should spend a lot of time to solve a problem may be nonexistent. bq. If the workarounds (like submitting a new application or a new set of constraints) are easier to understand ... Regarding to the new set of constraint, do you mean user need to cancel the first set of constraints and submit new constraints with different allocation-id? Considering ResourceRequest problem I mentioned in YARN-6594, making SchedulingRequest which sent to scheduler to be immutable might be a good idea. (User can still cancel it, but cannot replace it with new requests). What's your thoughts here? > [API] Introduce Placement Constraint object > --- > > Key: YARN-6593 > URL: https://issues.apache.org/jira/browse/YARN-6593 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Fix For: 3.0.0-alpha3 > > Attachments: YARN-6593.001.patch, YARN-6593.002.patch, > YARN-6593.003.patch, YARN-6593.004.patch > > > This JIRA introduces an object for defining placement constraints. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[
https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095217#comment-16095217
]
Chris Douglas commented on YARN-6593:
-
bq. I found it is very important otherwise we cannot support complex placement
request which need to be updated.
Do you have specific use cases in mind? If there's a restricted form we could
use to support them (e.g., adjusting cardinality, as in your example), that
would be easier for us to support and for users to reason about. Since we don't
have applications that use placement constraints yet, it may be difficult for
us to predict where they need to change during execution (if at all).
bq. Regarding to semantics, I prefer to apply to all containers placed
subsequently, this is also the closest behavior of existing YARN. We just need
to verify updated placement request is still valid, probably we don't need to
restricted to some parameters.
I don't have a clear definition of validity across placement requests,
particularly preserving it across a sequence of updates to the constraints. We
could support relaxations of existing constraints, probably. Still, updates
also require the LRA scheduler to maintain lineage for all its internal
structures. A likely implementation will convert users' expressions to some
normal form, combine those with admin constraints, forecast future allocations,
inject requests into the scheduler, etc. Even if we could offer well-defined
semantics for updates, the implementation and maintenance cost could outweigh
the marginal benefit to users. If the workarounds (like submitting a new
application or a new set of constraints) are easier to understand, that's
probably what users will prefer, anyway.
Placement constraint updates also compound the {{ResourceRequest}} problem you
cited in YARN-6594. Which epoch of the placement constraints applied to a
container returned by the RM, and for which RR? If a user's application isn't
getting containers, how is that debugged? If someone wants to reason about a
group of constraints for a production cluster while applications change clauses
programmatically at runtime, then that analysis goes from difficult to
intractable.
You guys are implementing it, but I'd push this to future work.
> [API] Introduce Placement Constraint object
> ---
>
> Key: YARN-6593
> URL: https://issues.apache.org/jira/browse/YARN-6593
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Fix For: 3.0.0-alpha3
>
> Attachments: YARN-6593.001.patch, YARN-6593.002.patch,
> YARN-6593.003.patch, YARN-6593.004.patch
>
>
> This JIRA introduces an object for defining placement constraints.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[ https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095049#comment-16095049 ] Wangda Tan commented on YARN-6593: -- [~chris.douglas], bq. Do we want to support users adding new transforms? If not, some of the implementation details could be package-private. I think we should not support users adding new transforms. bq. Wangda Tan: what are the semantics of updated constraints? ... Yeah we missed it in our design doc, however I found it is very important otherwise we cannot support complex placement request which need to be updated. Regarding to semantics, I prefer to apply to all containers placed subsequently, this is also the closest behavior of existing YARN. We just need to verify updated placement request is still valid, probably we don't need to restricted to some parameters. What's your thoughts here? > [API] Introduce Placement Constraint object > --- > > Key: YARN-6593 > URL: https://issues.apache.org/jira/browse/YARN-6593 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Fix For: 3.0.0-alpha3 > > Attachments: YARN-6593.001.patch, YARN-6593.002.patch, > YARN-6593.003.patch, YARN-6593.004.patch > > > This JIRA introduces an object for defining placement constraints. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[
https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16094052#comment-16094052
]
Chris Douglas commented on YARN-6593:
-
I like the {{T accept(Visitor visitor)}} pattern and composing expressions
with {{PlacementConstraints}}; these are well polished. I agree with
[~leftnoteasy] on {{PlacementConstraints}} being the primary {{\@Public}} API.
Do we want to support users adding new transforms? If not, some of the
implementation details could be package-private.
[~leftnoteasy]: what are the semantics of updated constraints? Do they apply to
all containers placed subsequently, or could it cause a reconfiguration of
allocated containers? Or are updates restricted to (some?) parameters of the
expression? This isn't covered in the design doc on YARN-6592.
Minor:
* {{convert}} methods in {{PlacementConstraintFromProtoConverter}} should fail
if composite constraints have no children? Or would these invariants be checked
by a validator after construction?
* {{PlacementConstraints#timedMillisConstraint}} could accept a
[TimeUnit|https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/TimeUnit.html]
and convert to ms
* In this expression:
{noformat}
+ if (constraint.getOp() == TargetOperator.IN) {
+newConstraint = new SingleConstraint(constraint.getScope(), 1,
+Integer.MAX_VALUE, constraint.getTargetExpressions());
+ } else {
{noformat}
Might operator types be extended in the future, where this is not correct?
* All the constraints derive from the inner, {{AbstractConstraint}} type. This
avoids having {{PlacementConstraint}} accept a Visitor?
* A unit test demonstrating the PB serialization/deserialization would
demonstrate the converter classes.
> [API] Introduce Placement Constraint object
> ---
>
> Key: YARN-6593
> URL: https://issues.apache.org/jira/browse/YARN-6593
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Fix For: 3.0.0-alpha3
>
> Attachments: YARN-6593.001.patch, YARN-6593.002.patch,
> YARN-6593.003.patch, YARN-6593.004.patch
>
>
> This JIRA introduces an object for defining placement constraints.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[
https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16081175#comment-16081175
]
Wangda Tan commented on YARN-6593:
--
Thanks [~kkaranasos],
I just took a closer look at the patch, some questions/comments (I
consolidated all my above questions/comments to this one, so you only need to
look at this comment and after).
1) For classes/methods are marked to be {{@public}}, it should be user-facing
APIs or wire-protocol-format. I'm not sure if we should mark following classes
to {{@private}}.
- Visitable/Visitor/PlacementConstraintTransformations
- PlacementConstraintToProtoConverter (Methods to transform a Constraint class
to protocol should not be {{@public}}, but behavior of the process should be
compatible).
And do you think we should explicitly add comment to following class to say
they are marked to {{@public}} only because of wire-format compatibility?
2) Is there any reason to put PlacementConstraints in {{hadoop-yarn-common}}
project instead of {{hadoop-yarn-api}}?
3) I'm not sure when you plan to use
{{SingleConstraintTransformer}}/{{SpecializedConstraintTransformer}}. If you
agree that they should not be user-facing, do you think we should move them to
a separate JIRA while doing implementation?
4) Can we revert changes to TestContainerLaunch?
5) In general: more javadocs need to be added for user-facing APIs, we can do
this once we have a general agreement on APIs.
> [API] Introduce Placement Constraint object
> ---
>
> Key: YARN-6593
> URL: https://issues.apache.org/jira/browse/YARN-6593
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Fix For: 3.0.0-alpha3
>
> Attachments: YARN-6593.001.patch, YARN-6593.002.patch,
> YARN-6593.003.patch, YARN-6593.004.patch
>
>
> This JIRA introduces an object for defining placement constraints.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[ https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1601#comment-1601 ] Konstantinos Karanasos commented on YARN-6593: -- Hi [~wangda], I will be traveling starting Monday and will get back to the code on July 17th. So there will be time for you to look at the patch -- just make sure you send me your comments by July 17th. I would also like to get feedback from [~chris.douglas] on the patch. > [API] Introduce Placement Constraint object > --- > > Key: YARN-6593 > URL: https://issues.apache.org/jira/browse/YARN-6593 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Fix For: 3.0.0-alpha3 > > Attachments: YARN-6593.001.patch, YARN-6593.002.patch, > YARN-6593.003.patch, YARN-6593.004.patch > > > This JIRA introduces an object for defining placement constraints. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[
https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065840#comment-16065840
]
Wangda Tan commented on YARN-6593:
--
Thanks [~kkaranasos]/[~chris.douglas]/[~subru] for putting this together!
I haven't checked many details in the patch, the user facing API
PlacementConstraints looks very clear.
So far, I'm not sure Is it possible to reduce {{@public}} APIs? For example,
Visitor APIs should not be user-faceable, correct?
I will be traveling till next Tue, could you wait me to look at details of the
patch next week?
> [API] Introduce Placement Constraint object
> ---
>
> Key: YARN-6593
> URL: https://issues.apache.org/jira/browse/YARN-6593
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Fix For: 3.0.0-alpha3
>
> Attachments: YARN-6593.001.patch, YARN-6593.002.patch,
> YARN-6593.003.patch, YARN-6593.004.patch
>
>
> This JIRA introduces an object for defining placement constraints.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[
https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063960#comment-16063960
]
Hadoop QA commented on YARN-6593:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
43s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
55s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in trunk has 5 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
31s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
42s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 51s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 56 new + 97 unchanged - 0 fixed = 153 total (was 97) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
13s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
23s{color} | {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api
generated 2 new + 123 unchanged - 0 fixed = 125 total (was 123) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
33s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
29s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m
1s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 70m 50s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common |
| | Unread field:PlacementConstraintFromProtoConverter.java:[line 60] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6593 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12874573/YARN-6593.004.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux 6dd2d348d55c 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[
https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063943#comment-16063943
]
Hadoop QA commented on YARN-6593:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
24s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
51s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in trunk has 5 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
46s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
31s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 52s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 60 new + 83 unchanged - 21 fixed = 143 total (was 104) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
26s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
28s{color} | {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api
generated 2 new + 123 unchanged - 0 fixed = 125 total (was 123) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
34s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
25s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
35s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m
14s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 75m 14s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common |
| | Unread field:PlacementConstraintFromProtoConverter.java:[line 60] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6593 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12874565/YARN-6593.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux f0c1f1f01815 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[ https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022072#comment-16022072 ] Wangda Tan commented on YARN-6593: -- bq. Let's avoid bold in the JIRAs without a reason guys I don't think so, next time I will put the reasons to fat / large / red font so you won't miss it :). bq. We have a different approach in mind, but that's OK. Let's try to see the other person's point of view. Yes agree, looping [~jianhe]/[~sunilg]/[~vinodkv] for suggestions. bq. So, Wangda Tan, you would prefer different subclasses for each type of constraint. Therefore, we will have one for the target, one for the cardinality, and one that combines both, right? Yes, that's correct. > [API] Introduce Placement Constraint object > --- > > Key: YARN-6593 > URL: https://issues.apache.org/jira/browse/YARN-6593 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-6593.001.patch, YARN-6593.002.patch > > > This JIRA introduces an object for defining placement constraints. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[ https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022030#comment-16022030 ] Konstantinos Karanasos commented on YARN-6593: -- Let's avoid bold in the JIRAs without a reason guys :) We have a different approach in mind, but that's OK. Let's try to see the other person's point of view, it will help the end users too :) The difference is minor; we have a POC that does not require differentiating constraint classes, so that will break the symmetry from the other side, but that's OK, if it will help us converge. I do find it a pity to add unnecessary subclasses, but I want to move forward. So, [~leftnoteasy], you would prefer different subclasses for each type of constraint. Therefore, we will have one for the target, one for the cardinality, and one that combines both, right? > [API] Introduce Placement Constraint object > --- > > Key: YARN-6593 > URL: https://issues.apache.org/jira/browse/YARN-6593 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-6593.001.patch, YARN-6593.002.patch > > > This JIRA introduces an object for defining placement constraints. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[ https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022006#comment-16022006 ] Wangda Tan commented on YARN-6593: -- bq. I don't see why we should not expose SimplePlacementConstraint externally as well. That's because it's not understandable by users. We should not expose it to as a part of user facing API. bq. In any case, we can have Builders/Validators that can take care of this internally right ? bq. only once we get convinced that they will lead to more efficient implementation Like what I emphasized many times, builder can only solve the setting issue, *we need a symmetric getter* as well. bq. Since this feature is still nascent and in the process of development, we should probably err on the side of flexibility rather than stricter syntax. All dependent code (YARN native services) work can go on in parallel. One we have a working implementation, and we are close to a release, we can "late-bind" to a more restrictive API. This is foundation of the feature, I don't think we should move ahead before get consensus. I'm fine with delaying string-based api and other fancier builders. > [API] Introduce Placement Constraint object > --- > > Key: YARN-6593 > URL: https://issues.apache.org/jira/browse/YARN-6593 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-6593.001.patch, YARN-6593.002.patch > > > This JIRA introduces an object for defining placement constraints. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[
https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021971#comment-16021971
]
Konstantinos Karanasos commented on YARN-6593:
--
bq. If the scheduler can handle a single class efficiently, we don't need a
separate representation in scheduler side. However as I stated, we don't know
this yet.
We should then create subclasses (that will not be consistent with the
protobufs) only once we get convinced that they will lead to more efficient
implementation. Why add subclasses without knowing this is true?
The users will see different constraint types through the builder, as in
{{addTargetConstraint}}, {{addCardinalityConstraint}}. If we see that adding
them as subclasses is important for implementation, we can sure go ahead and
add them later. The API in the builders will not need to change.
> [API] Introduce Placement Constraint object
> ---
>
> Key: YARN-6593
> URL: https://issues.apache.org/jira/browse/YARN-6593
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Attachments: YARN-6593.001.patch, YARN-6593.002.patch
>
>
> This JIRA introduces an object for defining placement constraints.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[ https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021954#comment-16021954 ] Arun Suresh commented on YARN-6593: --- Given that what ideally should be perceived as the API is the proto definitions, and since we agreed to merge both TargetConstraint and CardinalityConstraint as a single proto struct, I don't see why we should not expose SimplePlacementConstraint externally as well. In any case, we can have Builders/Validators that can take care of this internally right ? > [API] Introduce Placement Constraint object > --- > > Key: YARN-6593 > URL: https://issues.apache.org/jira/browse/YARN-6593 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-6593.001.patch, YARN-6593.002.patch > > > This JIRA introduces an object for defining placement constraints. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[ https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021919#comment-16021919 ] Wangda Tan commented on YARN-6593: -- [~arun.sur...@gmail.com], Regardless of implementation, I think we should clearly define it before moving forward. I'm OK with moving some language-sugar APIs like string-based or some fancier build patterns, etc. However the TargetConstrant/CardinalityConstraint are what we clearly defined on the design doc, we should follow that. This API is targeted for YARN app developers to use in the future, if we cannot get converged between several of us, how we can make it widely used by YARN community developers. > [API] Introduce Placement Constraint object > --- > > Key: YARN-6593 > URL: https://issues.apache.org/jira/browse/YARN-6593 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-6593.001.patch, YARN-6593.002.patch > > > This JIRA introduces an object for defining placement constraints. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[ https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021902#comment-16021902 ] Arun Suresh commented on YARN-6593: --- Guys, I feel this discussion is not tending to convergence.. To move this forward, I recommend for this JIRA, let us just stick to SimplePlacementConstraint class thereby keeping it as close to the .proto definitions. [~leftnoteasy], I see your argument about how specializing the SimplePlacementConsrtaint into sub class might lead to the implementation optimizations, but let us track that perhaps in a separate JIRA and tackle that maybe in conjunction with when we are working on the implementation ? > [API] Introduce Placement Constraint object > --- > > Key: YARN-6593 > URL: https://issues.apache.org/jira/browse/YARN-6593 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-6593.001.patch, YARN-6593.002.patch > > > This JIRA introduces an object for defining placement constraints. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[ https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021880#comment-16021880 ] Wangda Tan commented on YARN-6593: -- [~kkaranasos], bq. You see my point? I saw your point, however I don't think we should assume scheduler changes before start writing code. I can still see values to handle separate constraints (like optimize logics) instead of operate on a 3-fields one. bq. The end user API can still evolve, as long as it remains backwards compatible. I think my suggestion won't change how we handle backward compatible either. Since my suggested approach won't change structures defined in proto file. bq. If the scheduler does not need to do any if statement (as I explain above), what is the advantage of having separate classes for each constraint type (target, cardinality, and targetCardinality/admin)? Is there an implementation concern? If the scheduler can handle a single class efficiently, we don't need a separate representation in scheduler side. However as I stated, we don't know this yet. Also as I mentioned several times, user can access/understand constraint easier with separate classes. > [API] Introduce Placement Constraint object > --- > > Key: YARN-6593 > URL: https://issues.apache.org/jira/browse/YARN-6593 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-6593.001.patch, YARN-6593.002.patch > > > This JIRA introduces an object for defining placement constraints. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[ https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021836#comment-16021836 ] Konstantinos Karanasos commented on YARN-6593: -- [~leftnoteasy], what I am trying to say is that the scheduler does not need to do any if. It will handle internally one type of constraint that includes cardinality, scope, and target. If one of them happens to be a default value (self for target, 0 for min cardinality, inf for max cardinality), it does not change anything from the scheduler's perspective. After all the scheduler has to deal with the constraint that specifies all three fields (what we called operator constraints in the doc), so it will just duplicate code if we have three ifs. You see my point? The end user API can still evolve, as long as it remains backwards compatible. For instance, when we add the admin constraints, we can simply add an additional builder utility method, without requiring changes in any other part of the code. Similar when we do the string representation of the constraints. If the scheduler does not need to do any if statement (as I explain above), what is the advantage of having separate classes for each constraint type (target, cardinality, and targetCardinality/admin)? Is there an implementation concern? > [API] Introduce Placement Constraint object > --- > > Key: YARN-6593 > URL: https://issues.apache.org/jira/browse/YARN-6593 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-6593.001.patch, YARN-6593.002.patch > > > This JIRA introduces an object for defining placement constraints. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[
https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021765#comment-16021765
]
Wangda Tan commented on YARN-6593:
--
[~kkaranasos],
bq. That's exactly the advantage of having a single simple constraint. When it
is a target constraint, min cardinality is 0 and max cardinality is infinite.
When it is a cardinality constraint, target is yourself. This way the scheduler
has to deal with a single placement constraint. No need for checking
constraint-types at any point.
This is what I want to avoid, implicitly define internal fields makes harder
for scheduler to handle and user to understand.
Pseudo code of you were describing is:
{code}
SimplePlacementConstraint c = ...;
if (c.minCardinality == 0 && c.maxCardinality == inf) {
// this is a target constraint
TargetExp t = c.getTargetExpression ...
} else if (c.target == null /* or self */ ) {
int minCardinality = c.getMinCardinality();
int maxCardinality = c.getMaxCardinality();
} else {
// throw exception, not allow to set all 3 fields
}
// If we need get TargetConstraint / Cardinality in other places, we have to
copy above logics everywhere.
{code}
Instead, what we can do is
{code}
SimplePalcementConstraint c = ...;
if (c.getType() == TargetConstraint) {
TargetConstraint tc = (TargetConstraint) c;
} else if (c.getType() == CardinalityConstraint) {
CardinalityConstraint cc = (CardinalityConstraint) c;
}
// tc and cc can be saved, we don't have to do the cast every time ..
{code}
bq. We keep it simple for the applications to express constraints through the
utility methods, and we keep it simple for the scheduler to deal with a single
constraint type.
But it cannot let applications to access constraints added easily.
bq. Moreover, if we have multiple classes for simple constraints, when building
the objects from the protobufs in the scheduler side, we will have to cast to
each constraint class. And if we decide to change those classes in the future
(to expose a different way for users to write constraints), we will have to
change the way the scheduler deals with constraints, which should really be
avoided.
This is end user API, I don't think we should rapidly change this. YARN
applications (like native services) will consume this once it is available.
That is why I want to keep it stable and easy to use from the beginning.
> [API] Introduce Placement Constraint object
> ---
>
> Key: YARN-6593
> URL: https://issues.apache.org/jira/browse/YARN-6593
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Attachments: YARN-6593.001.patch, YARN-6593.002.patch
>
>
> This JIRA introduces an object for defining placement constraints.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[ https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021657#comment-16021657 ] Konstantinos Karanasos commented on YARN-6593: -- bq. otherwise we have to check constraint-type everywhere That's exactly the advantage of having a single simple constraint. When it is a target constraint, min cardinality is 0 and max cardinality is infinite. When it is a cardinality constraint, target is yourself. This way the scheduler has to deal with a single placement constraint. No need for checking constraint-types at any point. We keep it simple for the applications to express constraints through the utility methods, and we keep it simple for the scheduler to deal with a single constraint type. Moreover, if we have multiple classes for simple constraints, when building the objects from the protobufs in the scheduler side, we will have to cast to each constraint class. And if we decide to change those classes in the future (to expose a different way for users to write constraints), we will have to change the way the scheduler deals with constraints, which should really be avoided. > [API] Introduce Placement Constraint object > --- > > Key: YARN-6593 > URL: https://issues.apache.org/jira/browse/YARN-6593 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-6593.001.patch, YARN-6593.002.patch > > > This JIRA introduces an object for defining placement constraints. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[
https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021590#comment-16021590
]
Wangda Tan commented on YARN-6593:
--
[~kkaranasos],
bq. What I realized is that we are trying to define at the same time the
internal representation that will be used by the scheduler and the one that
will be user-friendly.
Exactly.
bq. I think we should split the two. I suggest to keep the java classes
implementing the PBImpls to be in sync with them, and then add a utility class
that allows users to create constraints in a more intuitive way.
I still want to add my thoughts to this part. I agree that we should try to
make Java PB implementation sync with .proto. But instead of a utility class to
create, I think we should make user-facing API as simple as possible. I think
it is also important to let user can access added constraints.
I'm open to the suggestions to make user can specify a string-based constraints
or fluent style APIs. But my comment is, if an API is marked to {{@Public}}, it
has to be simple and easy to be understood by YARN users. The
SimplePlacementConstraint with 3 fields is not simple to be understand.
bq. BTW, the reason I have not created separate classes for target and
cardinality constraints is that I we also have the more general constraint (the
one we mention as "cluster operator constraint" in the document, such as "don't
allow more than 5 ZooKeeper containers per rack") that includes all three.
I'm not quite agree with this part, in our implementation to support
constraints can also benefit from split constraints, otherwise we have to check
constraint-type everywhere. The syntax of cluster operator constraint is quite
different from TargetConstraint/CardinalityConstraint (cardinality to myself
v.s. cardinality to others). I'm fine with adding a ClusterOperatorConstraint
based on SimplePlacementConstraint if needed.
> [API] Introduce Placement Constraint object
> ---
>
> Key: YARN-6593
> URL: https://issues.apache.org/jira/browse/YARN-6593
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Attachments: YARN-6593.001.patch, YARN-6593.002.patch
>
>
> This JIRA introduces an object for defining placement constraints.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[ https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021546#comment-16021546 ] Arun Suresh commented on YARN-6593: --- +1 on separating the Builder functionality into separate util classes. Apart from the reasons provided by [~kkaranasos], I also feel simple Builders (that promote the [fluent|https://www.martinfowler.com/bliki/FluentInterface.html] style of coding) might not really be super useful for nested structures. > [API] Introduce Placement Constraint object > --- > > Key: YARN-6593 > URL: https://issues.apache.org/jira/browse/YARN-6593 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-6593.001.patch, YARN-6593.002.patch > > > This JIRA introduces an object for defining placement constraints. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[
https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021533#comment-16021533
]
Konstantinos Karanasos commented on YARN-6593:
--
Thanks, [~leftnoteasy].
What I realized is that we are trying to define at the same time the internal
representation that will be used by the scheduler and the one that will be
user-friendly.
I think we should split the two. I suggest to keep the java classes
implementing the PBImpls to be in sync with them, and then add a utility class
that allows users to create constraints in a more intuitive way. The utility
class can expose all the constraint creation methods and hide all the protobuf
details (e.g., the fact that we have a PlacementConstraint class internally).
This will allow us to evolve separately the way users specify constraints
without needing to change the PBImpl classes or their subclasses etc.
Moreover, we can create a string utility method that can parse a string
representation of the constraints and create PBImpl objects, which I think will
be really useful too.
Let me know what you guys think.
BTW, the reason I have not created separate classes for target and cardinality
constraints is that I we also have the more general constraint (the one we
mention as "cluster operator constraint" in the document, such as "don't allow
more than 5 ZooKeeper containers per rack") that includes all three. So I don't
see the use of adding three different classes for this. Especially if we have
the utility class I mentioned above.
PS: I forgot to reply to [~pg1...@imperial.ac.uk], since we chatted offline,
but I am adding it here for completeness. Unfortunately protobuf 2.5, which is
still the required version in Hadoop, does not support the {{extends}}
construct to define sub/superclasses in the protobufs.
> [API] Introduce Placement Constraint object
> ---
>
> Key: YARN-6593
> URL: https://issues.apache.org/jira/browse/YARN-6593
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Attachments: YARN-6593.001.patch, YARN-6593.002.patch
>
>
> This JIRA introduces an object for defining placement constraints.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[
https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020389#comment-16020389
]
Wangda Tan commented on YARN-6593:
--
Thanks [~kkaranasos],
I checked the latest patch, it doesn't create hierarchies in Java side, however
I think it is a solution which makes consistency between Java/PB and clear in
Java side. So in general it is a good approach to me, only one major suggestion
to SimplePlacementConstraint, now mixing all three fields
(scope/target/cardinality) in getters could still confuse developers.
I suggest to make SimplePlacementConstraint to be parent class of
TargetPlacementConstraint/CardinalityPlacementConstraint, which has:
- Move builders to protected.
- Move getters/setters to protected.
- Remove targetConstraint/maxCardinalityConstraint, etc. static constructors to
children class.
Children classes such as TargetPlacementConstraint, it extends
SimplePlacementConstraint, and has targetConstraint static constructor. So
basically, SimplePlacementConstraint is Java side API in parallel with PB
definition, and TargetPlacementConstraint/ConstraintPlacementConstraint are
actual user APIs.
More detailed comments:
1) PlacementConstraint:
- newInstance: Could we add a check to make sure only one of Simple/Compound is
non-null?
- setSimpleConstraint/setCompoundConstraint: could be {{@Private}}
- It could be better if we add a getPlacementConstraint type field, what you
think? We can move {{SimplePlacementConstraint#ConstraintType}} to a separate
class which include enums: \{ COMPOUND_CONSTRAINT / TARGET_CONSTRAINT /
CARDINALITY_CONSTRAINT \}
I have more detailed comments, but I want to reach a consensus for the major
suggestions before posting them. One more suggestion is, since this API will be
consumed by developers and scheduler logics, it will be better if you could
draft an example Java source code which includes few examples. With that it
will be easier to get if it is straightforward enough to use/consume the API.
> [API] Introduce Placement Constraint object
> ---
>
> Key: YARN-6593
> URL: https://issues.apache.org/jira/browse/YARN-6593
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Attachments: YARN-6593.001.patch, YARN-6593.002.patch
>
>
> This JIRA introduces an object for defining placement constraints.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[
https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16016212#comment-16016212
]
Panagiotis Garefalakis commented on YARN-6593:
--
[~kkaranasos] thanks for the patch!
The discussion totally makes sense to me. Some comments:
* Totally agree on using a more object oriented way of representing both
PlacementConstraint -> CompoundPlacementConstraint/SimplePlacementConstraint
and SimplePlacementConstraint -> TargetConstraint/CardinalityConstraint. I
think the main value for doing so is usability.
* Protobuf extentions might also be something we could use. For example:
{code:java}
message TargetConstraintProto {
extend SimplePlacementConstraintProto
{
required TargetConstraintProto costraint = 10; // Unique extension number
}
}
message CardinalityConstraintProto {
extend SimplePlacementConstraintProto
{
required CardinalityConstraintProto costraint = 11; // Unique extension
number
}
}
{code}
* We will definitely need a validator implementation - also as a way to ensure
users type constraints that do make sense
* I am also wondering if IN_ANY should be a separate **TargetOperator** - in
a case like C5 design-doc example we would avoid using any TargetValues
Panagiotis
> [API] Introduce Placement Constraint object
> ---
>
> Key: YARN-6593
> URL: https://issues.apache.org/jira/browse/YARN-6593
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Attachments: YARN-6593.001.patch
>
>
> This JIRA introduces an object for defining placement constraints.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[
https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16015209#comment-16015209
]
Wangda Tan commented on YARN-6593:
--
Thanks [~kkaranasos],
bq. Self-reference works fine even with the current protobuf version. I think I
saw a single use of it in our yarn_protos (I think it was in the QueueInfo or
something related).
Sounds good, thanks for confirmation.
bq. (1) the proto becomes too complicated (and I am afraid we will be the only
ones understanding it
This statement is true. :)
bq. (2) the client will still have access to very unrelated getters and setters.
This part is avoidable, we can avoid add un-related getters and setters to Java
API class, and access PB methods in -PBImpl class.
For example:
{code}
class TargetConstraint {
// Only expose what required
get/setTargetExpression();
get/setScope();
}
class TargetConstraintPBImpl implements TargetConstraint {
void setTargetExpression(...) {
// Access fields in Proto in -PBImpl only
SimplePlacementConstraintProtoBuilder.setTargetExpression(..);
}
}
{code}
Please let me know what you think.
> [API] Introduce Placement Constraint object
> ---
>
> Key: YARN-6593
> URL: https://issues.apache.org/jira/browse/YARN-6593
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Attachments: YARN-6593.001.patch
>
>
> This JIRA introduces an object for defining placement constraints.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[ https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16015028#comment-16015028 ] Konstantinos Karanasos commented on YARN-6593: -- Thanks for the comments guys. So, we all agree on the validator, will open a JIRA to track this. +1 on having a string representation and a library to convert it to constraints. I think it will become very useful soon, but agreed it is not urgent. Self-reference works fine even with the current protobuf version. I think I saw a single use of it in our yarn_protos (I think it was in the QueueInfo or something related). That said, both approaches make sense to me. I guess the approach in the current JIRA is a little bit more cumbersome to right (we will have one extra call to create a PlacementConstraint). Is that its disadvantage or there is something else too? On the other hand, what I don't like if we coalesce the two, is that (1) the proto becomes too complicated (and I am afraid we will be the only ones understanding it :)), and (2) the client will still have access to very unrelated getters and setters. For example, the user could be creating a CompoundConstraint and have access to a getTargetExpression() function. > [API] Introduce Placement Constraint object > --- > > Key: YARN-6593 > URL: https://issues.apache.org/jira/browse/YARN-6593 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-6593.001.patch > > > This JIRA introduces an object for defining placement constraints. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[ https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16015013#comment-16015013 ] Wangda Tan commented on YARN-6593: -- [~asuresh], All make sense to me, bq. But I think the problem with that is I think we would require self-referential structs. Good point, let's check if it works. bq. I would also like to have library that might take a string representation of the Constraints +1, I think we can use an existing format, such as JSON / YAML. This can be done in a separate patch (with lower priority I think). > [API] Introduce Placement Constraint object > --- > > Key: YARN-6593 > URL: https://issues.apache.org/jira/browse/YARN-6593 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-6593.001.patch > > > This JIRA introduces an object for defining placement constraints. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[
https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16015008#comment-16015008
]
Arun Suresh commented on YARN-6593:
---
Thanks for the work on this [~kkaranasos]
bq. For example, PlacementConstraints: Inside the PlacementConstraintProto, we
can add CompoundType / Children / DelayCriteria / Everything in the
SimplePlacementConstraintProto. And a field to indicate if it is a
CompoundConstraint / (Simple)TargetConstraint / (Simple)CardinalityConstraint.
>From a Client point of view, I tend to agree with [~leftnoteasy]. But I think
>the problem with that is I think we would require self-referential structs.
>essentially, the {{PlacementConstraintProto}} of type _Compound_ should
>container a repeated {{PlacementConstraintProto}} field. Am not sure this is
>supported in protobufs currently.
W.r.t Validations, as discussed offline with [~kkaranasos],
This is to enforce correctness of the expression which is currently not
enforcable via the protobuf language. For eg:
* We need to enforce that a compound expression should have aleast 2 child
placement expressions.
W.r.t use of the Builder.
As much as I like the Builder, it might be difficult to use it to build nested
expressions.
+1 for a Validation library, which can be reused both in the AMRMClient and the
ClientRMService.
I would also like to have library that might take a string representation of
the Constraints - as what was defined in the doc, and convert it to a proto /
API object. This can be tackled later I guess.
> [API] Introduce Placement Constraint object
> ---
>
> Key: YARN-6593
> URL: https://issues.apache.org/jira/browse/YARN-6593
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Attachments: YARN-6593.001.patch
>
>
> This JIRA introduces an object for defining placement constraints.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[ https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16014940#comment-16014940 ] Wangda Tan commented on YARN-6593: -- Thanks [~kkaranasos] for your explanations. Yeah I think you're correct, for the current PB, we cannot do extend easily on the PB side. However we can still improve Java API: For example, PlacementConstraints: Inside the PlacementConstraintProto, we can add CompoundType / Children / DelayCriteria / Everything in the SimplePlacementConstraintProto. And a field to indicate if it is a CompoundConstraint / (Simple)TargetConstraint / (Simple)CardinalityConstraint. With this, we can at least make Java API can be very clear, same as what we defined in design doc, and Java side takes responsibility to translate user-facing API and PB. Please share your thoughts, +[~asuresh]. bq. My question was more about how do we validate constraints +1 to have a validator in common layer so client side can do validation as well. I think validation has two phases: 1) Client side static validation 2) Server side dynamic validation, #2 will look at cluster state and check things like if constraints added to cluster or not, etc. > [API] Introduce Placement Constraint object > --- > > Key: YARN-6593 > URL: https://issues.apache.org/jira/browse/YARN-6593 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-6593.001.patch > > > This JIRA introduces an object for defining placement constraints. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[ https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16014835#comment-16014835 ] Konstantinos Karanasos commented on YARN-6593: -- Thanks for the feedback, [~leftnoteasy]. Agreed with points (2), (3), (5), and (6) -- will fix them and update the patch. Regarding the rest: bq. PlacementConstraint should be parent class of CompoundPlacementConstraint/SimplePlacementConstraint. Now it added one additional hierarchy to each node in the tree. Can you explain a bit more what you mean here? My goal was to allow arbitrary nesting of compound placement constraints. Given that the protobuf version that we use does not allow subclassing, don't think there is a cleaner way of achieving this. bq. Suggest to follow what we defined in doc, add two separate classes TargetConstraint​/CardinalityConstraint​, both extends PlacementConstraint. Limit to use 2 out of 3 fields is hard for app developer to understand/use. I added the ConstraintType to show the different types of simple placement constraints, namely target and cardinality (again, given the lack of subclassing in protobuf). Moreover, I wanted to allow later to add constraints that include all three fields, like we mention in the document for cluster admin constraints. I think we can restrict the use of the right fields at each time through the newInstance methods, like I do currently in the patch. But to make it more clear, how about we add a Builder that allows the creation only of targetConstraint() and cardinalityConstraint()? This way we use a SimplePlacementConstraint, but allow creating specific types of simple constraints. bq. I suggest treat allocationTags as value, and key of allocationTags is always empty. Yep, agreed with that. My question was more about how do we validate constraints. I think we should create a validator class, so that we can parse the constraints and decide if they are of the correct format. One example is to not allow a targetKey when you specify a constraint with allocation tags. Another could be to allow ORDERED_OR expressions to be only on top of the hierarchy, etc. Was discussing that yesterday with [~asuresh] too. Of course this is not urgent, but would be good to have. Thoughts? > [API] Introduce Placement Constraint object > --- > > Key: YARN-6593 > URL: https://issues.apache.org/jira/browse/YARN-6593 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-6593.001.patch > > > This JIRA introduces an object for defining placement constraints. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
[
https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16014767#comment-16014767
]
Wangda Tan commented on YARN-6593:
--
[~kkaranasos], thanks for updating the patch, some suggestions:
1) PlacementConstraint should be parent class of
CompoundPlacementConstraint/SimplePlacementConstraint. Now it added one
additional hierarchy to each node in the tree.
2) Suggest to move delay DelayUnit/delayValue from CompoundPlacementConstraint
to a separate class such as DelayCriterion.
3) Suggest to introduce Builder to CompoundPlacementConstraint, with that, YARN
app can call:
{code}
CompoundPlacementConstraint.newBuilder().and/or(,
, ...)
CompoundPlacementConstraint.newBuilder().and/or(,
, ...)
CompoundPlacementConstraint.newBuilder().and/or(List,
, , ...)
{code}
4) SimplePlacementConstraint:
Suggest to follow what we defined in doc, add two separate classes
TargetConstraint​/CardinalityConstraint​, both extends PlacementConstraint.
Limit to use 2 out of 3 fields is hard for app developer to understand/use.
5) Similarily, PlacementConstraintTarget. Suggest to introduce a Builder, which
we can support following creations modes:
{code}
- toNodeAttributes(op, nodeAttributeKey, List [API] Introduce Placement Constraint object
> ---
>
> Key: YARN-6593
> URL: https://issues.apache.org/jira/browse/YARN-6593
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Attachments: YARN-6593.001.patch
>
>
> This JIRA introduces an object for defining placement constraints.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
