[jira] [Comment Edited] (YARN-6593) [API] Introduce Placement Constraint object
[ https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16110248#comment-16110248 ] Jian He edited comment on YARN-6593 at 8/2/17 4:05 AM: --- The use-case is to be able to select containers based on key and values, say, I want to find my containers with version=v1, and env = test, and foo !=bar and name=web1 or name = web2. Yes, currently it's a set of values not a single value, I misspoke. We can still make it work by searching the entire string as Arun said. But that's not explicit. The AM, client, or even UI then needs to parse the string to extract the keys and values. I was thinking this change will affect this jira's implementation also, hence continue commenting here. If you think this anyways can be revisited. We can commit this first and work on it in YARN-6594. It doesn't matter to me. was (Author: jianhe): The use-case is to be able to select containers based on key and values, say, I want to find my containers with version=v1, and env= test, and name=web1 or name = web2. Yes, currently it's a set of values not a single value, I misspoke. We can still make it work by searching the entire string as Arun said. But that's not explicit. The AM, client, or even UI then needs to parse the string to extract the keys and values. I was thinking this change will affect this jira's implementation also, hence continue commenting here. If you think this anyways can be revisited. We can commit this first and work on it in YARN-6594. It doesn't matter to me. > [API] Introduce Placement Constraint object > --- > > Key: YARN-6593 > URL: https://issues.apache.org/jira/browse/YARN-6593 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-6593.001.patch, YARN-6593.002.patch, > YARN-6593.003.patch, YARN-6593.004.patch, YARN-6593.005.patch, > YARN-6593.006.patch, YARN-6593.007.patch, YARN-6593.008.patch > > > Just removed Fixed version and moved it to target version as we set fix > version only after patch is committed. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6593) [API] Introduce Placement Constraint object
[ https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16109921#comment-16109921 ] Arun Suresh edited comment on YARN-6593 at 8/1/17 10:38 PM: bq. Actually, my comments should apply to YARN-6594 rather this jira. [~jianhe], I am guessing you are referring to the *AllocationTags* .. then yes, we should continue that discussion in YARN-6594, since this pertains just to the *PlacementConstraint* object. (Can you please cross-post it there as well ?) My opinion though is that the allocation tags should remain a list of string keys. I understand having it as a map might offload some work from the AM, but do you see it helping the RM/scheduler in making a scheduling decision ? If it is just to select a bunch of containers, one can still perform a search like: select containers where allocationtags contains "env=test" etc. Forcing the tags to be a map might even make the querying more complicated. was (Author: asuresh): bq. Actually, my comments should apply to YARN-6594 rather this jira. [~jianhe], I am guessing you are referring to the *AllocationTags* .. then yes, we should continue that discussion in YARN-6594, since this pertains just to the *PlacementConstraint* object. (Can you please cross-post it there as well ?) My opinion though is that the allocation tags should remain a list of string keys. I understand this might offload some work from the AM, but do you see it helping the RM/scheduler in making a scheduling decision ? If it is just to select a bunch of containers, one can still perform a search like: select containers where allocationtags contains "env=test" etc. Forcing the tags to be a map might even make the querying more complicated. > [API] Introduce Placement Constraint object > --- > > Key: YARN-6593 > URL: https://issues.apache.org/jira/browse/YARN-6593 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-6593.001.patch, YARN-6593.002.patch, > YARN-6593.003.patch, YARN-6593.004.patch, YARN-6593.005.patch, > YARN-6593.006.patch, YARN-6593.007.patch, YARN-6593.008.patch > > > Just removed Fixed version and moved it to target version as we set fix > version only after patch is committed. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6593) [API] Introduce Placement Constraint object
[ https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16109872#comment-16109872 ] Jian He edited comment on YARN-6593 at 8/1/17 10:05 PM: Actually, my comments should apply to YARN-6594 rather this jira. For example, I want to associate some meta info like "env=test" and "version=v2" info to the container and the AM can define the keys by itself. AM can then do select based on these key and values for the containers returned. To me, the tag is not just used for scheduling decisions. It's also for associating metainfo to the container, otherwise, the AM has to maintain and persist the mapping by itself. If YARN natively supports this, it'll be a lot easier for the AM. bq. I have some first examples in PlacementConstraints. Do you think we should add them in a different class? I was thinking having a class which lists all sorts scenarios and the implementation, user can pretty much copy the code as needed. Reading javadoc is one option but still not straightfoward. Or if the javadoc is comprehensive to cover all scenarios in one place, rather than letting user look for different places, that should also be fine. was (Author: jianhe): Actually, my comments should apply to YARN-6594 rather this jira. For example, I want to associate some meta info like "env=test" and "version=v2" info to the container and the AM can define the keys by itself. AM can then do select based on these key and values for the containers returned. To me, the tag is not just used for scheduling decisions. It's also for associating metainfo to the container, otherwise, the AM has to maintain and persist the mapping by itself. If YARN natively supports this, it'll be a lot easier for the AM. bq. I have some first examples in PlacementConstraints. Do you think we should add them in a different class? I was thinking having a class which lists all sorts scenarios and the API definitions, user can pretty much copy the semantics. Reading javadoc is one option but still not straightfoward. > [API] Introduce Placement Constraint object > --- > > Key: YARN-6593 > URL: https://issues.apache.org/jira/browse/YARN-6593 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-6593.001.patch, YARN-6593.002.patch, > YARN-6593.003.patch, YARN-6593.004.patch, YARN-6593.005.patch, > YARN-6593.006.patch, YARN-6593.007.patch, YARN-6593.008.patch > > > Just removed Fixed version and moved it to target version as we set fix > version only after patch is committed. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6593) [API] Introduce Placement Constraint object
[ https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16107632#comment-16107632 ] Jian He edited comment on YARN-6593 at 7/31/17 5:25 PM: should we have an example class to demonstrate how to use the APIs for different scenarios as mentioned in the document? I think that's useful for users. was (Author: jianhe): should we have an example class to demonstrate how to use the APIs? I think that's useful for users. > [API] Introduce Placement Constraint object > --- > > Key: YARN-6593 > URL: https://issues.apache.org/jira/browse/YARN-6593 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-6593.001.patch, YARN-6593.002.patch, > YARN-6593.003.patch, YARN-6593.004.patch, YARN-6593.005.patch, > YARN-6593.006.patch, YARN-6593.007.patch, YARN-6593.008.patch > > > Just removed Fixed version and moved it to target version as we set fix > version only after patch is committed. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6593) [API] Introduce Placement Constraint object
[
https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16081175#comment-16081175
]
Wangda Tan edited comment on YARN-6593 at 7/10/17 11:48 PM:
Thanks [~kkaranasos],
I just took a closer look at the patch, some questions/comments (I
consolidated all my above questions/comments to this one, so you only need to
look at this comment and after).
1) For classes/methods are marked to be {{@public}}, it should be user-facing
APIs or wire-protocol-format. I'm not sure if we should mark following classes
to {{@private}}.
- Visitable/Visitor/PlacementConstraintTransformations
- PlacementConstraintToProtoConverter (Methods to transform a Constraint class
to protocol should not be {{@public}}, but behavior of the process should be
compatible).
And do you think we should explicitly add comment to following class to say
they are marked to {{@public}} only because of wire-format compatibility?
2) Is there any reason to put PlacementConstraints in {{hadoop-yarn-common}}
project instead of {{hadoop-yarn-api}}?
3) I'm not sure when you plan to use
{{SingleConstraintTransformer}}/{{SpecializedConstraintTransformer}}. If you
agree that they should not be user-facing, do you think we should move them to
a separate JIRA while doing implementation?
4) Can we revert changes to TestContainerLaunch?
5) In general: more javadocs need to be added for user-facing APIs, we can do
this once we have a general agreement on APIs.
= EDITs (July 10), added few more points: =
6) As I commented on YARN-6594, one of the biggest issue of old ResourceRequest
is it cannot properly handle resource update.
For example, at time T1 app request "host1" with #container=2, and at time T2
app decide to request "host1" for 2 more container, so it request "host1" with
#containers=4, however, at the same time, scheduler allocated 2 containers
already, so scheduler receive the request and update pending #containers=4.
When this happens, app can receive much more allocated containers than
originally requested.
There're two separate problems:
6.1) Partially update a placement request within the tree. To alleviate problem
above, we can partially update single or multiple placement constraints, by
doing this I think we can allocate unique-constraint-id for each contraint
internally and every time send diff to RM, end-user should not even notice that.
For example:
{code}
A placement constraint specified by user
and {
OR {
AND {
target-constraint { host IN "host1" },
cardinality-constraint { max_cardinality = 2}
},
AND {
target-constraint { host IN "host2" },
cardinality-constraint { max_cardinality = 4 }
}
}
{code}
unique-constraint-id (UCID) will be added in client and will be sent to server
for each constraint.
{code}
A placement constraint specified by user
and (ucid=1) {
OR (ucid=2) {
AND (ucid=3) {
target-constraint (ucid=4) { host IN "host1" },
cardinality-constraint (ucid=5) { max_cardinality = 2}
},
AND (ucid=6) {
target-constraint (ucid=7) { host IN "host2" },
cardinality-constraint (ucid=8) { max_cardinality = 4 }
}
}
{code}
If application updated max_cardinality to 3 for host1, only diff will be sent
to scheduler:
{code}
diff: [
{
cardinality-constraint (ucid=5) { max_cardinality = 3}
}
]
{code}
6.2) Increment/decrement #pending asks.
Only partially update constraint is not enough because it cannot avoid the
problem mentioned in 6). In addition to proposed in 6.1, do you think in the
protocol we can say: +2 for maximum-cardinality for ucid=5? When scheduler
received such requests, it will look at allocated-but-not-yet-acquired
containers to avoid allocate more containers than requested.
Requesting: [~asuresh]/[~subru]/[~chris.douglas]/[~sunilg] for more suggestions.
was (Author: leftnoteasy):
Thanks [~kkaranasos],
I just took a closer look at the patch, some questions/comments (I
consolidated all my above questions/comments to this one, so you only need to
look at this comment and after).
1) For classes/methods are marked to be {{@public}}, it should be user-facing
APIs or wire-protocol-format. I'm not sure if we should mark following classes
to {{@private}}.
- Visitable/Visitor/PlacementConstraintTransformations
- PlacementConstraintToProtoConverter (Methods to transform a Constraint class
to protocol should not be {{@public}}, but behavior of the process should be
compatible).
And do you think we should explicitly add comment to following class to say
they are marked to {{@public}} only because of wire-format compatibility?
2) Is there any reason to put PlacementConstraints in {{hadoop-yarn-common}}
project instead of {{hadoop-yarn-api}}?
3) I'm not sure when you plan to use
{{SingleConstraintTransformer}}/{{SpecializedConstraintTransformer}}. If you
agree
[jira] [Comment Edited] (YARN-6593) [API] Introduce Placement Constraint object
[ https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021954#comment-16021954 ] Arun Suresh edited comment on YARN-6593 at 5/23/17 10:09 PM: - Given that what ideally should be perceived as the API is the proto definitions, and since we agreed to merge both TargetConstraint and CardinalityConstraint as a single proto struct, I don't see why we should not expose SimplePlacementConstraint externally as well. In any case, we can have Builders/Validators that can take care of this internally right ? Since this feature is still nascent and in the process of development, we should probably err on the side of flexibility rather than stricter syntax. All dependent code (YARN native services) work can go on in parallel. One we have a working implementation, and we are close to a release, we can "late-bind" to a more restrictive API. was (Author: asuresh): Given that what ideally should be perceived as the API is the proto definitions, and since we agreed to merge both TargetConstraint and CardinalityConstraint as a single proto struct, I don't see why we should not expose SimplePlacementConstraint externally as well. In any case, we can have Builders/Validators that can take care of this internally right ? > [API] Introduce Placement Constraint object > --- > > Key: YARN-6593 > URL: https://issues.apache.org/jira/browse/YARN-6593 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-6593.001.patch, YARN-6593.002.patch > > > This JIRA introduces an object for defining placement constraints. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6593) [API] Introduce Placement Constraint object
[ https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16015028#comment-16015028 ] Konstantinos Karanasos edited comment on YARN-6593 at 5/18/17 1:13 AM: --- Thanks for the comments guys. So, we all agree on the validator, will open a JIRA to track this. +1 on having a string representation and a library to convert it to constraints. I think it will become very useful soon, but agreed it is not urgent. Self-reference works fine even with the current protobuf version. I think I saw a single use of it in our yarn_protos (I think it was in the QueueInfo or something related). That said, both approaches make sense to me. I guess the approach in the current patch is a little bit more cumbersome to right (we will have one extra call to create a PlacementConstraint). Is that its disadvantage or there is something else too? On the other hand, what I don't like if we coalesce the two, is that (1) the proto becomes too complicated (and I am afraid we will be the only ones understanding it :)), and (2) the client will still have access to very unrelated getters and setters. For example, the user could be creating a CompoundConstraint and have access to a getTargetExpression() function. was (Author: kkaranasos): Thanks for the comments guys. So, we all agree on the validator, will open a JIRA to track this. +1 on having a string representation and a library to convert it to constraints. I think it will become very useful soon, but agreed it is not urgent. Self-reference works fine even with the current protobuf version. I think I saw a single use of it in our yarn_protos (I think it was in the QueueInfo or something related). That said, both approaches make sense to me. I guess the approach in the current JIRA is a little bit more cumbersome to right (we will have one extra call to create a PlacementConstraint). Is that its disadvantage or there is something else too? On the other hand, what I don't like if we coalesce the two, is that (1) the proto becomes too complicated (and I am afraid we will be the only ones understanding it :)), and (2) the client will still have access to very unrelated getters and setters. For example, the user could be creating a CompoundConstraint and have access to a getTargetExpression() function. > [API] Introduce Placement Constraint object > --- > > Key: YARN-6593 > URL: https://issues.apache.org/jira/browse/YARN-6593 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-6593.001.patch > > > This JIRA introduces an object for defining placement constraints. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6593) [API] Introduce Placement Constraint object
[ https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16014435#comment-16014435 ] Konstantinos Karanasos edited comment on YARN-6593 at 5/17/17 5:34 PM: --- Attaching first patch for the constraints API. [~leftnoteasy], [~asuresh], [~pg1...@imperial.ac.uk], please give it a first look. was (Author: kkaranasos): Attaching first patch for the constraints API. [~leftnoteasy], [~asuresh], please give it a first look. > [API] Introduce Placement Constraint object > --- > > Key: YARN-6593 > URL: https://issues.apache.org/jira/browse/YARN-6593 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-6593.001.patch > > > This JIRA introduces an object for defining placement constraints. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
