[jira] [Comment Edited] (YARN-7863) Modify placement constraints to support node attributes
[ https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593109#comment-16593109 ] Naganarasimha G R edited comment on YARN-7863 at 8/27/18 1:58 AM: -- Hi [~sunilg], As discussed some comments can be handled in a separate jira, but still i could see test case coverage is not to that extent there for scheduler side of code. *PlacementConstraintsUtil* : "Evaluation of Placement expression containing Node Attributes". Also i believe i think [~cheersyang]'s comment is also valid, can you take a stab at it ? Lets conclude next one as the final patch so that i can go ahead and commit it to unblock the overall functionality! For all others lets raise separate jira's. was (Author: naganarasimha): Hi [~sunilg], As discussed some comments can be handled in a separate jira, but still i could see test case coverage is not to that extent there from scheduler side of code. *PlacementConstraintsUtil* : "Evaluation of Placement expression containing Node Attributes". Also i believe i think [~cheersyang]'s comment is also valid, can you take a stab at it ? Lets conclude next one as the final patch so that i can go ahead and commit it to unblock the overall functionality! For all others lets raise separate jira's. > Modify placement constraints to support node attributes > --- > > Key: YARN-7863 > URL: https://issues.apache.org/jira/browse/YARN-7863 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Attachments: YARN-7863-YARN-3409.002.patch, > YARN-7863-YARN-3409.003.patch, YARN-7863-YARN-3409.004.patch, > YARN-7863-YARN-3409.005.patch, YARN-7863-YARN-3409.006.patch, > YARN-7863-YARN-3409.007.patch, YARN-7863-YARN-3409.008.patch, > YARN-7863-YARN-3409.009.patch, YARN-7863-YARN-3409.010.patch, > YARN-7863-YARN-3409.011.patch, YARN-7863.v0.patch > > > This Jira will track to *Modify existing placement constraints to support > node attributes.* -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7863) Modify placement constraints to support node attributes
[
https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16592627#comment-16592627
]
Sunil Govindan edited comment on YARN-7863 at 8/25/18 3:32 PM:
---
Thanks [~cheersyang] and [~Naganarasimha]
Improved syntax in latest patch. Now user can say like {{-placement_spec
foo=4,java=1.8}}. By this user can easily specify constraint as EQ or NE. These
are the 2 op codes which are supported now.
Kindly help to check the patch.
was (Author: sunilg):
Thanks [~cheersyang] and [~Naganarasimha]
Improved syntax in latest patch. Now user can say like {{-placement_spec
foo=4,java=1.8}}. By this user can easily specify constraint and EQ or NE.
These are the 2 op codes which are supported now.
Kindly help to check the patch.
> Modify placement constraints to support node attributes
> ---
>
> Key: YARN-7863
> URL: https://issues.apache.org/jira/browse/YARN-7863
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Sunil Govindan
>Assignee: Sunil Govindan
>Priority: Major
> Attachments: YARN-7863-YARN-3409.002.patch,
> YARN-7863-YARN-3409.003.patch, YARN-7863-YARN-3409.004.patch,
> YARN-7863-YARN-3409.005.patch, YARN-7863-YARN-3409.006.patch,
> YARN-7863-YARN-3409.007.patch, YARN-7863-YARN-3409.008.patch,
> YARN-7863-YARN-3409.009.patch, YARN-7863.v0.patch
>
>
> This Jira will track to *Modify existing placement constraints to support
> node attributes.*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7863) Modify placement constraints to support node attributes
[
https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16561858#comment-16561858
]
Weiwei Yang edited comment on YARN-7863 at 7/30/18 12:33 PM:
-
{quote}
For DS, I think its better we correlate with placement_spec for now for better
easiness in review and api compatibility.
{quote}
+1 for this idea. This way we can specify allocation tags and node attributes
in PC expressions under same syntax.
but for this expression:
{noformat}
-placement_spec
OR(IN,python=true:NOTIN,java=1.8):IN,os!=centos:NOTIN,env=prod,dev
{noformat}
I don't think we should use ":" to represent AND. The convention was to use ":"
as the delimiter between each child of a conjunction expression (AND or OR). In
the released version of DS, referring to
[https://hadoop.apache.org/docs/r3.1.0/hadoop-yarn/hadoop-yarn-site/PlacementConstraints.html|https://hadoop.apache.org/docs/r3.1.0/hadoop-yarn/hadoop-yarn-site/PlacementConstraints.html],
{noformat}
zk=3,NOTIN,NODE,zk:hbase=5,IN,RACK,zk:spark=7,CARDINALITY,NODE,hbase,1,3
{noformat}
this actually represents 3 separate placement constraints instead of a AND with
3 children.
was (Author: cheersyang):
{quote}
For DS, I think its better we correlate with placement_spec for now for better
easiness in review and api compatibility.
{quote}
+1 for this idea. This way we can specify allocation tags and node attributes
in PC expressions under same syntax.
but for this expression:
{noformat}
-placement_spec
OR(IN,python=true:NOTIN,java=1.8):IN,os!=centos:NOTIN,env=prod,dev
{noformat}
I don't think we should use ":" to represent AND. The convention was to use ":"
as the delimiter between each child of a conjunction expression (AND or OR). In
the released version of DS, referring to
[https://hadoop.apache.org/docs/r3.1.0/hadoop-yarn/hadoop-yarn-site/PlacementConstraints.html|https://hadoop.apache.org/docs/r3.1.0/hadoop-yarn/hadoop-yarn-site/PlacementConstraints.html],
{noformat}
zk=3,NOTIN,NODE,zk:hbase=5,IN,RACK,zk:spark=7,CARDINALITY,NODE,hbase,1,3
{noformat}
this actually represents 3 separate placement constraints instead of a AND with
3 children.
> Modify placement constraints to support node attributes
> ---
>
> Key: YARN-7863
> URL: https://issues.apache.org/jira/browse/YARN-7863
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Sunil Govindan
>Assignee: Sunil Govindan
>Priority: Major
> Attachments: YARN-7863-YARN-3409.002.patch,
> YARN-7863-YARN-3409.003.patch, YARN-7863.v0.patch
>
>
> This Jira will track to *Modify existing placement constraints to support
> node attributes.*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7863) Modify placement constraints to support node attributes
[
https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16558728#comment-16558728
]
Sunil Govindan edited comment on YARN-7863 at 7/26/18 6:35 PM:
---
Thanks [~Naganarasimha] and [~cheersyang] for comments
A quick summary based on an offline discussion:
{noformat}
DS Example:
bin/yarn org.apache.hadoop.yarn.applications.distributedshell.Client -jar
share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.0-SNAPSHOT.jar
-shell_command Sleep
-shell_args 10
-num_containers 2
-master_memory 200
-container_memory 200
-placement_spec
OR(IN,python=true:NOTIN,java=1.8):IN,os!=centos:NOTIN,env=prod,dev
{noformat}
For DS, I think its better we correlate with placement_spec for now for better
easiness in review and api compatibility. A refactor improvement could be done
later to improvise this inline with native service approach.
Proposal for jira for later time is
{{-placement_spec (python=true OR java!=1.8) AND (os!=centos) AND (env NOTIN
(prod,dev))}}
Now to expand more on below approach in patch,
{noformat}
-placement_spec
OR(IN,python=true:NOTIN,java=1.8):IN,os!=centos:NOTIN,env=prod,dev{noformat}
we could see this like
*put me to a node where it has node-attribute python=true.*
OR
*dont put me in a node where it has node-attribute java=1.8*
AND
*put me to a node where it has node-attribute os is not centos.*
AND
*dont put me in a node where it has attributes env with values prod or dev*
Also attaching a patch based on this including comments from Naga. I ll add
test case in next patch.
was (Author: sunilg):
Thanks [~Naganarasimha] and [~cheersyang] for comments
A quick summary based on an offline discussion:
{noformat}
DS Example:
bin/yarn org.apache.hadoop.yarn.applications.distributedshell.Client -jar
share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.0-SNAPSHOT.jar
-shell_command Sleep
-shell_args 10
-num_containers 2
-master_memory 200
-container_memory 200
-placement_spec
OR(IN,python=true:NOTIN,java=1.8):IN,os!=centos:NOTIN,env=prod,dev
{noformat}
For DS, I think its better we correlate with placement_spec for now for better
easiness in review and api compatibility. A refactor improvement could be done
later to improvise this inline with native service approach.
Proposal for jira for later time is
{{-placement_spec (python=true OR java!=1.8) AND (os!=centos) AND (env NOTIN
(prod,dev))}}
Now to expand more on below approach in patch,
{noformat}
-placement_spec
OR(IN,python=true:NOTIN,java=1.8):IN,os!=centos:NOTIN,env=prod,dev{noformat}
we could see this like
*put me to a node where it has node-attribute python=true.*
OR
*dont put me in a node where it has node-attribute java=1.8*
AND
*put me to a node where it has node-attribute os is not centos.*
AND
*dont put me in a node where it has attributes env with values prod or dev*
Also attaching a patch based on this including comments from Naga.
> Modify placement constraints to support node attributes
> ---
>
> Key: YARN-7863
> URL: https://issues.apache.org/jira/browse/YARN-7863
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Sunil Govindan
>Assignee: Sunil Govindan
>Priority: Major
> Attachments: YARN-7863-YARN-3409.002.patch,
> YARN-7863-YARN-3409.003.patch, YARN-7863.v0.patch
>
>
> This Jira will track to *Modify existing placement constraints to support
> node attributes.*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7863) Modify placement constraints to support node attributes
[ https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16529111#comment-16529111 ] Naganarasimha G R edited comment on YARN-7863 at 7/1/18 2:19 PM: - Thanks for the patch [~sunilg], It gives a very good idea what you plan to do in scheduler for node attributes. At the high level overview approach seems to be fine. Hope you can change the status of the Jira to Patch available so that Jenkins run can happen. Also please modify the attached patch format so that it jenkins runs against YARN-3409 branch was (Author: naganarasimha): Thanks for the patch [~sunilg], It gives a very good idea what you plan to do in scheduler for node attributes. At the high level overview approach seems to be fine. Hope you can change the status of the Jira to Patch available so that Jenkins run can happen. > Modify placement constraints to support node attributes > --- > > Key: YARN-7863 > URL: https://issues.apache.org/jira/browse/YARN-7863 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Attachments: YARN-7863.v0.patch > > > This Jira will track to *Modify existing placement constraints to support > node attributes.* -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7863) Modify placement constraints to support node attributes
[ https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346391#comment-16346391 ] Arun Suresh edited comment on YARN-7863 at 1/31/18 7:59 AM: [~sunilg], There are actually two aspects to this: # Supporting node attributes as a Target Expression: For eg: place containers with allocation tag "foo" on nodes with attribute java_version = 1.8. # Supporting node attributes as a Scope: For eg: place 5 containers, no more than one per failure domain. Assuming all nodes in cluster has a node attribute called "failure_domain" with values 1 - 10. 1) can be tackled here I guess. but 2) needs some modifications to the AllocationTagsManager and should be handled as part of YARN-7858. Thoughts [~kkaranasos] / [~leftnoteasy] ? was (Author: asuresh): [~sunilg], There are actually two aspects to this: # Supporting node attributes as a Target Expression: For eg: place containers with allocation tag "foo" on nodes with attribute java_version = 1.8. # Supporting node attributes as a Scope: For eg: place 5 containers, no more than one per failure domain. Assuming all nodes in cluster has a node attribute called "failure_domain" with values 1 - 10. 1) can be tackled here I guess. but 2) needs some modifications to the AllocationTagsManager and should be handled YARN-7858. Thoughts [~kkaranasos] / [~leftnoteasy] ? > Modify placement constraints to support node attributes > --- > > Key: YARN-7863 > URL: https://issues.apache.org/jira/browse/YARN-7863 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil G >Assignee: Sunil G >Priority: Major > > This Jira will track to *Modify existing placement constraints to support > node attributes.* -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7863) Modify placement constraints to support node attributes
[ https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346391#comment-16346391 ] Arun Suresh edited comment on YARN-7863 at 1/31/18 7:59 AM: [~sunilg], There are actually two aspects to this: # Supporting node attributes as a Target Expression: For eg: place containers with allocation tag "foo" on nodes with attribute java_version = 1.8. # Supporting node attributes as a Scope: For eg: place 5 containers, no more than one per failure domain. Assuming all nodes in cluster has a node attribute called "failure_domain" with values 1 - 10. 1) can be tackled here I guess. but 2) needs some modifications to the AllocationTagsManager and should be handled YARN-7858. Thoughts [~kkaranasos] / [~leftnoteasy] ? was (Author: asuresh): [~sunilg], There are actually two aspects to this: # Supporting node attributes as a Target Expression: For eg: place containers with allocation tags on nodes with attribute java_version = 1.8. # Supporting node attributes as a Scope: For eg: place 5 containers, no more than one per failure domain. Assuming all nodes in cluster has a node attribute called "failure_domain" with values 1 - 10. 1) can be tackled here I guess. but 2) needs some modifications to the AllocationTagsManager and should be handled YARN-7858. Thoughts [~kkaranasos] / [~leftnoteasy] ? > Modify placement constraints to support node attributes > --- > > Key: YARN-7863 > URL: https://issues.apache.org/jira/browse/YARN-7863 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil G >Assignee: Sunil G >Priority: Major > > This Jira will track to *Modify existing placement constraints to support > node attributes.* -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7863) Modify placement constraints to support node attributes
[ https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346391#comment-16346391 ] Arun Suresh edited comment on YARN-7863 at 1/31/18 7:58 AM: [~sunilg], There are actually two aspects to this: # Supporting node attributes as a Target Expression: For eg: place containers with allocation tags on nodes with attribute java_version = 1.8. # Supporting node attributes as a Scope: For eg: place 5 containers, no more than one per failure domain. Assuming all nodes in cluster has a node attribute called "failure_domain" with values 1 - 10. 1) can be tackled here I guess. but 2) needs some modifications to the AllocationTagsManager and should be handled YARN-7858. Thoughts [~kkaranasos] / [~leftnoteasy] ? was (Author: asuresh): [~sunilg], There are actually two aspects to this: # Supporting node attributes as a Target Expression: For eg: place containers with allocation tags on nodes with attribute java_version = 1.8. # Supporting a node attribute as a Scope: For eg: place 5 containers, no more than one per failure domain. Assuming all nodes in cluster has a node attribute called "failure_domain" with values 1 - 10. 1) can be tackled here I guess. but 2) needs some modifications to the AllocationTagsManager and should be handled YARN-7858. Thoughts [~kkaranasos] / [~leftnoteasy] ? > Modify placement constraints to support node attributes > --- > > Key: YARN-7863 > URL: https://issues.apache.org/jira/browse/YARN-7863 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil G >Assignee: Sunil G >Priority: Major > > This Jira will track to *Modify existing placement constraints to support > node attributes.* -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
