[jira] [Comment Edited] (YARN-7863) Modify placement constraints to support node attributes

2018-08-26 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593109#comment-16593109
 ] 

Naganarasimha G R edited comment on YARN-7863 at 8/27/18 1:58 AM:
--

Hi [~sunilg],

             As discussed some comments can be handled in a separate jira, but 
still i could see test case coverage is not to that extent there for scheduler 
side of code.  *PlacementConstraintsUtil* : "Evaluation of Placement expression 
containing Node Attributes".

Also i believe i think [~cheersyang]'s comment is also valid, can you take a 
stab at it ?

Lets conclude next one as the final patch so that i can go ahead and commit it 
to unblock the overall functionality! For all others lets raise separate jira's.

 


was (Author: naganarasimha):
Hi [~sunilg],

             As discussed some comments can be handled in a separate jira, but 
still i could see test case coverage is not to that extent there from scheduler 
side of code.  *PlacementConstraintsUtil* : "Evaluation of Placement expression 
containing Node Attributes".

Also i believe i think [~cheersyang]'s comment is also valid, can you take a 
stab at it ?

Lets conclude next one as the final patch so that i can go ahead and commit it 
to unblock the overall functionality! For all others lets raise separate jira's.

 

> Modify placement constraints to support node attributes
> ---
>
> Key: YARN-7863
> URL: https://issues.apache.org/jira/browse/YARN-7863
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Sunil Govindan
>Assignee: Sunil Govindan
>Priority: Major
> Attachments: YARN-7863-YARN-3409.002.patch, 
> YARN-7863-YARN-3409.003.patch, YARN-7863-YARN-3409.004.patch, 
> YARN-7863-YARN-3409.005.patch, YARN-7863-YARN-3409.006.patch, 
> YARN-7863-YARN-3409.007.patch, YARN-7863-YARN-3409.008.patch, 
> YARN-7863-YARN-3409.009.patch, YARN-7863-YARN-3409.010.patch, 
> YARN-7863-YARN-3409.011.patch, YARN-7863.v0.patch
>
>
> This Jira will track to *Modify existing placement constraints to support 
> node attributes.*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-7863) Modify placement constraints to support node attributes

2018-08-25 Thread Sunil Govindan (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16592627#comment-16592627
 ] 

Sunil Govindan edited comment on YARN-7863 at 8/25/18 3:32 PM:
---

Thanks [~cheersyang] and [~Naganarasimha]

Improved syntax in latest patch. Now user can say like {{-placement_spec 
foo=4,java=1.8}}. By this user can easily specify constraint as EQ or NE. These 
are the 2 op codes which are supported now. 

Kindly help to check the patch.


was (Author: sunilg):
Thanks [~cheersyang] and [~Naganarasimha]

Improved syntax in latest patch. Now user can say like {{-placement_spec 
foo=4,java=1.8}}. By this user can easily specify constraint and EQ or NE. 
These are the 2 op codes which are supported now. 

Kindly help to check the patch.

> Modify placement constraints to support node attributes
> ---
>
> Key: YARN-7863
> URL: https://issues.apache.org/jira/browse/YARN-7863
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Sunil Govindan
>Assignee: Sunil Govindan
>Priority: Major
> Attachments: YARN-7863-YARN-3409.002.patch, 
> YARN-7863-YARN-3409.003.patch, YARN-7863-YARN-3409.004.patch, 
> YARN-7863-YARN-3409.005.patch, YARN-7863-YARN-3409.006.patch, 
> YARN-7863-YARN-3409.007.patch, YARN-7863-YARN-3409.008.patch, 
> YARN-7863-YARN-3409.009.patch, YARN-7863.v0.patch
>
>
> This Jira will track to *Modify existing placement constraints to support 
> node attributes.*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-7863) Modify placement constraints to support node attributes

2018-07-30 Thread Weiwei Yang (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16561858#comment-16561858
 ] 

Weiwei Yang edited comment on YARN-7863 at 7/30/18 12:33 PM:
-

{quote}
For DS, I think its better we correlate with placement_spec for now for better 
easiness in review and api compatibility.
{quote}
+1 for this idea. This way we can specify allocation tags and node attributes 
in PC expressions under same syntax.

but for this expression:
{noformat}
-placement_spec 
OR(IN,python=true:NOTIN,java=1.8):IN,os!=centos:NOTIN,env=prod,dev
{noformat}

I don't think we should use ":" to represent AND. The convention was to use ":" 
as the delimiter between each child of a conjunction expression (AND or OR). In 
the released version of DS, referring to 
[https://hadoop.apache.org/docs/r3.1.0/hadoop-yarn/hadoop-yarn-site/PlacementConstraints.html|https://hadoop.apache.org/docs/r3.1.0/hadoop-yarn/hadoop-yarn-site/PlacementConstraints.html],

{noformat}
zk=3,NOTIN,NODE,zk:hbase=5,IN,RACK,zk:spark=7,CARDINALITY,NODE,hbase,1,3
{noformat}
this actually represents 3 separate placement constraints instead of a AND with 
3 children.


was (Author: cheersyang):
{quote}
For DS, I think its better we correlate with placement_spec for now for better 
easiness in review and api compatibility.
{quote}
+1 for this idea. This way we can specify allocation tags and node attributes 
in PC expressions under same syntax.

but for this expression:
{noformat}
-placement_spec 
OR(IN,python=true:NOTIN,java=1.8):IN,os!=centos:NOTIN,env=prod,dev
{noformat}

I don't think we should use ":" to represent AND. The convention was to use ":" 
as the delimiter between each child of a conjunction expression (AND or OR). In 
the released version of DS, referring to 
[https://hadoop.apache.org/docs/r3.1.0/hadoop-yarn/hadoop-yarn-site/PlacementConstraints.html|https://hadoop.apache.org/docs/r3.1.0/hadoop-yarn/hadoop-yarn-site/PlacementConstraints.html],

{noformat}
zk=3,NOTIN,NODE,zk:hbase=5,IN,RACK,zk:spark=7,CARDINALITY,NODE,hbase,1,3
{noformat}
this actually represents 3 separate placement constraints instead of a AND with 
3 children.


 

 

> Modify placement constraints to support node attributes
> ---
>
> Key: YARN-7863
> URL: https://issues.apache.org/jira/browse/YARN-7863
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Sunil Govindan
>Assignee: Sunil Govindan
>Priority: Major
> Attachments: YARN-7863-YARN-3409.002.patch, 
> YARN-7863-YARN-3409.003.patch, YARN-7863.v0.patch
>
>
> This Jira will track to *Modify existing placement constraints to support 
> node attributes.*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-7863) Modify placement constraints to support node attributes

2018-07-26 Thread Sunil Govindan (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16558728#comment-16558728
 ] 

Sunil Govindan edited comment on YARN-7863 at 7/26/18 6:35 PM:
---

Thanks [~Naganarasimha] and [~cheersyang] for comments

A quick summary based on an offline discussion:
{noformat}
DS Example:

bin/yarn org.apache.hadoop.yarn.applications.distributedshell.Client -jar 
share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.0-SNAPSHOT.jar 
-shell_command Sleep
-shell_args 10
-num_containers 2
-master_memory 200
-container_memory 200
-placement_spec 
OR(IN,python=true:NOTIN,java=1.8):IN,os!=centos:NOTIN,env=prod,dev
{noformat}
For DS, I think its better we correlate with placement_spec for now for better 
easiness in review and api compatibility. A refactor improvement could be done 
later to improvise this inline with native service approach.
 Proposal for jira for later time is
 {{-placement_spec (python=true OR java!=1.8) AND (os!=centos) AND (env NOTIN 
(prod,dev))}}

Now to expand more on below approach in patch, 
{noformat}
-placement_spec 
OR(IN,python=true:NOTIN,java=1.8):IN,os!=centos:NOTIN,env=prod,dev{noformat}
 
 we could see this like
 *put me to a node where it has node-attribute python=true.*
 OR
 *dont put me in a node where it has node-attribute java=1.8*

AND
 *put me to a node where it has node-attribute os is not centos.*
 AND
 *dont put me in a node where it has attributes env with values prod or dev*

Also attaching a patch based on this including comments from Naga. I ll add 
test case in next patch.


was (Author: sunilg):
Thanks [~Naganarasimha] and [~cheersyang] for comments

A quick summary based on an offline discussion:
{noformat}
DS Example:

bin/yarn org.apache.hadoop.yarn.applications.distributedshell.Client -jar 
share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.0-SNAPSHOT.jar 
-shell_command Sleep
-shell_args 10
-num_containers 2
-master_memory 200
-container_memory 200
-placement_spec 
OR(IN,python=true:NOTIN,java=1.8):IN,os!=centos:NOTIN,env=prod,dev
{noformat}
For DS, I think its better we correlate with placement_spec for now for better 
easiness in review and api compatibility. A refactor improvement could be done 
later to improvise this inline with native service approach.
 Proposal for jira for later time is
 {{-placement_spec (python=true OR java!=1.8) AND (os!=centos) AND (env NOTIN 
(prod,dev))}}

Now to expand more on below approach in patch, 
{noformat}
-placement_spec 
OR(IN,python=true:NOTIN,java=1.8):IN,os!=centos:NOTIN,env=prod,dev{noformat}
 
 we could see this like
 *put me to a node where it has node-attribute python=true.*
 OR
 *dont put me in a node where it has node-attribute java=1.8*

AND
 *put me to a node where it has node-attribute os is not centos.*
 AND
 *dont put me in a node where it has attributes env with values prod or dev*

 

Also attaching a patch based on this including comments from Naga.

 

> Modify placement constraints to support node attributes
> ---
>
> Key: YARN-7863
> URL: https://issues.apache.org/jira/browse/YARN-7863
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Sunil Govindan
>Assignee: Sunil Govindan
>Priority: Major
> Attachments: YARN-7863-YARN-3409.002.patch, 
> YARN-7863-YARN-3409.003.patch, YARN-7863.v0.patch
>
>
> This Jira will track to *Modify existing placement constraints to support 
> node attributes.*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-7863) Modify placement constraints to support node attributes

2018-07-01 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16529111#comment-16529111
 ] 

Naganarasimha G R edited comment on YARN-7863 at 7/1/18 2:19 PM:
-

Thanks for the patch [~sunilg],  It gives a very good idea what you plan to do 
in scheduler for node attributes.  At the high level overview approach seems to 
be  fine. Hope you can change the status of the Jira to Patch available so that 
Jenkins run can happen. 

Also please modify the attached patch format so that it jenkins runs against 
YARN-3409 branch


was (Author: naganarasimha):
Thanks for the patch [~sunilg],  It gives a very good idea what you plan to do 
in scheduler for node attributes.  At the high level overview approach seems to 
be  fine. Hope you can change the status of the Jira to Patch available so that 
Jenkins run can happen. 

> Modify placement constraints to support node attributes
> ---
>
> Key: YARN-7863
> URL: https://issues.apache.org/jira/browse/YARN-7863
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Sunil Govindan
>Assignee: Sunil Govindan
>Priority: Major
> Attachments: YARN-7863.v0.patch
>
>
> This Jira will track to *Modify existing placement constraints to support 
> node attributes.*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-7863) Modify placement constraints to support node attributes

2018-01-31 Thread Arun Suresh (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346391#comment-16346391
 ] 

Arun Suresh edited comment on YARN-7863 at 1/31/18 7:59 AM:


[~sunilg], There are actually two aspects to this:
# Supporting node attributes as a Target Expression: For eg: place containers 
with allocation tag "foo" on nodes with attribute java_version = 1.8.
# Supporting node attributes as a Scope: For eg: place 5 containers, no more 
than one per failure domain. Assuming all nodes in cluster has a node attribute 
called "failure_domain" with values 1 - 10.

1) can be tackled here I guess. but 2) needs some modifications to the 
AllocationTagsManager and should be handled as part of YARN-7858.

Thoughts [~kkaranasos] / [~leftnoteasy] ? 


was (Author: asuresh):
[~sunilg], There are actually two aspects to this:
# Supporting node attributes as a Target Expression: For eg: place containers 
with allocation tag "foo" on nodes with attribute java_version = 1.8.
# Supporting node attributes as a Scope: For eg: place 5 containers, no more 
than one per failure domain. Assuming all nodes in cluster has a node attribute 
called "failure_domain" with values 1 - 10.

1) can be tackled here I guess. but 2) needs some modifications to the 
AllocationTagsManager and should be handled YARN-7858.

Thoughts [~kkaranasos] / [~leftnoteasy] ? 

> Modify placement constraints to support node attributes
> ---
>
> Key: YARN-7863
> URL: https://issues.apache.org/jira/browse/YARN-7863
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Sunil G
>Assignee: Sunil G
>Priority: Major
>
> This Jira will track to *Modify existing placement constraints to support 
> node attributes.*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-7863) Modify placement constraints to support node attributes

2018-01-31 Thread Arun Suresh (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346391#comment-16346391
 ] 

Arun Suresh edited comment on YARN-7863 at 1/31/18 7:59 AM:


[~sunilg], There are actually two aspects to this:
# Supporting node attributes as a Target Expression: For eg: place containers 
with allocation tag "foo" on nodes with attribute java_version = 1.8.
# Supporting node attributes as a Scope: For eg: place 5 containers, no more 
than one per failure domain. Assuming all nodes in cluster has a node attribute 
called "failure_domain" with values 1 - 10.

1) can be tackled here I guess. but 2) needs some modifications to the 
AllocationTagsManager and should be handled YARN-7858.

Thoughts [~kkaranasos] / [~leftnoteasy] ? 


was (Author: asuresh):
[~sunilg], There are actually two aspects to this:
# Supporting node attributes as a Target Expression: For eg: place containers 
with allocation tags on nodes with attribute java_version = 1.8.
# Supporting node attributes as a Scope: For eg: place 5 containers, no more 
than one per failure domain. Assuming all nodes in cluster has a node attribute 
called "failure_domain" with values 1 - 10.

1) can be tackled here I guess. but 2) needs some modifications to the 
AllocationTagsManager and should be handled YARN-7858.

Thoughts [~kkaranasos] / [~leftnoteasy] ? 

> Modify placement constraints to support node attributes
> ---
>
> Key: YARN-7863
> URL: https://issues.apache.org/jira/browse/YARN-7863
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Sunil G
>Assignee: Sunil G
>Priority: Major
>
> This Jira will track to *Modify existing placement constraints to support 
> node attributes.*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-7863) Modify placement constraints to support node attributes

2018-01-30 Thread Arun Suresh (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346391#comment-16346391
 ] 

Arun Suresh edited comment on YARN-7863 at 1/31/18 7:58 AM:


[~sunilg], There are actually two aspects to this:
# Supporting node attributes as a Target Expression: For eg: place containers 
with allocation tags on nodes with attribute java_version = 1.8.
# Supporting node attributes as a Scope: For eg: place 5 containers, no more 
than one per failure domain. Assuming all nodes in cluster has a node attribute 
called "failure_domain" with values 1 - 10.

1) can be tackled here I guess. but 2) needs some modifications to the 
AllocationTagsManager and should be handled YARN-7858.

Thoughts [~kkaranasos] / [~leftnoteasy] ? 


was (Author: asuresh):
[~sunilg], There are actually two aspects to this:
# Supporting node attributes as a Target Expression: For eg: place containers 
with allocation tags on nodes with attribute java_version = 1.8.
# Supporting a node attribute as a Scope: For eg: place 5 containers, no more 
than one per failure domain. Assuming all nodes in cluster has a node attribute 
called "failure_domain" with values 1 - 10.

1) can be tackled here I guess. but 2) needs some modifications to the 
AllocationTagsManager and should be handled YARN-7858.

Thoughts [~kkaranasos] / [~leftnoteasy] ? 

> Modify placement constraints to support node attributes
> ---
>
> Key: YARN-7863
> URL: https://issues.apache.org/jira/browse/YARN-7863
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Sunil G
>Assignee: Sunil G
>Priority: Major
>
> This Jira will track to *Modify existing placement constraints to support 
> node attributes.*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org