[jira] [Commented] (YARN-6857) Support REST for Node Attributes configurations

2020-06-03 Thread Naganarasimha G R (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124842#comment-17124842
 ] 

Naganarasimha G R commented on YARN-6857:
-

[~prabhujoseph], Glad that you and [~bilwa_st]  can take it forward, please go 
ahead and assign it to yourself!

> Support REST for Node Attributes configurations
> ---
>
> Key: YARN-6857
> URL: https://issues.apache.org/jira/browse/YARN-6857
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: api, capacityscheduler, client
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
>Priority: Major
> Attachments: YARN-6857-YARN-3409.001.patch
>
>
> This jira focusses on supporting mapping of Nodes to  Attributes through REST



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-8768) Javadoc error in node attributes

2018-09-12 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612753#comment-16612753
 ] 

Naganarasimha G R commented on YARN-8768:
-

Thanks [~shaneku...@gmail.com] for merging it

> Javadoc error in node attributes
> 
>
> Key: YARN-8768
> URL: https://issues.apache.org/jira/browse/YARN-8768
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Sunil Govindan
>Assignee: Sunil Govindan
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: YARN-8768.001.patch
>
>
> fix java doc error from node attributes in yarn-api module.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-3409) Support Node Attribute functionality

2018-09-12 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612010#comment-16612010
 ] 

Naganarasimha G R commented on YARN-3409:
-

All open items has been moved to YARN-8766, and other planned jiras will also 
be tracked as part of YARN-8766.

> Support Node Attribute functionality
> 
>
> Key: YARN-3409
> URL: https://issues.apache.org/jira/browse/YARN-3409
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: api, client, RM
>Reporter: Wangda Tan
>Assignee: Naganarasimha G R
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: 3409-apiChanges_v2.pdf (4).pdf, 
> Constraint-Node-Labels-Requirements-Design-doc_v1.pdf, 
> Node-Attributes-Requirements-Design-doc_v2.pdf, YARN-3409.WIP.001.patch
>
>
> Specify only one label for each node (IAW, partition a cluster) is a way to 
> determinate how resources of a special set of nodes could be shared by a 
> group of entities (like teams, departments, etc.). Partitions of a cluster 
> has following characteristics:
> - Cluster divided to several disjoint sub clusters.
> - ACL/priority can apply on partition (Only market team / marke team has 
> priority to use the partition).
> - Percentage of capacities can apply on partition (Market team has 40% 
> minimum capacity and Dev team has 60% of minimum capacity of the partition).
> Attributes are orthogonal to partition, they’re describing features of node’s 
> hardware/software just for affinity. Some example of attributes:
> - glibc version
> - JDK version
> - Type of CPU (x86_64/i686)
> - Type of OS (windows, linux, etc.)
> With this, application can be able to ask for resource has (glibc.version >= 
> 2.20 && JDK.version >= 8u20 && x86_64).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Resolved] (YARN-3409) Support Node Attribute functionality

2018-09-12 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R resolved YARN-3409.
-
   Resolution: Fixed
Fix Version/s: 3.2.0
 Release Note: Node Attribute Feature has been added as part of this jira.

> Support Node Attribute functionality
> 
>
> Key: YARN-3409
> URL: https://issues.apache.org/jira/browse/YARN-3409
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: api, client, RM
>Reporter: Wangda Tan
>Assignee: Naganarasimha G R
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: 3409-apiChanges_v2.pdf (4).pdf, 
> Constraint-Node-Labels-Requirements-Design-doc_v1.pdf, 
> Node-Attributes-Requirements-Design-doc_v2.pdf, YARN-3409.WIP.001.patch
>
>
> Specify only one label for each node (IAW, partition a cluster) is a way to 
> determinate how resources of a special set of nodes could be shared by a 
> group of entities (like teams, departments, etc.). Partitions of a cluster 
> has following characteristics:
> - Cluster divided to several disjoint sub clusters.
> - ACL/priority can apply on partition (Only market team / marke team has 
> priority to use the partition).
> - Percentage of capacities can apply on partition (Market team has 40% 
> minimum capacity and Dev team has 60% of minimum capacity of the partition).
> Attributes are orthogonal to partition, they’re describing features of node’s 
> hardware/software just for affinity. Some example of attributes:
> - glibc version
> - JDK version
> - Type of CPU (x86_64/i686)
> - Type of OS (windows, linux, etc.)
> With this, application can be able to ask for resource has (glibc.version >= 
> 2.20 && JDK.version >= 8u20 && x86_64).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-8715) Make allocation tags in the placement spec optional for node-attributes

2018-09-12 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-8715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R updated YARN-8715:

Parent Issue: YARN-8766  (was: YARN-3409)

> Make allocation tags in the placement spec optional for node-attributes
> ---
>
> Key: YARN-8715
> URL: https://issues.apache.org/jira/browse/YARN-8715
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
>
> YARN-7863 adds support to specify constraints targeting to node-attributes, 
> including the support in distributed shell, but it still needs to specify 
> {{allocationTags=numOfContainers}} in the spec. We should make this optional 
> as it is not required for node-attribute expressions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-8102) Retrospect on having enable and disable flag for Node Attribute

2018-09-12 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-8102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R updated YARN-8102:

Parent Issue: YARN-8766  (was: YARN-3409)

> Retrospect on having enable and disable flag for Node Attribute
> ---
>
> Key: YARN-8102
> URL: https://issues.apache.org/jira/browse/YARN-8102
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Major
>
> Currently node attribute feature is by default enabled. We have to revisit on 
> the same.
> Enabling by default means will try to create store for all cluster 
> installation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-8101) Add UT to verify node-attributes in RM nodes rest API

2018-09-12 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-8101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R updated YARN-8101:

Parent Issue: YARN-8766  (was: YARN-3409)

> Add UT to verify node-attributes in RM nodes rest API
> -
>
> Key: YARN-8101
> URL: https://issues.apache.org/jira/browse/YARN-8101
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: resourcemanager, restapi
>Affects Versions: YARN-3409
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Minor
>
> YARN-8092 exposed node-attributes in RM rest api, this task is created to add 
> UT for that functionality.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-7877) Node attributes error handling based on RM validation response

2018-09-12 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R updated YARN-7877:

Parent Issue: YARN-8766  (was: YARN-3409)

> Node attributes error handling based on RM validation response
> --
>
> Key: YARN-7877
> URL: https://issues.apache.org/jira/browse/YARN-7877
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
>
> When NM reports its distributed attributes to RM, RM needs to do proper 
> validation of the received attributes, if attributes were not valid or failed 
> to update, RM needs to notify NM about such failures. Such validation needs 
> to be also done in NM registration as well.
> This JIRA tracks how NM re-act against RM response, probably we can display 
> some info through UI/metrics/log etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-6857) Support REST for Node Attributes configurations

2018-09-12 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R updated YARN-6857:

Parent Issue: YARN-8766  (was: YARN-3409)

> Support REST for Node Attributes configurations
> ---
>
> Key: YARN-6857
> URL: https://issues.apache.org/jira/browse/YARN-6857
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: api, capacityscheduler, client
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
>Priority: Major
> Attachments: YARN-6857-YARN-3409.001.patch
>
>
> This jira focusses on supporting mapping of Nodes to  Attributes through REST



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-8766) Support for Node Attribute Functionality, Phase 2

2018-09-12 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-8766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R updated YARN-8766:

Description: This Jira will track all the improvements on the Node 
Attribute functionality developed in YARN-3409

> Support for Node Attribute Functionality, Phase 2
> -
>
> Key: YARN-8766
> URL: https://issues.apache.org/jira/browse/YARN-8766
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
>Priority: Major
>
> This Jira will track all the improvements on the Node Attribute functionality 
> developed in YARN-3409



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Created] (YARN-8766) Support for Node Attribute Functionality, Phase 2

2018-09-12 Thread Naganarasimha G R (JIRA)
Naganarasimha G R created YARN-8766:
---

 Summary: Support for Node Attribute Functionality, Phase 2
 Key: YARN-8766
 URL: https://issues.apache.org/jira/browse/YARN-8766
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Naganarasimha G R






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Assigned] (YARN-8766) Support for Node Attribute Functionality, Phase 2

2018-09-12 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-8766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R reassigned YARN-8766:
---

Assignee: Naganarasimha G R

> Support for Node Attribute Functionality, Phase 2
> -
>
> Key: YARN-8766
> URL: https://issues.apache.org/jira/browse/YARN-8766
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-8766) Support for Node Attribute Functionality, Phase 2

2018-09-12 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-8766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R updated YARN-8766:

Issue Type: Improvement  (was: Bug)

> Support for Node Attribute Functionality, Phase 2
> -
>
> Key: YARN-8766
> URL: https://issues.apache.org/jira/browse/YARN-8766
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-7877) Node attributes error handling based on RM validation response

2018-09-12 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R updated YARN-7877:

Fix Version/s: (was: YARN-3409)

> Node attributes error handling based on RM validation response
> --
>
> Key: YARN-7877
> URL: https://issues.apache.org/jira/browse/YARN-7877
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
>
> When NM reports its distributed attributes to RM, RM needs to do proper 
> validation of the received attributes, if attributes were not valid or failed 
> to update, RM needs to notify NM about such failures. Such validation needs 
> to be also done in NM registration as well.
> This JIRA tracks how NM re-act against RM response, probably we can display 
> some info through UI/metrics/log etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-8670) Support scheduling request for SLS input and attributes for Node in SLS

2018-09-12 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611957#comment-16611957
 ] 

Naganarasimha G R commented on YARN-8670:
-

hi [~Sichen Zhao],   please ensure to set target version for desired version to 
be handled and not Fix version, fix version is updated once its closed

> Support scheduling request for SLS input and attributes for Node in SLS
> ---
>
> Key: YARN-8670
> URL: https://issues.apache.org/jira/browse/YARN-8670
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: scheduler-load-simulator
>Affects Versions: YARN-3409
>Reporter: Sichen zhao
>Priority: Major
>
> YARN-3409 introduces placement constraint, Currently SLS does not support 
> specify placement constraint. 
> YARN-8007 support specifying placement constraint for task containers in SLS. 
> But there are still 
> some room for improvement:
>  # YARN-8007 only support placement constraint for the jobs level. In fact, 
> the more flexible way is support placement constraint for the tasks level.
>  # In most scenarios, node itself has some characteristics, called attribute, 
> which is not supported in SLS. So we can add the attribute on Nodes.
>  # YARN-8007 use the SYNTH as schedulingrequest input. But SYNTH can't create 
> a large number of specific resource requests. We wanna create a new 
> schedulingrequest input format(like sis format) for the more authentic input. 
> We can add some field in sis format, and this is schedulingrequest input 
> format.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-8670) Support scheduling request for SLS input and attributes for Node in SLS

2018-09-12 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-8670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R updated YARN-8670:

Fix Version/s: (was: YARN-3409)

> Support scheduling request for SLS input and attributes for Node in SLS
> ---
>
> Key: YARN-8670
> URL: https://issues.apache.org/jira/browse/YARN-8670
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: scheduler-load-simulator
>Affects Versions: YARN-3409
>Reporter: Sichen zhao
>Priority: Major
>
> YARN-3409 introduces placement constraint, Currently SLS does not support 
> specify placement constraint. 
> YARN-8007 support specifying placement constraint for task containers in SLS. 
> But there are still 
> some room for improvement:
>  # YARN-8007 only support placement constraint for the jobs level. In fact, 
> the more flexible way is support placement constraint for the tasks level.
>  # In most scenarios, node itself has some characteristics, called attribute, 
> which is not supported in SLS. So we can add the attribute on Nodes.
>  # YARN-8007 use the SYNTH as schedulingrequest input. But SYNTH can't create 
> a large number of specific resource requests. We wanna create a new 
> schedulingrequest input format(like sis format) for the more authentic input. 
> We can add some field in sis format, and this is schedulingrequest input 
> format.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-7988) Refactor FSNodeLabelStore code for Node Attributes store support

2018-09-12 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R updated YARN-7988:

Fix Version/s: (was: YARN-3409)
   3.2.0

> Refactor FSNodeLabelStore code for Node Attributes store support
> 
>
> Key: YARN-7988
> URL: https://issues.apache.org/jira/browse/YARN-7988
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: YARN-7988-YARN-3409.002.patch, 
> YARN-7988-YARN-3409.003.patch, YARN-7988-YARN-3409.004.patch, 
> YARN-7988-YARN-3409.005.patch, YARN-7988-YARN-3409.006.patch, 
> YARN-7988-YARN-3409.007.patch, YARN-7988.001.patch
>
>
> # Abstract out file FileSystemStore operation
> # Define EditLog Operartions  and Mirror operation
> # Support compatibility with old nodelabel store



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-7965) NodeAttributeManager add/get API is not working properly

2018-09-12 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R updated YARN-7965:

Fix Version/s: (was: YARN-3409)
   3.2.0

> NodeAttributeManager add/get API is not working properly
> 
>
> Key: YARN-7965
> URL: https://issues.apache.org/jira/browse/YARN-7965
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: resourcemanager
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Critical
> Fix For: 3.2.0
>
> Attachments: YARN-7965-YARN-3409.001.patch, 
> YARN-7965-YARN-3409.002.patch, YARN-7965-YARN-3409.003.patch, 
> YARN-7965-YARN-3409.004.patch
>
>
> Fix following issues,
>  # After add node attributes to the manager, could not retrieve newly added 
> attributes
>  # Get cluster attributes API should return empty set when given prefix has 
> no match
>  # When an attribute is removed from all nodes, the manager did not remove 
> this mapping
> and add UT



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-7892) Revisit NodeAttribute class structure

2018-09-12 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R updated YARN-7892:

Fix Version/s: (was: YARN-3409)
   3.2.0

> Revisit NodeAttribute class structure
> -
>
> Key: YARN-7892
> URL: https://issues.apache.org/jira/browse/YARN-7892
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: resourcemanager
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: YARN-7892-YARN-3409.001.patch, 
> YARN-7892-YARN-3409.002.patch, YARN-7892-YARN-3409.003.WIP.patch, 
> YARN-7892-YARN-3409.003.patch, YARN-7892-YARN-3409.004.patch, 
> YARN-7892-YARN-3409.005.patch, YARN-7892-YARN-3409.006.patch, 
> YARN-7892-YARN-3409.007.patch, YARN-7892-YARN-3409.008.patch, 
> YARN-7892-YARN-3409.009.patch, YARN-7892-YARN-3409.010.patch
>
>
> In the existing structure, we had kept the type and value along with the 
> attribute which would create confusion to the user to understand the APIs as 
> they would not be clear as to what needs to be sent for type and value while 
> fetching the mappings for node(s).
> As well as equals will not make sense when we compare only for prefix and 
> name where as values for them might be different.  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-7875) Node Attribute store for storing and recovering attributes

2018-09-12 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R updated YARN-7875:

Fix Version/s: (was: YARN-3409)
   3.2.0

> Node Attribute store for storing and recovering attributes
> --
>
> Key: YARN-7875
> URL: https://issues.apache.org/jira/browse/YARN-7875
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: YARN-7875-WIP.patch, YARN-7875-YARN-3409.001.patch, 
> YARN-7875-YARN-3409.002.patch, YARN-7875-YARN-3409.003.patch, 
> YARN-7875-YARN-3409.004.patch, YARN-7875-YARN-3409.005.patch, 
> YARN-7875-YARN-3409.006.patch, YARN-7875-YARN-3409.007.patch, 
> YARN-7875-YARN-3409.008.patch
>
>
> Similar to NodeLabelStore need to support NodeAttributeStore for persisting 
> attributes mapping to Nodes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-7871) Node attributes reporting from NM to RM

2018-09-12 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R updated YARN-7871:

Fix Version/s: (was: YARN-3409)
   3.2.0

> Node attributes reporting from NM to RM 
> 
>
> Key: YARN-7871
> URL: https://issues.apache.org/jira/browse/YARN-7871
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: YARN-7871-YARN-3409.001.patch, 
> YARN-7871-YARN-3409.002.patch, YARN-7871-YARN-3409.003.patch, 
> YARN-7871-YARN-3409.004.patch
>
>
> Support to initialize proper attribute provider based on user's configuration.
> NM collects node attributes from a configured attribute provider and send 
> them to RM via HB. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-7865) Node attributes documentation

2018-09-12 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R updated YARN-7865:

Fix Version/s: (was: YARN-3409)
   3.2.0

> Node attributes documentation
> -
>
> Key: YARN-7865
> URL: https://issues.apache.org/jira/browse/YARN-7865
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: documentation
>Reporter: Weiwei Yang
>Assignee: Naganarasimha G R
>Priority: Major
>  Labels: node-attributes
> Fix For: 3.2.0
>
> Attachments: NodeAttributes.html, YARN-7865-YARN-3409.001.patch, 
> YARN-7865-YARN-3409.002.patch, YARN-7865-YARN-3409.003.patch, 
> YARN-7865-YARN-3409.004.patch, YARN-7865-YARN-3409.005.patch
>
>
> We need proper docs to introduce how to enable node-attributes how to 
> configure providers, how to specify script paths, arguments in configuration, 
> what should be the proper permission of the script and who will run the 
> script. Also it would be good to add more info to the description of the 
> configuration properties.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-7863) Modify placement constraints to support node attributes

2018-09-12 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R updated YARN-7863:

Fix Version/s: (was: YARN-3409)
   3.2.0

> Modify placement constraints to support node attributes
> ---
>
> Key: YARN-7863
> URL: https://issues.apache.org/jira/browse/YARN-7863
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Sunil Govindan
>Assignee: Sunil Govindan
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: YARN-7863-YARN-3409.002.patch, 
> YARN-7863-YARN-3409.003.patch, YARN-7863-YARN-3409.004.patch, 
> YARN-7863-YARN-3409.005.patch, YARN-7863-YARN-3409.006.patch, 
> YARN-7863-YARN-3409.007.patch, YARN-7863-YARN-3409.008.patch, 
> YARN-7863-YARN-3409.009.patch, YARN-7863-YARN-3409.010.patch, 
> YARN-7863-YARN-3409.011.patch, YARN-7863.v0.patch
>
>
> This Jira will track to *Modify existing placement constraints to support 
> node attributes.*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-7856) Validate Node Attributes from NM

2018-09-12 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R updated YARN-7856:

Fix Version/s: (was: YARN-3409)
   3.2.0

> Validate Node Attributes from NM
> 
>
> Key: YARN-7856
> URL: https://issues.apache.org/jira/browse/YARN-7856
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager, RM
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: YARN-7856-YARN-3409.001.patch, 
> YARN-7856-YARN-3409.002.patch
>
>
> NM needs to do proper validation about the attributes before sending them to 
> RM, this includes
> # a valid prefix is presented
> # no duplicate entries
> # do not allow two attributes with same prefix/name but different types
> This could be an utility class that can be used on both RM/NM sides.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-7842) PB changes to carry node-attributes in NM heartbeat

2018-09-12 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R updated YARN-7842:

Fix Version/s: (was: YARN-3409)
   3.2.0

> PB changes to carry node-attributes in NM heartbeat
> ---
>
> Key: YARN-7842
> URL: https://issues.apache.org/jira/browse/YARN-7842
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: YARN-7842-YARN-3409.001.patch, 
> YARN-7842-YARN-3409.002.patch
>
>
> PB changes to carry node-attributes in NM heartbeat. Split from a larger 
> patch for easier review.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-7840) Update PB for prefix support of node attributes

2018-09-12 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R updated YARN-7840:

Fix Version/s: (was: YARN-3409)
   3.2.0

> Update PB for prefix support of node attributes
> ---
>
> Key: YARN-7840
> URL: https://issues.apache.org/jira/browse/YARN-7840
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Naganarasimha G R
>Priority: Blocker
> Fix For: 3.2.0
>
> Attachments: YARN-7840-YARN-3409.001.patch, 
> YARN-7840-YARN-3409.002.patch, YARN-7840-YARN-3409.003.patch, 
> YARN-7840-YARN-3409.004.patch
>
>
> We need to support prefix (namespace) for node attributes, this will add the 
> flexibility to provide ability to do proper ACL, avoid naming conflicts etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-7757) Refactor NodeLabelsProvider to be more generic and reusable for node attributes providers

2018-09-12 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R updated YARN-7757:

Fix Version/s: (was: YARN-3409)
   3.2.0

> Refactor NodeLabelsProvider to be more generic and reusable for node 
> attributes providers
> -
>
> Key: YARN-7757
> URL: https://issues.apache.org/jira/browse/YARN-7757
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Blocker
> Fix For: 3.2.0
>
> Attachments: YARN-7757-YARN-3409.001.patch, 
> YARN-7757-YARN-3409.002.patch, YARN-7757-YARN-3409.003.patch, 
> YARN-7757-YARN-3409.004.patch, YARN-7757-YARN-3409.005.patch, 
> YARN-7757-YARN-3409.006.patch, 
> nodeLabelsProvider_refactor_class_hierarchy.pdf, 
> nodeLabelsProvider_refactor_v2.pdf, nodeLabelsProvider_refactor_v3.pdf
>
>
> Propose to do refactor on {{NodeLabelsProvider}}, 
> {{AbstractNodeLabelsProvider}} to be more generic, so node attributes 
> providers can reuse these interface/abstract classes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-7031) Support distributed node attributes

2018-09-12 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R updated YARN-7031:

Fix Version/s: (was: YARN-3409)
   3.2.0

> Support distributed node attributes
> ---
>
> Key: YARN-7031
> URL: https://issues.apache.org/jira/browse/YARN-7031
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: Distributed node attributes v1.pdf, 
> YARN-7031-YARN-3409.001.patch
>
>
> Allow nodemanagers to push its attributes to RM



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-6858) Attribute Manager to store and provide node attributes in RM

2018-09-12 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-6858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R updated YARN-6858:

Fix Version/s: (was: YARN-3409)
   3.2.0

> Attribute Manager to store and provide node attributes in RM
> 
>
> Key: YARN-6858
> URL: https://issues.apache.org/jira/browse/YARN-6858
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: api, capacityscheduler, client
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: YARN-6858-YARN-3409.001.patch, 
> YARN-6858-YARN-3409.002.patch, YARN-6858-YARN-3409.003.patch, 
> YARN-6858-YARN-3409.004.patch, YARN-6858-YARN-3409.005.patch, 
> YARN-6858-YARN-3409.006.patch, YARN-6858-YARN-3409.007.patch, 
> YARN-6858-YARN-3409.008.patch, YARN-6858-YARN-3409.009.patch
>
>
> Similar to CommonNodeLabelsManager we need to have a centralized manager for 
> Node Attributes too.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-6856) Support CLI for Node Attributes Mapping

2018-09-12 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-6856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R updated YARN-6856:

Fix Version/s: (was: YARN-3409)
   3.2.0

> Support CLI for Node Attributes Mapping
> ---
>
> Key: YARN-6856
> URL: https://issues.apache.org/jira/browse/YARN-6856
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: api, capacityscheduler, client
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: YARN-6856-YARN-3409.001.patch, 
> YARN-6856-YARN-3409.002.patch, YARN-6856-YARN-3409.004.patch, 
> YARN-6856-yarn-3409.003.patch, YARN-6856-yarn-3409.004.patch
>
>
> This focuses on the new CLI for the mapping of Node Attributes



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-6855) CLI Proto Modifications to support Node Attributes

2018-09-12 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-6855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R updated YARN-6855:

Fix Version/s: (was: YARN-3409)
   3.2.0

> CLI Proto Modifications to support Node Attributes
> --
>
> Key: YARN-6855
> URL: https://issues.apache.org/jira/browse/YARN-6855
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: api, capacityscheduler, client
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
>Priority: Major
>  Labels: node-attributes
> Fix For: 3.2.0
>
> Attachments: YARN-6855-YARN-3409.001.patch, 
> YARN-6855-YARN-3409.002.patch, YARN-6855-YARN-3409.003.patch, 
> YARN-6855-YARN-3409.004.patch, YARN-6855-YARN-3409.005.patch, 
> YARN-6855-YARN-3409.006.patch, YARN-6855-yarn-3409.006.patch
>
>
> This jira focuses only on the proto modifications required for the CLI



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-7865) Node attributes documentation

2018-08-30 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598053#comment-16598053
 ] 

Naganarasimha G R commented on YARN-7865:
-

Thanks [~cheersyang] for the valuable comments, 

Have addressed it in the latest patch, please check ! 

 

> Node attributes documentation
> -
>
> Key: YARN-7865
> URL: https://issues.apache.org/jira/browse/YARN-7865
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: documentation
>Reporter: Weiwei Yang
>Assignee: Naganarasimha G R
>Priority: Major
> Attachments: NodeAttributes.html, YARN-7865-YARN-3409.001.patch, 
> YARN-7865-YARN-3409.002.patch, YARN-7865-YARN-3409.003.patch, 
> YARN-7865-YARN-3409.004.patch, YARN-7865-YARN-3409.005.patch
>
>
> We need proper docs to introduce how to enable node-attributes how to 
> configure providers, how to specify script paths, arguments in configuration, 
> what should be the proper permission of the script and who will run the 
> script. Also it would be good to add more info to the description of the 
> configuration properties.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-7865) Node attributes documentation

2018-08-30 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R updated YARN-7865:

Attachment: YARN-7865-YARN-3409.005.patch

> Node attributes documentation
> -
>
> Key: YARN-7865
> URL: https://issues.apache.org/jira/browse/YARN-7865
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: documentation
>Reporter: Weiwei Yang
>Assignee: Naganarasimha G R
>Priority: Major
> Attachments: NodeAttributes.html, YARN-7865-YARN-3409.001.patch, 
> YARN-7865-YARN-3409.002.patch, YARN-7865-YARN-3409.003.patch, 
> YARN-7865-YARN-3409.004.patch, YARN-7865-YARN-3409.005.patch
>
>
> We need proper docs to introduce how to enable node-attributes how to 
> configure providers, how to specify script paths, arguments in configuration, 
> what should be the proper permission of the script and who will run the 
> script. Also it would be good to add more info to the description of the 
> configuration properties.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-7865) Node attributes documentation

2018-08-30 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R updated YARN-7865:

Attachment: (was: NodeAttributes.html)

> Node attributes documentation
> -
>
> Key: YARN-7865
> URL: https://issues.apache.org/jira/browse/YARN-7865
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: documentation
>Reporter: Weiwei Yang
>Assignee: Naganarasimha G R
>Priority: Major
> Attachments: NodeAttributes.html, YARN-7865-YARN-3409.001.patch, 
> YARN-7865-YARN-3409.002.patch, YARN-7865-YARN-3409.003.patch, 
> YARN-7865-YARN-3409.004.patch
>
>
> We need proper docs to introduce how to enable node-attributes how to 
> configure providers, how to specify script paths, arguments in configuration, 
> what should be the proper permission of the script and who will run the 
> script. Also it would be good to add more info to the description of the 
> configuration properties.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-7865) Node attributes documentation

2018-08-30 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R updated YARN-7865:

Attachment: NodeAttributes.html

> Node attributes documentation
> -
>
> Key: YARN-7865
> URL: https://issues.apache.org/jira/browse/YARN-7865
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: documentation
>Reporter: Weiwei Yang
>Assignee: Naganarasimha G R
>Priority: Major
> Attachments: NodeAttributes.html, YARN-7865-YARN-3409.001.patch, 
> YARN-7865-YARN-3409.002.patch, YARN-7865-YARN-3409.003.patch, 
> YARN-7865-YARN-3409.004.patch
>
>
> We need proper docs to introduce how to enable node-attributes how to 
> configure providers, how to specify script paths, arguments in configuration, 
> what should be the proper permission of the script and who will run the 
> script. Also it would be good to add more info to the description of the 
> configuration properties.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-8102) Retrospect on having enable and disable flag for Node Attribute

2018-08-29 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596967#comment-16596967
 ] 

Naganarasimha G R commented on YARN-8102:
-

Hi [~bibinchundatt] & [~sunilg],

I think we had sufficient discussion on this, IMHO its not required and it can 
be safely configured to local host's tmp where in most of the clusters will 
have access else we can choose relative path to  *_"hadoop.tmp.dir"_*  which is 
what was also used in case of ATSv1 timeline store. I would prefer here to 
avoid any kind of new configuration being introduced. If agree then we can go 
ahead and do the required above changes and close this jira.

cc [~cheersyang]

> Retrospect on having enable and disable flag for Node Attribute
> ---
>
> Key: YARN-8102
> URL: https://issues.apache.org/jira/browse/YARN-8102
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Major
>
> Currently node attribute feature is by default enabled. We have to revisit on 
> the same.
> Enabling by default means will try to create store for all cluster 
> installation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-7865) Node attributes documentation

2018-08-29 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596963#comment-16596963
 ] 

Naganarasimha G R commented on YARN-7865:
-

Thanks [~sunilg], for the review,

Please find the attached patch addressing your comments.

> Node attributes documentation
> -
>
> Key: YARN-7865
> URL: https://issues.apache.org/jira/browse/YARN-7865
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: documentation
>Reporter: Weiwei Yang
>Assignee: Naganarasimha G R
>Priority: Major
> Attachments: NodeAttributes.html, YARN-7865-YARN-3409.001.patch, 
> YARN-7865-YARN-3409.002.patch, YARN-7865-YARN-3409.003.patch, 
> YARN-7865-YARN-3409.004.patch
>
>
> We need proper docs to introduce how to enable node-attributes how to 
> configure providers, how to specify script paths, arguments in configuration, 
> what should be the proper permission of the script and who will run the 
> script. Also it would be good to add more info to the description of the 
> configuration properties.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-7865) Node attributes documentation

2018-08-29 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R updated YARN-7865:

Attachment: YARN-7865-YARN-3409.004.patch

> Node attributes documentation
> -
>
> Key: YARN-7865
> URL: https://issues.apache.org/jira/browse/YARN-7865
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: documentation
>Reporter: Weiwei Yang
>Assignee: Naganarasimha G R
>Priority: Major
> Attachments: NodeAttributes.html, YARN-7865-YARN-3409.001.patch, 
> YARN-7865-YARN-3409.002.patch, YARN-7865-YARN-3409.003.patch, 
> YARN-7865-YARN-3409.004.patch
>
>
> We need proper docs to introduce how to enable node-attributes how to 
> configure providers, how to specify script paths, arguments in configuration, 
> what should be the proper permission of the script and who will run the 
> script. Also it would be good to add more info to the description of the 
> configuration properties.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-7865) Node attributes documentation

2018-08-29 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R updated YARN-7865:

Attachment: (was: NodeAttributes.html)

> Node attributes documentation
> -
>
> Key: YARN-7865
> URL: https://issues.apache.org/jira/browse/YARN-7865
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: documentation
>Reporter: Weiwei Yang
>Assignee: Naganarasimha G R
>Priority: Major
> Attachments: NodeAttributes.html, YARN-7865-YARN-3409.001.patch, 
> YARN-7865-YARN-3409.002.patch, YARN-7865-YARN-3409.003.patch
>
>
> We need proper docs to introduce how to enable node-attributes how to 
> configure providers, how to specify script paths, arguments in configuration, 
> what should be the proper permission of the script and who will run the 
> script. Also it would be good to add more info to the description of the 
> configuration properties.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-7865) Node attributes documentation

2018-08-29 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R updated YARN-7865:

Attachment: NodeAttributes.html

> Node attributes documentation
> -
>
> Key: YARN-7865
> URL: https://issues.apache.org/jira/browse/YARN-7865
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: documentation
>Reporter: Weiwei Yang
>Assignee: Naganarasimha G R
>Priority: Major
> Attachments: NodeAttributes.html, YARN-7865-YARN-3409.001.patch, 
> YARN-7865-YARN-3409.002.patch, YARN-7865-YARN-3409.003.patch
>
>
> We need proper docs to introduce how to enable node-attributes how to 
> configure providers, how to specify script paths, arguments in configuration, 
> what should be the proper permission of the script and who will run the 
> script. Also it would be good to add more info to the description of the 
> configuration properties.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-3409) Support Node Attribute functionality

2018-08-29 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R updated YARN-3409:

Attachment: Node-Attributes-Requirements-Design-doc_v2.pdf

> Support Node Attribute functionality
> 
>
> Key: YARN-3409
> URL: https://issues.apache.org/jira/browse/YARN-3409
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: api, client, RM
>Reporter: Wangda Tan
>Assignee: Naganarasimha G R
>Priority: Major
> Attachments: 3409-apiChanges_v2.pdf (4).pdf, 
> Constraint-Node-Labels-Requirements-Design-doc_v1.pdf, 
> Node-Attributes-Requirements-Design-doc_v2.pdf, YARN-3409.WIP.001.patch
>
>
> Specify only one label for each node (IAW, partition a cluster) is a way to 
> determinate how resources of a special set of nodes could be shared by a 
> group of entities (like teams, departments, etc.). Partitions of a cluster 
> has following characteristics:
> - Cluster divided to several disjoint sub clusters.
> - ACL/priority can apply on partition (Only market team / marke team has 
> priority to use the partition).
> - Percentage of capacities can apply on partition (Market team has 40% 
> minimum capacity and Dev team has 60% of minimum capacity of the partition).
> Attributes are orthogonal to partition, they’re describing features of node’s 
> hardware/software just for affinity. Some example of attributes:
> - glibc version
> - JDK version
> - Type of CPU (x86_64/i686)
> - Type of OS (windows, linux, etc.)
> With this, application can be able to ask for resource has (glibc.version >= 
> 2.20 && JDK.version >= 8u20 && x86_64).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-7865) Node attributes documentation

2018-08-28 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595792#comment-16595792
 ] 

Naganarasimha G R commented on YARN-7865:
-

[~bibinchundatt],

Have attached a patch without all whitespace issue fixed  and the remaining are 
required for the mark up to detect as next paragraph. Please Commit without 
*WHITESPACE=FIX* option if no one has any more comments.

> Node attributes documentation
> -
>
> Key: YARN-7865
> URL: https://issues.apache.org/jira/browse/YARN-7865
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: documentation
>Reporter: Weiwei Yang
>Assignee: Naganarasimha G R
>Priority: Major
> Attachments: NodeAttributes.html, YARN-7865-YARN-3409.001.patch, 
> YARN-7865-YARN-3409.002.patch, YARN-7865-YARN-3409.003.patch
>
>
> We need proper docs to introduce how to enable node-attributes how to 
> configure providers, how to specify script paths, arguments in configuration, 
> what should be the proper permission of the script and who will run the 
> script. Also it would be good to add more info to the description of the 
> configuration properties.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-7865) Node attributes documentation

2018-08-28 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R updated YARN-7865:

Attachment: YARN-7865-YARN-3409.003.patch

> Node attributes documentation
> -
>
> Key: YARN-7865
> URL: https://issues.apache.org/jira/browse/YARN-7865
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: documentation
>Reporter: Weiwei Yang
>Assignee: Naganarasimha G R
>Priority: Major
> Attachments: NodeAttributes.html, YARN-7865-YARN-3409.001.patch, 
> YARN-7865-YARN-3409.002.patch, YARN-7865-YARN-3409.003.patch
>
>
> We need proper docs to introduce how to enable node-attributes how to 
> configure providers, how to specify script paths, arguments in configuration, 
> what should be the proper permission of the script and who will run the 
> script. Also it would be good to add more info to the description of the 
> configuration properties.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-7865) Node attributes documentation

2018-08-28 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595119#comment-16595119
 ] 

Naganarasimha G R commented on YARN-7865:
-

Thanks [~bibinchundatt] for the review comments and have addressed all except 
for the second point as we replace only attributes for the  prefix “nm.yarn.io”.

Also have addressed the white space and tab issues reported by checkstyle in 
the latest patch

> Node attributes documentation
> -
>
> Key: YARN-7865
> URL: https://issues.apache.org/jira/browse/YARN-7865
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: documentation
>Reporter: Weiwei Yang
>Assignee: Naganarasimha G R
>Priority: Major
> Attachments: NodeAttributes.html, YARN-7865-YARN-3409.001.patch, 
> YARN-7865-YARN-3409.002.patch
>
>
> We need proper docs to introduce how to enable node-attributes how to 
> configure providers, how to specify script paths, arguments in configuration, 
> what should be the proper permission of the script and who will run the 
> script. Also it would be good to add more info to the description of the 
> configuration properties.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-7865) Node attributes documentation

2018-08-28 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R updated YARN-7865:

Attachment: YARN-7865-YARN-3409.002.patch

> Node attributes documentation
> -
>
> Key: YARN-7865
> URL: https://issues.apache.org/jira/browse/YARN-7865
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: documentation
>Reporter: Weiwei Yang
>Assignee: Naganarasimha G R
>Priority: Major
> Attachments: NodeAttributes.html, YARN-7865-YARN-3409.001.patch, 
> YARN-7865-YARN-3409.002.patch
>
>
> We need proper docs to introduce how to enable node-attributes how to 
> configure providers, how to specify script paths, arguments in configuration, 
> what should be the proper permission of the script and who will run the 
> script. Also it would be good to add more info to the description of the 
> configuration properties.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-7865) Node attributes documentation

2018-08-28 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R updated YARN-7865:

Attachment: NodeAttributes.html

> Node attributes documentation
> -
>
> Key: YARN-7865
> URL: https://issues.apache.org/jira/browse/YARN-7865
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: documentation
>Reporter: Weiwei Yang
>Assignee: Naganarasimha G R
>Priority: Major
> Attachments: NodeAttributes.html, YARN-7865-YARN-3409.001.patch
>
>
> We need proper docs to introduce how to enable node-attributes how to 
> configure providers, how to specify script paths, arguments in configuration, 
> what should be the proper permission of the script and who will run the 
> script. Also it would be good to add more info to the description of the 
> configuration properties.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-7865) Node attributes documentation

2018-08-28 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R updated YARN-7865:

Attachment: (was: NodeAttributes.html)

> Node attributes documentation
> -
>
> Key: YARN-7865
> URL: https://issues.apache.org/jira/browse/YARN-7865
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: documentation
>Reporter: Weiwei Yang
>Assignee: Naganarasimha G R
>Priority: Major
> Attachments: YARN-7865-YARN-3409.001.patch
>
>
> We need proper docs to introduce how to enable node-attributes how to 
> configure providers, how to specify script paths, arguments in configuration, 
> what should be the proper permission of the script and who will run the 
> script. Also it would be good to add more info to the description of the 
> configuration properties.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-7865) Node attributes documentation

2018-08-27 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594540#comment-16594540
 ] 

Naganarasimha G R commented on YARN-7865:
-

[~sunilg], [~cheersyang] & [~bibinchundatt],

Can all of you have a look at the documentation and share if any modifications 
required ?

> Node attributes documentation
> -
>
> Key: YARN-7865
> URL: https://issues.apache.org/jira/browse/YARN-7865
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: documentation
>Reporter: Weiwei Yang
>Assignee: Naganarasimha G R
>Priority: Major
> Attachments: NodeAttributes.html, YARN-7865-YARN-3409.001.patch
>
>
> We need proper docs to introduce how to enable node-attributes how to 
> configure providers, how to specify script paths, arguments in configuration, 
> what should be the proper permission of the script and who will run the 
> script. Also it would be good to add more info to the description of the 
> configuration properties.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-7865) Node attributes documentation

2018-08-27 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R updated YARN-7865:

Attachment: YARN-7865-YARN-3409.001.patch

> Node attributes documentation
> -
>
> Key: YARN-7865
> URL: https://issues.apache.org/jira/browse/YARN-7865
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: documentation
>Reporter: Weiwei Yang
>Assignee: Naganarasimha G R
>Priority: Major
> Attachments: NodeAttributes.html, YARN-7865-YARN-3409.001.patch
>
>
> We need proper docs to introduce how to enable node-attributes how to 
> configure providers, how to specify script paths, arguments in configuration, 
> what should be the proper permission of the script and who will run the 
> script. Also it would be good to add more info to the description of the 
> configuration properties.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-7865) Node attributes documentation

2018-08-27 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R updated YARN-7865:

Attachment: NodeAttributes.html

> Node attributes documentation
> -
>
> Key: YARN-7865
> URL: https://issues.apache.org/jira/browse/YARN-7865
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: documentation
>Reporter: Weiwei Yang
>Assignee: Naganarasimha G R
>Priority: Major
> Attachments: NodeAttributes.html
>
>
> We need proper docs to introduce how to enable node-attributes how to 
> configure providers, how to specify script paths, arguments in configuration, 
> what should be the proper permission of the script and who will run the 
> script. Also it would be good to add more info to the description of the 
> configuration properties.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-7865) Node attributes documentation

2018-08-27 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R updated YARN-7865:

Attachment: (was: YARN-7865-YARN-3409.001.patch)

> Node attributes documentation
> -
>
> Key: YARN-7865
> URL: https://issues.apache.org/jira/browse/YARN-7865
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: documentation
>Reporter: Weiwei Yang
>Assignee: Naganarasimha G R
>Priority: Major
> Attachments: NodeAttributes.html
>
>
> We need proper docs to introduce how to enable node-attributes how to 
> configure providers, how to specify script paths, arguments in configuration, 
> what should be the proper permission of the script and who will run the 
> script. Also it would be good to add more info to the description of the 
> configuration properties.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-7865) Node attributes documentation

2018-08-27 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R updated YARN-7865:

Attachment: (was: YARN-8721-YARN-3409.001.patch)

> Node attributes documentation
> -
>
> Key: YARN-7865
> URL: https://issues.apache.org/jira/browse/YARN-7865
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: documentation
>Reporter: Weiwei Yang
>Assignee: Naganarasimha G R
>Priority: Major
> Attachments: YARN-7865-YARN-3409.001.patch
>
>
> We need proper docs to introduce how to enable node-attributes how to 
> configure providers, how to specify script paths, arguments in configuration, 
> what should be the proper permission of the script and who will run the 
> script. Also it would be good to add more info to the description of the 
> configuration properties.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-7865) Node attributes documentation

2018-08-27 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R updated YARN-7865:

Attachment: YARN-7865-YARN-3409.001.patch

> Node attributes documentation
> -
>
> Key: YARN-7865
> URL: https://issues.apache.org/jira/browse/YARN-7865
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: documentation
>Reporter: Weiwei Yang
>Assignee: Naganarasimha G R
>Priority: Major
> Attachments: YARN-7865-YARN-3409.001.patch
>
>
> We need proper docs to introduce how to enable node-attributes how to 
> configure providers, how to specify script paths, arguments in configuration, 
> what should be the proper permission of the script and who will run the 
> script. Also it would be good to add more info to the description of the 
> configuration properties.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-7865) Node attributes documentation

2018-08-27 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R updated YARN-7865:

Attachment: YARN-8721-YARN-3409.001.patch

> Node attributes documentation
> -
>
> Key: YARN-7865
> URL: https://issues.apache.org/jira/browse/YARN-7865
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: documentation
>Reporter: Weiwei Yang
>Assignee: Naganarasimha G R
>Priority: Major
> Attachments: YARN-7865-YARN-3409.001.patch
>
>
> We need proper docs to introduce how to enable node-attributes how to 
> configure providers, how to specify script paths, arguments in configuration, 
> what should be the proper permission of the script and who will run the 
> script. Also it would be good to add more info to the description of the 
> configuration properties.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-8721) Scheduler should accept nodes which doesnt have an attribute when NodeAttributeType.NE is used

2018-08-27 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594432#comment-16594432
 ] 

Naganarasimha G R commented on YARN-8721:
-

[~sunilg], IIRC idea for the boolean was not to support different type but to 
have value as empty string and have expression to support "exist" and "not 
exist" operator. I am fine with achieving it after merge but my concern is 
whether others like Microsoft will agree during merge!  

> Scheduler should accept nodes which doesnt have an attribute when 
> NodeAttributeType.NE is used
> --
>
> Key: YARN-8721
> URL: https://issues.apache.org/jira/browse/YARN-8721
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Sunil Govindan
>Assignee: Sunil Govindan
>Priority: Major
> Attachments: YARN-8721-YARN-3409.001.patch, 
> YARN-8721-YARN-3409.002.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-8721) Scheduler should accept nodes which doesnt have an attribute when NodeAttributeType.NE is used

2018-08-27 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594248#comment-16594248
 ] 

Naganarasimha G R commented on YARN-8721:
-

Hi [~sunilg],

          Just had a overview and the patch seems to be fine, but one query i 
had was to how are we supporting Boolean type attributes ? i.e. we are 
currently supporting EQ( =)  and NE ( != ) only but during requirement 
discussion there were lot of use cases(from Microsoft) where in they wanted to 
use exists not exists kind of  operation instead of value  based comparison. I 
think this might pop up during merge. Thoughts?

> Scheduler should accept nodes which doesnt have an attribute when 
> NodeAttributeType.NE is used
> --
>
> Key: YARN-8721
> URL: https://issues.apache.org/jira/browse/YARN-8721
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Sunil Govindan
>Assignee: Sunil Govindan
>Priority: Major
> Attachments: YARN-8721-YARN-3409.001.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Issue Comment Deleted] (YARN-8721) Scheduler should accept nodes which doesnt have an attribute when NodeAttributeType.NE is used

2018-08-27 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-8721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R updated YARN-8721:

Comment: was deleted

(was: Hi [~sunilg],

There are some checkstyle issues in  YARN-8718 , can that be addressed along 
with this patch itself ?)

> Scheduler should accept nodes which doesnt have an attribute when 
> NodeAttributeType.NE is used
> --
>
> Key: YARN-8721
> URL: https://issues.apache.org/jira/browse/YARN-8721
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Sunil Govindan
>Assignee: Sunil Govindan
>Priority: Major
> Attachments: YARN-8721-YARN-3409.001.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-8721) Scheduler should accept nodes which doesnt have an attribute when NodeAttributeType.NE is used

2018-08-27 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594234#comment-16594234
 ] 

Naganarasimha G R commented on YARN-8721:
-

Hi [~sunilg],

There are some checkstyle issues in  YARN-8718 , can that be addressed along 
with this patch itself ?

> Scheduler should accept nodes which doesnt have an attribute when 
> NodeAttributeType.NE is used
> --
>
> Key: YARN-8721
> URL: https://issues.apache.org/jira/browse/YARN-8721
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Sunil Govindan
>Assignee: Sunil Govindan
>Priority: Major
> Attachments: YARN-8721-YARN-3409.001.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-7863) Modify placement constraints to support node attributes

2018-08-26 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593114#comment-16593114
 ] 

Naganarasimha G R commented on YARN-7863:
-

Yes [~sunilg], as both the issues are related and we would require a test case 
for the weiwei's scenario too, hence i agree can be covered in other jira. 

Going ahead and committing the patch !

> Modify placement constraints to support node attributes
> ---
>
> Key: YARN-7863
> URL: https://issues.apache.org/jira/browse/YARN-7863
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Sunil Govindan
>Assignee: Sunil Govindan
>Priority: Major
> Attachments: YARN-7863-YARN-3409.002.patch, 
> YARN-7863-YARN-3409.003.patch, YARN-7863-YARN-3409.004.patch, 
> YARN-7863-YARN-3409.005.patch, YARN-7863-YARN-3409.006.patch, 
> YARN-7863-YARN-3409.007.patch, YARN-7863-YARN-3409.008.patch, 
> YARN-7863-YARN-3409.009.patch, YARN-7863-YARN-3409.010.patch, 
> YARN-7863-YARN-3409.011.patch, YARN-7863.v0.patch
>
>
> This Jira will track to *Modify existing placement constraints to support 
> node attributes.*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-7863) Modify placement constraints to support node attributes

2018-08-26 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593109#comment-16593109
 ] 

Naganarasimha G R edited comment on YARN-7863 at 8/27/18 1:58 AM:
--

Hi [~sunilg],

             As discussed some comments can be handled in a separate jira, but 
still i could see test case coverage is not to that extent there for scheduler 
side of code.  *PlacementConstraintsUtil* : "Evaluation of Placement expression 
containing Node Attributes".

Also i believe i think [~cheersyang]'s comment is also valid, can you take a 
stab at it ?

Lets conclude next one as the final patch so that i can go ahead and commit it 
to unblock the overall functionality! For all others lets raise separate jira's.

 


was (Author: naganarasimha):
Hi [~sunilg],

             As discussed some comments can be handled in a separate jira, but 
still i could see test case coverage is not to that extent there from scheduler 
side of code.  *PlacementConstraintsUtil* : "Evaluation of Placement expression 
containing Node Attributes".

Also i believe i think [~cheersyang]'s comment is also valid, can you take a 
stab at it ?

Lets conclude next one as the final patch so that i can go ahead and commit it 
to unblock the overall functionality! For all others lets raise separate jira's.

 

> Modify placement constraints to support node attributes
> ---
>
> Key: YARN-7863
> URL: https://issues.apache.org/jira/browse/YARN-7863
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Sunil Govindan
>Assignee: Sunil Govindan
>Priority: Major
> Attachments: YARN-7863-YARN-3409.002.patch, 
> YARN-7863-YARN-3409.003.patch, YARN-7863-YARN-3409.004.patch, 
> YARN-7863-YARN-3409.005.patch, YARN-7863-YARN-3409.006.patch, 
> YARN-7863-YARN-3409.007.patch, YARN-7863-YARN-3409.008.patch, 
> YARN-7863-YARN-3409.009.patch, YARN-7863-YARN-3409.010.patch, 
> YARN-7863-YARN-3409.011.patch, YARN-7863.v0.patch
>
>
> This Jira will track to *Modify existing placement constraints to support 
> node attributes.*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-7863) Modify placement constraints to support node attributes

2018-08-26 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593109#comment-16593109
 ] 

Naganarasimha G R commented on YARN-7863:
-

Hi [~sunilg],

             As discussed some comments can be handled in a separate jira, but 
still i could see test case coverage is not to that extent there from scheduler 
side of code.  *PlacementConstraintsUtil* : "Evaluation of Placement expression 
containing Node Attributes".

Also i believe i think [~cheersyang]'s comment is also valid, can you take a 
stab at it ?

Lets conclude next one as the final patch so that i can go ahead and commit it 
to unblock the overall functionality! For all others lets raise separate jira's.

 

> Modify placement constraints to support node attributes
> ---
>
> Key: YARN-7863
> URL: https://issues.apache.org/jira/browse/YARN-7863
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Sunil Govindan
>Assignee: Sunil Govindan
>Priority: Major
> Attachments: YARN-7863-YARN-3409.002.patch, 
> YARN-7863-YARN-3409.003.patch, YARN-7863-YARN-3409.004.patch, 
> YARN-7863-YARN-3409.005.patch, YARN-7863-YARN-3409.006.patch, 
> YARN-7863-YARN-3409.007.patch, YARN-7863-YARN-3409.008.patch, 
> YARN-7863-YARN-3409.009.patch, YARN-7863-YARN-3409.010.patch, 
> YARN-7863-YARN-3409.011.patch, YARN-7863.v0.patch
>
>
> This Jira will track to *Modify existing placement constraints to support 
> node attributes.*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-7863) Modify placement constraints to support node attributes

2018-08-26 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16592847#comment-16592847
 ] 

Naganarasimha G R commented on YARN-7863:
-

Hi [~sunilg], 

Latest patch does not seem to consider all my 
[comments|https://issues.apache.org/jira/browse/YARN-7863?focusedCommentId=16589559=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16589559]
 and also latest patch seems to have some checkstyle issues to be addressed. 
Can you please take a look at them?

 

> Modify placement constraints to support node attributes
> ---
>
> Key: YARN-7863
> URL: https://issues.apache.org/jira/browse/YARN-7863
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Sunil Govindan
>Assignee: Sunil Govindan
>Priority: Major
> Attachments: YARN-7863-YARN-3409.002.patch, 
> YARN-7863-YARN-3409.003.patch, YARN-7863-YARN-3409.004.patch, 
> YARN-7863-YARN-3409.005.patch, YARN-7863-YARN-3409.006.patch, 
> YARN-7863-YARN-3409.007.patch, YARN-7863-YARN-3409.008.patch, 
> YARN-7863-YARN-3409.009.patch, YARN-7863-YARN-3409.010.patch, 
> YARN-7863.v0.patch
>
>
> This Jira will track to *Modify existing placement constraints to support 
> node attributes.*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-7863) Modify placement constraints to support node attributes

2018-08-23 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590410#comment-16590410
 ] 

Naganarasimha G R commented on YARN-7863:
-

Thanks [~cheersyang] for the detailed clarification,
{quote} PCs are not associated with resource, so it's like an extra check after 
all other checks are done. Scheduler still calculates how much resource 
available in a partition for a given queue, assign resource from a node in this 
partition to a request, but if PC is not satisfied then the allocation proposal 
will be rejected. et
{quote}
Agree PC's are not directly involved with resource, But IIUC with the new API 
:_SchedulingRequest_ for allocation of containers has only PC as the way to 
specify partitions which is tied with resource . This is the reason i had my 
concerns with the OR. Let me go through the code with new API and understand 
whether Partition is handled for all scenarios. I was always under the 
impression that there will be a constraint explicitly for the Partition based 
on which resources are accounted ( USED, PENDING , RESERVED etc...). But if its 
internal to the constraint  which is also OR'd with others i am not completely 
sure its addressing .
{quote}Partition in PC is not ready, to be honest, I am not sure if everything 
is align with existing label-based scheduling. I suggested in YARN-8015 to open 
a separate task for further enhance that.
{quote}
Let me too try to understand the PC and new api modifications in detail and 
then share my feedback in that Jira and anyway its out of the scope for this 
jira.

 

 

> Modify placement constraints to support node attributes
> ---
>
> Key: YARN-7863
> URL: https://issues.apache.org/jira/browse/YARN-7863
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Sunil Govindan
>Assignee: Sunil Govindan
>Priority: Major
> Attachments: YARN-7863-YARN-3409.002.patch, 
> YARN-7863-YARN-3409.003.patch, YARN-7863-YARN-3409.004.patch, 
> YARN-7863-YARN-3409.005.patch, YARN-7863-YARN-3409.006.patch, 
> YARN-7863-YARN-3409.007.patch, YARN-7863-YARN-3409.008.patch, 
> YARN-7863.v0.patch
>
>
> This Jira will track to *Modify existing placement constraints to support 
> node attributes.*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-7863) Modify placement constraints to support node attributes

2018-08-22 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589559#comment-16589559
 ] 

Naganarasimha G R commented on YARN-7863:
-

Hi [~sunilg],

    few other nits in the patch :
 * NodeAttributesManagerImpl ln no 205: as i was mentioning there is both debug 
and info log here ,i think we can remove the debug log.
 * NodeAttributesManagerImpl ln no 212-226 : Here you are sending out complete 
update of the node collections, and not just the modified NM's only. There are 
multiple impacts due to this, in large cluster we are unnecessarily sending lot 
of updates to scheduler secondly removed attributes will not be captured in 
this way. Where in later is more important
 * NodeAttributesManagerImpl ln no 212-226 : In general idea earlier was to 
make use of the AttributeValue by the scheduler so that the converted value is 
stored and used for comparison . But if we have the flexibility later on to 
change the scheduler event which is being pushed from NAM to Schedulers then i 
am fine with the Event being sent out, else i would suggest to send the 
AttributeValue itself 
 * Test cases for AND and OR are covered ? Though i could see AND not 
declaratively covered in TestPlacementConstraintParser but better to cover with 
AND and OR explicitly
 * PlacementSpec ln no 51: typo "teh"

> Modify placement constraints to support node attributes
> ---
>
> Key: YARN-7863
> URL: https://issues.apache.org/jira/browse/YARN-7863
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Sunil Govindan
>Assignee: Sunil Govindan
>Priority: Major
> Attachments: YARN-7863-YARN-3409.002.patch, 
> YARN-7863-YARN-3409.003.patch, YARN-7863-YARN-3409.004.patch, 
> YARN-7863-YARN-3409.005.patch, YARN-7863-YARN-3409.006.patch, 
> YARN-7863-YARN-3409.007.patch, YARN-7863-YARN-3409.008.patch, 
> YARN-7863.v0.patch
>
>
> This Jira will track to *Modify existing placement constraints to support 
> node attributes.*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-7863) Modify placement constraints to support node attributes

2018-08-22 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589171#comment-16589171
 ] 

Naganarasimha G R commented on YARN-7863:
-

Thanks [~cheersyang], for some clarifications but at the same time was 
discussing with  Sunil too.
{quote} PC doesn't affect any logic how scheduler selects requests, it is still 
how it is handled now. A PC is simply checked twice when 1) creating an 
allocation proposal on a node and 2) at commit phase against a specific node.
{quote}
I am not specifying that changes in  this Jira alone introducing anything 
implications but my point was, earlier when partitions was introduced it was 
easy to determine pending resources per partition per queue in the earlier api 
using resource request. Now with the New api introduced with PC(not just from 
this Jira alone) there is no way to find out for a given queue how much pending 
resources are there in each partition it can access. This is because PC can 
have an Partition OR'd with Allocation TAG or with this Jira  OR'd with 
Attributes. Impact of this would be that Cluster admin will not be able to plan 
resources for the partition per queue. And also i am not able to envisage the 
scenario where in partition needs to be OR'd with Allocation tags or Attributes.
{quote}No, not necessarily. Just to keep the changes clean and incremental, we 
can allow this form for now. Because this is the spec we used for distributed 
shell. Since we don't have an --allocationTags argument. A "foo=3" is the only 
way to specify allocation tags right now.
{quote}
Agree, My bad but the reason i got confused is all the parsing logic for the  
Distributed shell's expression is present in  "hadoop-yarn-api" project which 
more so implies that this the API to be used by other apps too.
{quote}I want to reinforce the phase-target (for the merge), we want 
node-attributes can be integrated into PC and support simple operation "=" and 
"!=".
{quote}
Yes I concur with you on this, as long as we are able to clearly capture for 
others what kind of Java API needs to be used to specify Node attributes.

[~sunilg],

Could there be a simple test case or example which captures how to write a java 
api where in node attributes can be specified for a given SchedulingRequest  
i.e. without any of these expression DSL's ?

Also i could not see a test case where in CS is handling scheduling of 
containers with PC having attributes. So that  modifications in 
PlacementConstraintsUtil is tested.

 

 

 

> Modify placement constraints to support node attributes
> ---
>
> Key: YARN-7863
> URL: https://issues.apache.org/jira/browse/YARN-7863
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Sunil Govindan
>Assignee: Sunil Govindan
>Priority: Major
> Attachments: YARN-7863-YARN-3409.002.patch, 
> YARN-7863-YARN-3409.003.patch, YARN-7863-YARN-3409.004.patch, 
> YARN-7863-YARN-3409.005.patch, YARN-7863-YARN-3409.006.patch, 
> YARN-7863-YARN-3409.007.patch, YARN-7863-YARN-3409.008.patch, 
> YARN-7863.v0.patch
>
>
> This Jira will track to *Modify existing placement constraints to support 
> node attributes.*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-7863) Modify placement constraints to support node attributes

2018-08-22 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589075#comment-16589075
 ] 

Naganarasimha G R commented on YARN-7863:
-

Hi [~sunilg],

Now i am confused when we have everything in one single expression .
 * IIUC we can specify allocation tags, attribute expression  and partition 
label all in single expression ?
 * If its the case then how are we going select the outstanding/ pending 
requests for a given queue for a given partition, as there could be OR with 
allocation tags or attribute expression right ? I think this issue should be 
already existing when we have OR ing involved with allocation tags ?
 * Based on the test cases in 
TestPlacementConstraintParser.testParseNodeAttributeSpec, Allocation tags are 
created even if we want to specify attribute expression ?  ex : 
"xyz,in,rm.yarn.io/foo=true"  xyz is allocation tag ? If so users will be 
really confused what to pass and what not !
 * if i specify "IN" and have attribute expression as  
"xyz,in,rm.yarn.io/foo{color:#d04437}*!=*{color}true" is it valid ? In future 
when we come with more operators it would not make sense. I would suggest to go 
with "," as separator. 

I am not sure how many cases would be there where we want to specify all the 
constraints in the same expression but we are making the users life complex 
with having such a complex DSL. 

 

> Modify placement constraints to support node attributes
> ---
>
> Key: YARN-7863
> URL: https://issues.apache.org/jira/browse/YARN-7863
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Sunil Govindan
>Assignee: Sunil Govindan
>Priority: Major
> Attachments: YARN-7863-YARN-3409.002.patch, 
> YARN-7863-YARN-3409.003.patch, YARN-7863-YARN-3409.004.patch, 
> YARN-7863-YARN-3409.005.patch, YARN-7863-YARN-3409.006.patch, 
> YARN-7863-YARN-3409.007.patch, YARN-7863-YARN-3409.008.patch, 
> YARN-7863.v0.patch
>
>
> This Jira will track to *Modify existing placement constraints to support 
> node attributes.*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-7863) Modify placement constraints to support node attributes

2018-08-19 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16585329#comment-16585329
 ] 

Naganarasimha G R commented on YARN-7863:
-

Thanks [~sunilg], for the latest patch,

Some of the things [~cheersyang] mentioned i too had the same issues and had 
mentioned in my earlier comment  like : 
 *  _PlacementConstraintParser.NodeConstraintParser_  not  to store value (had 
captured 2nd point) . 
 * _getConstraintNameWithNameSpace()_ defaults the namespace (had captured 4th 
point) . 

And seems like many other comments like the following was not addressed or 
replied yet:
 * I replaced expression @  _TestPlacementConstraintParser ln no 481,_ with 
"OR(IN,python=true:NOTIN,java=1.8):IN,os!=centos:NOTIN,env=prod,dev" and i see 
exception as "_Invalid constraint expression 
OR(IN,python=true:NOTIN,java=1.8)_".  IIUC our branch does not support OR 
expressions yet, So does it requires a rebase to confirm the same ?
 * Distributed shell needs to support both specs for the allocation tags and 
attributes,  which i believe can only be achieved having AttributeSpec .  
Unless we have to seriously meet the deadlines then lets avoid it but if you 
already have it ready then why not go for it ?  
 * TestPlacementConstraintParser ln no 453 : testParseNodeAttributeSpec has 
attributeName1, attributeName2; which is unused
 * ResourceManager ln no 652-654 createNodeAttributesManager should either 
return NodeAttributesManagerImpl or take parameter as rmContext instead of type 
casting
 * findbugs seems to be existent from the previous patch.

 

I concur with most  of the [~cheersyang]'s comments except 
 * We need to have a info log or rather an audit to track the attributes being 
modified but nevertheless there is both info and a debug log of the same 
message @202 and 205 , but 222 can be avoided all together

 

*TestCases :*
 * I would love to have a DistributedShell test case with Node attributes in 
the placement Contstraints. As discussed if required we can take the DS changes 
from here to a separate patch to make this patch less bulky
 * And  & OR Expressions in the expressions
 * Test case for covering scheduling part of the Placement expression 
containing Node Attributes

 

Shall we have a quick huddle some time this week, so that we can finalize on 
these stuff and then go ahead ? 

 

 

> Modify placement constraints to support node attributes
> ---
>
> Key: YARN-7863
> URL: https://issues.apache.org/jira/browse/YARN-7863
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Sunil Govindan
>Assignee: Sunil Govindan
>Priority: Major
> Attachments: YARN-7863-YARN-3409.002.patch, 
> YARN-7863-YARN-3409.003.patch, YARN-7863-YARN-3409.004.patch, 
> YARN-7863-YARN-3409.005.patch, YARN-7863-YARN-3409.006.patch, 
> YARN-7863.v0.patch
>
>
> This Jira will track to *Modify existing placement constraints to support 
> node attributes.*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-7863) Modify placement constraints to support node attributes

2018-08-12 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577644#comment-16577644
 ] 

Naganarasimha G R commented on YARN-7863:
-

Thanks for the patch [~sunilg] and apologies for the delayed review, as i was 
not acquainted with Placement Constraint part of the code, i required some time 
in understanding the placement constraints and then in relation how you have 
modified it. But still could not figure out few of the stuff
 *  Not completely clear sure how the parsing happens for the *And* and *Or* 
cases which was mentioned by [~cheersyang] in his earlier comment  
"_-placement_spec 
OR(IN,python=true:NOTIN,java=1.8):IN,os!=centos:NOTIN,env=prod,dev"_
Just tried to replace the same in your test case but seems like it fails at 
PlacementConstraintParser.parseNodeAttributeExpression ln no 669. Whether its 
expected to fail or it should assume it as *And* and proceed?
 * In PlacementConstraintParser.NodeConstraintParser.parse() PlacementTargets 
target in ln no 399 seems to include keys also in val.  Was not sure why we 
need to store in this way, would it not be bettert only values here
 * Can we place both constraints related to attributes and allocation tags 
together in a single placement spec for distributed shell? As I understand the 
Placement spec API supports to take multiple type of placement constraints for 
a single resource/allocation request. We too should support it right ?
 * NodeConstraintParser.parse seems to assume that attributeNameSpace has 
"rm.yarn.io/" but IIRC we concluded that user need to specify this explicitly 
right else exception is thrown right ?

 

some minor comments
 * TestPlacementConstraintParser ln no 453 : testParseNodeAttributeSpec has 
attributeName1, attributeName2; which is unused
 * ResourceManager ln no 652-654 createNodeAttributesManager should either 
return NodeAttributesManagerImpl or take parameter as rmContext instead of type 
casting
 * Seems like build seems to have find bugs and test case failures. Can you 
please have a look at it ?

> Modify placement constraints to support node attributes
> ---
>
> Key: YARN-7863
> URL: https://issues.apache.org/jira/browse/YARN-7863
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Sunil Govindan
>Assignee: Sunil Govindan
>Priority: Major
> Attachments: YARN-7863-YARN-3409.002.patch, 
> YARN-7863-YARN-3409.003.patch, YARN-7863-YARN-3409.004.patch, 
> YARN-7863.v0.patch
>
>
> This Jira will track to *Modify existing placement constraints to support 
> node attributes.*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-8593) Add new RM web service endpoint to get cluster user info

2018-07-30 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561751#comment-16561751
 ] 

Naganarasimha G R commented on YARN-8593:
-

Hi [~akhilpb],

May be can you please share more info on why this api is required ? and is it 
good to share with what user RM is running ? Even if required is it not good to 
be known from admin CLI rather than other forms ?

 

> Add new RM web service endpoint to get cluster user info
> 
>
> Key: YARN-8593
> URL: https://issues.apache.org/jira/browse/YARN-8593
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Reporter: Akhil PB
>Assignee: Akhil PB
>Priority: Major
> Attachments: YARN-8593.001.patch, YARN-8593.002.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-7863) Modify placement constraints to support node attributes

2018-07-29 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561134#comment-16561134
 ] 

Naganarasimha G R commented on YARN-7863:
-

Hi [~sunilg], 

Patch seems to fail after rebase (tried locally too). Can you please take a 
look at it ?

> Modify placement constraints to support node attributes
> ---
>
> Key: YARN-7863
> URL: https://issues.apache.org/jira/browse/YARN-7863
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Sunil Govindan
>Assignee: Sunil Govindan
>Priority: Major
> Attachments: YARN-7863-YARN-3409.002.patch, 
> YARN-7863-YARN-3409.003.patch, YARN-7863.v0.patch
>
>
> This Jira will track to *Modify existing placement constraints to support 
> node attributes.*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Assigned] (YARN-7863) Modify placement constraints to support node attributes

2018-07-29 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R reassigned YARN-7863:
---

Assignee: Sunil Govindan  (was: Naganarasimha G R)

> Modify placement constraints to support node attributes
> ---
>
> Key: YARN-7863
> URL: https://issues.apache.org/jira/browse/YARN-7863
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Sunil Govindan
>Assignee: Sunil Govindan
>Priority: Major
> Attachments: YARN-7863-YARN-3409.002.patch, 
> YARN-7863-YARN-3409.003.patch, YARN-7863.v0.patch
>
>
> This Jira will track to *Modify existing placement constraints to support 
> node attributes.*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Assigned] (YARN-7863) Modify placement constraints to support node attributes

2018-07-29 Thread Naganarasimha G R (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R reassigned YARN-7863:
---

Assignee: Naganarasimha G R  (was: Sunil Govindan)

> Modify placement constraints to support node attributes
> ---
>
> Key: YARN-7863
> URL: https://issues.apache.org/jira/browse/YARN-7863
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Sunil Govindan
>Assignee: Naganarasimha G R
>Priority: Major
> Attachments: YARN-7863-YARN-3409.002.patch, 
> YARN-7863-YARN-3409.003.patch, YARN-7863.v0.patch
>
>
> This Jira will track to *Modify existing placement constraints to support 
> node attributes.*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-8574) Allow dot in attribute values

2018-07-28 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561015#comment-16561015
 ] 

Naganarasimha G R commented on YARN-8574:
-

Thanks [~sunilg], for rebase.  Overall approach looks good to me.

I will commit the patch shortly.

> Allow dot in attribute values 
> --
>
> Key: YARN-8574
> URL: https://issues.apache.org/jira/browse/YARN-8574
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Minor
> Attachments: YARN-8574-YARN-3409.001.patch, 
> YARN-8574-YARN-3409.002.patch
>
>
> Currently "." is considered as invalid value. Enable  the same;



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-8574) Allow dot in attribute values

2018-07-26 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559060#comment-16559060
 ] 

Naganarasimha G R edited comment on YARN-8574 at 7/27/18 12:11 AM:
---

Hi [~bibinchundatt],

You are correct i meant when we start using it as namespace later on. but 
anyway as we are supporting for internal its as well good to be supported for 
others too.

My only doubt now is how was it NOT failing for earlier mappings for prefix?

 Have manually triggered the build too...


was (Author: naganarasimha):
Hi [~bibinchundatt],

You are correct i meant when we start using it as namespace later on. but 
anyway as we are supporting for internal its as well good to be supported for 
others too.

My ownly doubt now is how was it not failing for earlier mappings for prefix?

 

> Allow dot in attribute values 
> --
>
> Key: YARN-8574
> URL: https://issues.apache.org/jira/browse/YARN-8574
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Minor
> Attachments: YARN-8574-YARN-3409.001.patch
>
>
> Currently "." is considered as invalid value. Enable  the same;



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-8574) Allow dot in attribute values

2018-07-26 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559060#comment-16559060
 ] 

Naganarasimha G R commented on YARN-8574:
-

Hi [~bibinchundatt],

You are correct i meant when we start using it as namespace later on. but 
anyway as we are supporting for internal its as well good to be supported for 
others too.

My ownly doubt now is how was it not failing for earlier mappings for prefix?

 

> Allow dot in attribute values 
> --
>
> Key: YARN-8574
> URL: https://issues.apache.org/jira/browse/YARN-8574
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Minor
> Attachments: YARN-8574-YARN-3409.001.patch
>
>
> Currently "." is considered as invalid value. Enable  the same;



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-8574) Allow dot in attribute values

2018-07-25 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555499#comment-16555499
 ] 

Naganarasimha G R commented on YARN-8574:
-

Thanks for the patch [~bibinchundatt], my bad almost had forgotten to work upon 
it. 

One concern i had is agree that value should be allowed with a dot but not the 
prefix, so for prefix if we are using the same pattern it qill be a problem

 

 

 

> Allow dot in attribute values 
> --
>
> Key: YARN-8574
> URL: https://issues.apache.org/jira/browse/YARN-8574
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Minor
> Attachments: YARN-8574-YARN-3409.001.patch
>
>
> Currently "." is considered as invalid value. Enable  the same;



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-7863) Modify placement constraints to support node attributes

2018-07-23 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16553691#comment-16553691
 ] 

Naganarasimha G R commented on YARN-7863:
-

Have done just a first pass review,
few higher level comments :
 # I think Distributed shell related modifications can be pulled into separate 
Jira (YARN-8289) as discussed earlier
 # I could not get in the meeting wrt to Attributes and Partitions/labels, IIUC 
in ResourceRequest we specify the label expression directly but in 
SchedulingRequest we can give it as part of PlacementConstraint ? And if so we 
club it with the attribute expression ? May be we need capture that explicitly 
over here.can you share a sample request with both of them available?
 # Attach the doc which captured our conclusions on the expression or capture 
the essense of it as Jira's descritpion

Other minor aspects:
 * PlacementConstraints.java ln 52- 53, better to use 
NodeAttribute.PREFIX_DISTRIBUTED & PREFIX_CENTRALIZED
 * PlacementConstraintParser ln 363-436 NodeConstraintsTokenizer seems to be 
not used anywhere ?
 * ResourceManager ln no 652-654 createNodeAttributesManager should either 
return NodeAttributesManagerImpl or take parameter as rmContext instead of type 
casting
 * .bowerrc ln no 3, might not be related to this patch ?
 * TestPlacementConstraintParser : captures a single test case , we need to 
capture more examples of capturing the positive and negative expression 
validation cases
 * TestPlacementConstraintParser ln no 457 : is this the approach which we 
finalised ? i thought its much more expressive like the document in  
PlacementConstraintParser.parsePlacementSpecForAttributes

As well change the status of the Jira to patch submitted so that jenkins build 
will be triggered.

 

> Modify placement constraints to support node attributes
> ---
>
> Key: YARN-7863
> URL: https://issues.apache.org/jira/browse/YARN-7863
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Sunil Govindan
>Assignee: Sunil Govindan
>Priority: Major
> Attachments: YARN-7863-YARN-3409.002.patch, YARN-7863.v0.patch
>
>
> This Jira will track to *Modify existing placement constraints to support 
> node attributes.*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-7863) Modify placement constraints to support node attributes

2018-07-22 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16552201#comment-16552201
 ] 

Naganarasimha G R commented on YARN-7863:
-

Hi [~sunilg],

While i am going through the patch, would you mind converting the status from 
Inprogress to patch submitted so that the jenkins build is run ?

 

 

> Modify placement constraints to support node attributes
> ---
>
> Key: YARN-7863
> URL: https://issues.apache.org/jira/browse/YARN-7863
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Sunil Govindan
>Assignee: Sunil Govindan
>Priority: Major
> Attachments: YARN-7863-YARN-3409.002.patch, YARN-7863.v0.patch
>
>
> This Jira will track to *Modify existing placement constraints to support 
> node attributes.*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-8480) Add boolean option for resources

2018-07-22 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16552196#comment-16552196
 ] 

Naganarasimha G R commented on YARN-8480:
-

Sorry for pitching in late, ran through the discussions in the Jira and I 
concur with [~leftnoteasy],

I do not see the need for providing 2 approaches to address the same scenario. 
Understand there would be some initial hickups in using Node Attributes in FS 
but i think that would be the better approach than having 2 different 
approaches for FS and CS to address the same scenario. If we wear the user's 
cap and see it would be good to have a single approach to address a scenario 
irrespective of the scheduler. If we start addressing scenario's differently 
for FS and CS then there will 2 parallel systems within YARN which is not 
desirable. 

[~templedf],[~snemeth], May be if you guys can share all the features which you 
want from Node Attributes to address then we can confirm to what extent we have 
already covered in Phase 1. As far as i see if its only exist and not exist 
then we do already cover.

 

> Add boolean option for resources
> 
>
> Key: YARN-8480
> URL: https://issues.apache.org/jira/browse/YARN-8480
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Daniel Templeton
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: YARN-8480.001.patch, YARN-8480.002.patch
>
>
> Make it possible to define a resource with a boolean value.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-7863) Modify placement constraints to support node attributes

2018-07-01 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16529118#comment-16529118
 ] 

Naganarasimha G R commented on YARN-7863:
-

[~sunilg],


Few queries before going into the detail ...
1. who is doing the validation of the resource request?
2. If there are any open resource request for specified attribute, do we allow 
removing of that attribute from Attribute Manager ?
3. Confused little with the logic (may be earlier code to yours), i was always 
of the opinion that the partition is the first to be compared and then with tag 
or attribute here it seems to first check for the allocation tag and then for 
partition and then attribute. should it not be like under a given parition 
check any nodes having allocation tags and/or attributes satisfying the user's 
request?


Some comments :
1. PlacementConstraintsUtil, ln no 145-156 : Scheduler node needs to be having 
map of NodeAttributeKey -> NodeAttribute value to make it O(1), as well this 
will more useful when we go other types and operator based comparison happens. 
Or to consider for current scenario atleast it would be better to override 
NodeAttribute's equals method.

 

 

> Modify placement constraints to support node attributes
> ---
>
> Key: YARN-7863
> URL: https://issues.apache.org/jira/browse/YARN-7863
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Sunil Govindan
>Assignee: Sunil Govindan
>Priority: Major
> Attachments: YARN-7863.v0.patch
>
>
> This Jira will track to *Modify existing placement constraints to support 
> node attributes.*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-7863) Modify placement constraints to support node attributes

2018-07-01 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16529111#comment-16529111
 ] 

Naganarasimha G R edited comment on YARN-7863 at 7/1/18 2:19 PM:
-

Thanks for the patch [~sunilg],  It gives a very good idea what you plan to do 
in scheduler for node attributes.  At the high level overview approach seems to 
be  fine. Hope you can change the status of the Jira to Patch available so that 
Jenkins run can happen. 

Also please modify the attached patch format so that it jenkins runs against 
YARN-3409 branch


was (Author: naganarasimha):
Thanks for the patch [~sunilg],  It gives a very good idea what you plan to do 
in scheduler for node attributes.  At the high level overview approach seems to 
be  fine. Hope you can change the status of the Jira to Patch available so that 
Jenkins run can happen. 

> Modify placement constraints to support node attributes
> ---
>
> Key: YARN-7863
> URL: https://issues.apache.org/jira/browse/YARN-7863
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Sunil Govindan
>Assignee: Sunil Govindan
>Priority: Major
> Attachments: YARN-7863.v0.patch
>
>
> This Jira will track to *Modify existing placement constraints to support 
> node attributes.*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-7863) Modify placement constraints to support node attributes

2018-07-01 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16529111#comment-16529111
 ] 

Naganarasimha G R commented on YARN-7863:
-

Thanks for the patch [~sunilg],  It gives a very good idea what you plan to do 
in scheduler for node attributes.  At the high level overview approach seems to 
be  fine. Hope you can change the status of the Jira to Patch available so that 
Jenkins run can happen. 

> Modify placement constraints to support node attributes
> ---
>
> Key: YARN-7863
> URL: https://issues.apache.org/jira/browse/YARN-7863
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Sunil Govindan
>Assignee: Sunil Govindan
>Priority: Major
> Attachments: YARN-7863.v0.patch
>
>
> This Jira will track to *Modify existing placement constraints to support 
> node attributes.*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-8103) Add CLI interface to query node attributes

2018-06-27 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525758#comment-16525758
 ] 

Naganarasimha G R commented on YARN-8103:
-

Thanks for the contribution [~bibinchundatt] and additional review by 
[~cheersyang]. I have committed the patch to branch YARN-3409. 

> Add CLI interface to  query node attributes
> ---
>
> Key: YARN-8103
> URL: https://issues.apache.org/jira/browse/YARN-8103
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Major
> Attachments: YARN-8103-YARN-3409.001.patch, 
> YARN-8103-YARN-3409.002.patch, YARN-8103-YARN-3409.003.patch, 
> YARN-8103-YARN-3409.004.patch, YARN-8103-YARN-3409.005.patch, 
> YARN-8103-YARN-3409.006.patch, YARN-8103-YARN-3409.WIP.patch
>
>
> YARN-8100 will add API interface for querying the attributes. CLI interface 
> for querying node attributes for each nodes and list all attributes in 
> cluster.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-8103) Add CLI interface to query node attributes

2018-06-26 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16524084#comment-16524084
 ] 

Naganarasimha G R commented on YARN-8103:
-

Thanks for the updated patch [~bibinchundatt],  latest patch LGTM. If no 
further comments from [~cheersyang] and [~sunil.gov...@gmail.com], will go 
ahead and commit tomorrow EOD...

As discussed sorting based on need can be handled as additional improvement 
task.

> Add CLI interface to  query node attributes
> ---
>
> Key: YARN-8103
> URL: https://issues.apache.org/jira/browse/YARN-8103
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Major
> Attachments: YARN-8103-YARN-3409.001.patch, 
> YARN-8103-YARN-3409.002.patch, YARN-8103-YARN-3409.003.patch, 
> YARN-8103-YARN-3409.004.patch, YARN-8103-YARN-3409.005.patch, 
> YARN-8103-YARN-3409.006.patch, YARN-8103-YARN-3409.WIP.patch
>
>
> YARN-8100 will add API interface for querying the attributes. CLI interface 
> for querying node attributes for each nodes and list all attributes in 
> cluster.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-8103) Add CLI interface to query node attributes

2018-06-22 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16520880#comment-16520880
 ] 

Naganarasimha G R commented on YARN-8103:
-

Thanks for the patch [~bibinchundatt], Sorry my bad had done the validation 
wrong in "equals" and everything else is fine except for checkstyle issues 
having some unused imports. Also one small suggestion. would it be good to sort 
the attribute listing based one the attribute prefix/ name and listing it ? (in 
all places, nodecli, cluster cli and nodeattributecli).

Hope others can also take a look as its almost ready to go in ?

> Add CLI interface to  query node attributes
> ---
>
> Key: YARN-8103
> URL: https://issues.apache.org/jira/browse/YARN-8103
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Major
> Attachments: YARN-8103-YARN-3409.001.patch, 
> YARN-8103-YARN-3409.002.patch, YARN-8103-YARN-3409.003.patch, 
> YARN-8103-YARN-3409.004.patch, YARN-8103-YARN-3409.005.patch, 
> YARN-8103-YARN-3409.WIP.patch
>
>
> YARN-8100 will add API interface for querying the attributes. CLI interface 
> for querying node attributes for each nodes and list all attributes in 
> cluster.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-8103) Add CLI interface to query node attributes

2018-06-20 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16517924#comment-16517924
 ] 

Naganarasimha G R commented on YARN-8103:
-

IMHO, its better to be in multiple lines as it will be more readable and also 
when value comes in to picture it might be of any length. _"1-3 attributes per 
node"_ : cannot be pre determined based on use case different systems might 
have different number of attributes per node and also depends on how much 
system internal attributes we are coming up with. 

And obviously in the demo we will be using few attributes per node so its 
better to discuss in this Jira than waiting for the demo.

 

> Add CLI interface to  query node attributes
> ---
>
> Key: YARN-8103
> URL: https://issues.apache.org/jira/browse/YARN-8103
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Major
> Attachments: YARN-8103-YARN-3409.001.patch, 
> YARN-8103-YARN-3409.002.patch, YARN-8103-YARN-3409.003.patch, 
> YARN-8103-YARN-3409.004.patch, YARN-8103-YARN-3409.WIP.patch
>
>
> YARN-8100 will add API interface for querying the attributes. CLI interface 
> for querying node attributes for each nodes and list all attributes in 
> cluster.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-8103) Add CLI interface to query node attributes

2018-06-18 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16516100#comment-16516100
 ] 

Naganarasimha G R commented on YARN-8103:
-

Thanks for the patch [~bibinchundatt]
Some of my comments seems to be still not addressed:
 *  NodeAttributesCLI ln no 187: i think its better to use null here instead of 
"handler" for better readability
 *  TestNodeAttributesCLI ln no 405: testListAttributes is encapsulating 
NodesToAttributes tests too, may be it can captured as a different case
 *  NodeCLI ln no 317 : IMO it would better to wrap each attribute in a new 
line ?

And some of the [~cheersyang] comments needs to be discussed (though i am not 
more towards numeric form for to but we can discuss other better options)
and this comment needs to be handled
 * line 588: buildNodeLabelsMapFromStr -> buildNodeAttributesListFromStr

 

Apart from this many of the Checkstyle issues still seems to be valid and needs 
to be addressed.

> Add CLI interface to  query node attributes
> ---
>
> Key: YARN-8103
> URL: https://issues.apache.org/jira/browse/YARN-8103
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Major
> Attachments: YARN-8103-YARN-3409.001.patch, 
> YARN-8103-YARN-3409.002.patch, YARN-8103-YARN-3409.003.patch, 
> YARN-8103-YARN-3409.WIP.patch
>
>
> YARN-8100 will add API interface for querying the attributes. CLI interface 
> for querying node attributes for each nodes and list all attributes in 
> cluster.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-8103) Add CLI interface to query node attributes

2018-06-04 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501238#comment-16501238
 ] 

Naganarasimha G R edited comment on YARN-8103 at 6/5/18 3:53 AM:
-

Thanks for the patch and apologies for the delayed response.
 Major Comments :
  
 * I agree with the approach of not storing and updating the attributes in 
RMNode and instead request NodeAttributesManager to share the information. But 
was also wondering whether the API(RMNodeImpl.getAllNodeAttributes) is useful 
based on current scenarios ? All the callers are eventually converting it into 
a set of attributes and utilizing it. So i would prefer to change the api to 
just return the set of attributes applicable on a node and when needed let the 
caller take care of sorting based on the prefix( which is anyway not a current 
scenario)

Few other comments:
 * hadoop-yarn/bin/yarn ln no 58: i think "client" was missing from earlier 
which we need to add it
 * NodeAttributesCLI ln no 195: i think its better to use null here instead of 
"handler" for better readability
 * NodeAttributesCLI ln no 88,96 : unused variables
 * TestNodeAttributesCLI ln no 405: testListAttributes is encapsulating 
NodesToAttributes tests too, may be it can captured as a different case
 * NodeAttributesCLI ln no 355 : HashSet => HashSet
 * NodeAttributesCLI ln no 394 : HashSet => HashSet
 * NodeCLI ln no 349: formatNodeAttribute : instead of having new formatting 
here why not update NodeAttributePBImpl.toString ?
 * NodeCLI ln no 317 : IMO it would better to wrap each attribute in a new line 
?
 * TestClusterCLI : does not capture any case for listing of attributes 
(basically need to capture multiline)
 * BuilderUtils ln no 211 : space is required before Set

Some of the findbugs and checkstyle seems to be valid can you have a look at 
them ?

Kudos NodeAttributesCLI has been handled well !


was (Author: naganarasimha):
Thanks for the patch and apologies for the delayed response.
Major Comments :
 
 * I agree with the approach of not storing and updating the attributes in 
RMNode and instead request NodeAttributesManager to share the information. But 
was also wondering whether the API(RMNodeImpl.getAllNodeAttributes) is useful 
based on current scenarios ? All the callers are eventually converting it into 
a set of attributes and utilizing it. So i would prefer to change the api to 
just return the set of attributes applicable on a node and when needed let the 
caller take care of sorting based on the prefix( which is anyway not a current 
scenario)


Few other comments:
 * hadoop-yarn/bin/yarn ln no 58: i think "client" was missing from earlier 
which we need to add it
 *  NodeAttributesCLI ln no 195: i think its better to use null here instead of 
"handler" for better readability
 *  NodeAttributesCLI ln no 88,96 : unused variables
 *  TestNodeAttributesCLI ln no 405: testListAttributes is encapsulating 
NodesToAttributes tests too, may be it can captured as a different case

Some of the findbugs and checkstyle seems to be valid can you have a look at 
them ?

Kudos NodeAttributesCLI has been handled well !

> Add CLI interface to  query node attributes
> ---
>
> Key: YARN-8103
> URL: https://issues.apache.org/jira/browse/YARN-8103
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Major
> Attachments: YARN-8103-YARN-3409.001.patch, 
> YARN-8103-YARN-3409.002.patch, YARN-8103-YARN-3409.WIP.patch
>
>
> YARN-8100 will add API interface for querying the attributes. CLI interface 
> for querying node attributes for each nodes and list all attributes in 
> cluster.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-8103) Add CLI interface to query node attributes

2018-06-04 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501238#comment-16501238
 ] 

Naganarasimha G R commented on YARN-8103:
-

Thanks for the patch and apologies for the delayed response.
Major Comments :
 
 * I agree with the approach of not storing and updating the attributes in 
RMNode and instead request NodeAttributesManager to share the information. But 
was also wondering whether the API(RMNodeImpl.getAllNodeAttributes) is useful 
based on current scenarios ? All the callers are eventually converting it into 
a set of attributes and utilizing it. So i would prefer to change the api to 
just return the set of attributes applicable on a node and when needed let the 
caller take care of sorting based on the prefix( which is anyway not a current 
scenario)


Few other comments:
 * hadoop-yarn/bin/yarn ln no 58: i think "client" was missing from earlier 
which we need to add it
 *  NodeAttributesCLI ln no 195: i think its better to use null here instead of 
"handler" for better readability
 *  NodeAttributesCLI ln no 88,96 : unused variables
 *  TestNodeAttributesCLI ln no 405: testListAttributes is encapsulating 
NodesToAttributes tests too, may be it can captured as a different case

Some of the findbugs and checkstyle seems to be valid can you have a look at 
them ?

Kudos NodeAttributesCLI has been handled well !

> Add CLI interface to  query node attributes
> ---
>
> Key: YARN-8103
> URL: https://issues.apache.org/jira/browse/YARN-8103
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Major
> Attachments: YARN-8103-YARN-3409.001.patch, 
> YARN-8103-YARN-3409.002.patch, YARN-8103-YARN-3409.WIP.patch
>
>
> YARN-8100 will add API interface for querying the attributes. CLI interface 
> for querying node attributes for each nodes and list all attributes in 
> cluster.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-6578) Return container resource utilization from NM ContainerStatus call

2018-05-16 Thread Naganarasimha G R (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16477463#comment-16477463
 ] 

Naganarasimha G R edited comment on YARN-6578 at 5/16/18 1:56 PM:
--

[~cheersyang] & [~fly_in_gis], 

I went through the approach, but was wondering its not covering the scenario 
with resource profiles, if multiple resource types are supported then this 
would be like not a proper solution in terms of the Object 
{{ResourceUtilization}} being returned & changes in {{ContainerStatusPBImpl}} 
by storing only cpu, vmem and pmem as concatenated string.

Was wondering whether we need to return {{ResourceInformation}} and store 
resource type information and individual resource values as separate key value 
pairs in the {{ContainerStatusPBImpl.containerAttributes}}  map ?

cc [~sunil.gov...@gmail.com]


was (Author: naganarasimha):
[~cheersyang] & [~fly_in_gis], 

I went through the approach, but was wondering its not covering the scenario 
with resource profiles, if multiple resource types are supported then this 
would be like not a proper solution in terms of the Object 
{{ResourceUtilization}} being returned & changes in{{ContainerStatusPBImpl}} by 
storing only .

Was wondering whether we need to return {{ResourceInformation }}and store 
resource type information and individual resource values as separate key value 
pairs in the {{ContainerStatusPBImpl.containerAttributes}}  map ?

cc [~sunil.gov...@gmail.com]

> Return container resource utilization from NM ContainerStatus call
> --
>
> Key: YARN-6578
> URL: https://issues.apache.org/jira/browse/YARN-6578
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Yang Wang
>Assignee: Yang Wang
>Priority: Major
> Attachments: YARN-6578.001.patch, YARN-6578.002.patch, 
> YARN-6578.003.patch
>
>
> When the applicationMaster wants to change(increase/decrease) resources of an 
> allocated container, resource utilization is an important reference indicator 
> for decision making. So, when AM call NMClient.getContainerStatus, resource 
> utilization needs to be returned.
> Also container resource utilization need to report to RM to make better 
> scheduling.
> So put resource utilization in ContainerStatus.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-6578) Return container resource utilization from NM ContainerStatus call

2018-05-16 Thread Naganarasimha G R (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16477463#comment-16477463
 ] 

Naganarasimha G R commented on YARN-6578:
-

[~cheersyang] & [~fly_in_gis], 

I went through the approach, but was wondering its not covering the scenario 
with resource profiles, if multiple resource types are supported then this 
would be like not a proper solution in terms of the Object 
{{ResourceUtilization}} being returned & changes in{{ContainerStatusPBImpl}} by 
storing only .

Was wondering whether we need to return {{ResourceInformation }}and store 
resource type information and individual resource values as separate key value 
pairs in the {{ContainerStatusPBImpl.containerAttributes}}  map ?

cc [~sunil.gov...@gmail.com]

> Return container resource utilization from NM ContainerStatus call
> --
>
> Key: YARN-6578
> URL: https://issues.apache.org/jira/browse/YARN-6578
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Yang Wang
>Assignee: Yang Wang
>Priority: Major
> Attachments: YARN-6578.001.patch, YARN-6578.002.patch, 
> YARN-6578.003.patch
>
>
> When the applicationMaster wants to change(increase/decrease) resources of an 
> allocated container, resource utilization is an important reference indicator 
> for decision making. So, when AM call NMClient.getContainerStatus, resource 
> utilization needs to be returned.
> Also container resource utilization need to report to RM to make better 
> scheduling.
> So put resource utilization in ContainerStatus.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-8289) Modify distributedshell to support Node Attributes

2018-05-15 Thread Naganarasimha G R (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-8289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16475361#comment-16475361
 ] 

Naganarasimha G R commented on YARN-8289:
-

[~sunil.gov...@gmail.com],  Given that you had already working on the scheduler 
patch and testing it already with DS shell, would you like to take this Jira 
up? 

> Modify distributedshell to support Node Attributes
> --
>
> Key: YARN-8289
> URL: https://issues.apache.org/jira/browse/YARN-8289
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: distributed-shell
>Affects Versions: YARN-3409
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
>Priority: Major
>
> Modifications required in Distributed shell to support NodeAttributes



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-8289) Modify distributedshell to support Node Attributes

2018-05-14 Thread Naganarasimha G R (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-8289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16475264#comment-16475264
 ] 

Naganarasimha G R commented on YARN-8289:
-

Hi [~cheersyang], I have not completely thought through but felt this is kind 
of required for demo and kind of similar to yours YARN-7745. But would start 
once atleast YARN-7863  WIP is available. 

> Modify distributedshell to support Node Attributes
> --
>
> Key: YARN-8289
> URL: https://issues.apache.org/jira/browse/YARN-8289
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: distributed-shell
>Affects Versions: YARN-3409
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
>Priority: Major
>
> Modifications required in Distributed shell to support NodeAttributes



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-8289) Modify distributedshell to support Node Attributes

2018-05-14 Thread Naganarasimha G R (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-8289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474472#comment-16474472
 ] 

Naganarasimha G R commented on YARN-8289:
-

Based on the Scheduler API need to modify Distributed shell to support 
NodeAttributes

> Modify distributedshell to support Node Attributes
> --
>
> Key: YARN-8289
> URL: https://issues.apache.org/jira/browse/YARN-8289
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: distributed-shell
>Affects Versions: YARN-3409
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
>Priority: Major
>
> Modifications required in Distributed shell to support NodeAttributes



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Created] (YARN-8289) Modify distributedshell to support Node Attributes

2018-05-14 Thread Naganarasimha G R (JIRA)
Naganarasimha G R created YARN-8289:
---

 Summary: Modify distributedshell to support Node Attributes
 Key: YARN-8289
 URL: https://issues.apache.org/jira/browse/YARN-8289
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: distributed-shell
Affects Versions: YARN-3409
Reporter: Naganarasimha G R
Assignee: Naganarasimha G R


Modifications required in Distributed shell to support NodeAttributes



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-7863) Modify placement constraints to support node attributes

2018-05-14 Thread Naganarasimha G R (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474464#comment-16474464
 ] 

Naganarasimha G R commented on YARN-7863:
-

Thanks [~sunilg], Would support you in the review of it !

> Modify placement constraints to support node attributes
> ---
>
> Key: YARN-7863
> URL: https://issues.apache.org/jira/browse/YARN-7863
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Sunil G
>Assignee: Sunil G
>Priority: Major
>
> This Jira will track to *Modify existing placement constraints to support 
> node attributes.*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-8288) Fix wrong number of table columns in Resource Model doc

2018-05-14 Thread Naganarasimha G R (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-8288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474364#comment-16474364
 ] 

Naganarasimha G R commented on YARN-8288:
-

Thanks [~cheersyang] , i agree with you. Committing this patch.

> Fix wrong number of table columns in Resource Model doc
> ---
>
> Key: YARN-8288
> URL: https://issues.apache.org/jira/browse/YARN-8288
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8288.001.patch, after.jpg, before.jpg
>
>
> In resource model doc, resource-types.xml and node-resource.xml description 
> table has wrong number of columns defined, see 
> [https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceModel.html]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-8288) Fix wrong number of table columns in Resource Model doc

2018-05-14 Thread Naganarasimha G R (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-8288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474102#comment-16474102
 ] 

Naganarasimha G R commented on YARN-8288:
-

[~cheersyang], should it not be that we are putting the default value under 
"Value" column and push the description to the actual "Description" column ?

> Fix wrong number of table columns in Resource Model doc
> ---
>
> Key: YARN-8288
> URL: https://issues.apache.org/jira/browse/YARN-8288
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8288.001.patch, after.jpg, before.jpg
>
>
> In resource model doc, resource-types.xml and node-resource.xml description 
> table has wrong number of columns defined, see 
> [https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceModel.html]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-7892) Revisit NodeAttribute class structure

2018-05-14 Thread Naganarasimha G R (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-7892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474099#comment-16474099
 ] 

Naganarasimha G R commented on YARN-7892:
-

Thanks for the review  [~bibinchundatt] , [~sunilg] & [~leftnoteasy] and the 
commit by [~bibinchundatt].

> Revisit NodeAttribute class structure
> -
>
> Key: YARN-7892
> URL: https://issues.apache.org/jira/browse/YARN-7892
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: resourcemanager
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
>Priority: Major
> Fix For: YARN-3409
>
> Attachments: YARN-7892-YARN-3409.001.patch, 
> YARN-7892-YARN-3409.002.patch, YARN-7892-YARN-3409.003.WIP.patch, 
> YARN-7892-YARN-3409.003.patch, YARN-7892-YARN-3409.004.patch, 
> YARN-7892-YARN-3409.005.patch, YARN-7892-YARN-3409.006.patch, 
> YARN-7892-YARN-3409.007.patch, YARN-7892-YARN-3409.008.patch, 
> YARN-7892-YARN-3409.009.patch, YARN-7892-YARN-3409.010.patch
>
>
> In the existing structure, we had kept the type and value along with the 
> attribute which would create confusion to the user to understand the APIs as 
> they would not be clear as to what needs to be sent for type and value while 
> fetching the mappings for node(s).
> As well as equals will not make sense when we compare only for prefix and 
> name where as values for them might be different.  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-6578) Return container resource utilization from NM ContainerStatus call

2018-05-10 Thread Naganarasimha G R (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-6578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471395#comment-16471395
 ] 

Naganarasimha G R commented on YARN-6578:
-

Sorry for the delayed response [~cheersyang] & [~yangwang166], Was occupied for 
the past two days. Will try to conclude my review by today !

 

> Return container resource utilization from NM ContainerStatus call
> --
>
> Key: YARN-6578
> URL: https://issues.apache.org/jira/browse/YARN-6578
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Yang Wang
>Assignee: Yang Wang
>Priority: Major
> Attachments: YARN-6578.001.patch, YARN-6578.002.patch, 
> YARN-6578.003.patch
>
>
> When the applicationMaster wants to change(increase/decrease) resources of an 
> allocated container, resource utilization is an important reference indicator 
> for decision making. So, when AM call NMClient.getContainerStatus, resource 
> utilization needs to be returned.
> Also container resource utilization need to report to RM to make better 
> scheduling.
> So put resource utilization in ContainerStatus.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



  1   2   3   4   5   6   7   8   9   10   >