[jira] [Commented] (YARN-6857) Support REST for Node Attributes configurations
[ https://issues.apache.org/jira/browse/YARN-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17124842#comment-17124842 ] Naganarasimha G R commented on YARN-6857: - [~prabhujoseph], Glad that you and [~bilwa_st] can take it forward, please go ahead and assign it to yourself! > Support REST for Node Attributes configurations > --- > > Key: YARN-6857 > URL: https://issues.apache.org/jira/browse/YARN-6857 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, capacityscheduler, client >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R >Priority: Major > Attachments: YARN-6857-YARN-3409.001.patch > > > This jira focusses on supporting mapping of Nodes to Attributes through REST -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8768) Javadoc error in node attributes
[ https://issues.apache.org/jira/browse/YARN-8768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16612753#comment-16612753 ] Naganarasimha G R commented on YARN-8768: - Thanks [~shaneku...@gmail.com] for merging it > Javadoc error in node attributes > > > Key: YARN-8768 > URL: https://issues.apache.org/jira/browse/YARN-8768 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-8768.001.patch > > > fix java doc error from node attributes in yarn-api module. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3409) Support Node Attribute functionality
[ https://issues.apache.org/jira/browse/YARN-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16612010#comment-16612010 ] Naganarasimha G R commented on YARN-3409: - All open items has been moved to YARN-8766, and other planned jiras will also be tracked as part of YARN-8766. > Support Node Attribute functionality > > > Key: YARN-3409 > URL: https://issues.apache.org/jira/browse/YARN-3409 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, client, RM >Reporter: Wangda Tan >Assignee: Naganarasimha G R >Priority: Major > Fix For: 3.2.0 > > Attachments: 3409-apiChanges_v2.pdf (4).pdf, > Constraint-Node-Labels-Requirements-Design-doc_v1.pdf, > Node-Attributes-Requirements-Design-doc_v2.pdf, YARN-3409.WIP.001.patch > > > Specify only one label for each node (IAW, partition a cluster) is a way to > determinate how resources of a special set of nodes could be shared by a > group of entities (like teams, departments, etc.). Partitions of a cluster > has following characteristics: > - Cluster divided to several disjoint sub clusters. > - ACL/priority can apply on partition (Only market team / marke team has > priority to use the partition). > - Percentage of capacities can apply on partition (Market team has 40% > minimum capacity and Dev team has 60% of minimum capacity of the partition). > Attributes are orthogonal to partition, they’re describing features of node’s > hardware/software just for affinity. Some example of attributes: > - glibc version > - JDK version > - Type of CPU (x86_64/i686) > - Type of OS (windows, linux, etc.) > With this, application can be able to ask for resource has (glibc.version >= > 2.20 && JDK.version >= 8u20 && x86_64). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-3409) Support Node Attribute functionality
[ https://issues.apache.org/jira/browse/YARN-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R resolved YARN-3409. - Resolution: Fixed Fix Version/s: 3.2.0 Release Note: Node Attribute Feature has been added as part of this jira. > Support Node Attribute functionality > > > Key: YARN-3409 > URL: https://issues.apache.org/jira/browse/YARN-3409 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, client, RM >Reporter: Wangda Tan >Assignee: Naganarasimha G R >Priority: Major > Fix For: 3.2.0 > > Attachments: 3409-apiChanges_v2.pdf (4).pdf, > Constraint-Node-Labels-Requirements-Design-doc_v1.pdf, > Node-Attributes-Requirements-Design-doc_v2.pdf, YARN-3409.WIP.001.patch > > > Specify only one label for each node (IAW, partition a cluster) is a way to > determinate how resources of a special set of nodes could be shared by a > group of entities (like teams, departments, etc.). Partitions of a cluster > has following characteristics: > - Cluster divided to several disjoint sub clusters. > - ACL/priority can apply on partition (Only market team / marke team has > priority to use the partition). > - Percentage of capacities can apply on partition (Market team has 40% > minimum capacity and Dev team has 60% of minimum capacity of the partition). > Attributes are orthogonal to partition, they’re describing features of node’s > hardware/software just for affinity. Some example of attributes: > - glibc version > - JDK version > - Type of CPU (x86_64/i686) > - Type of OS (windows, linux, etc.) > With this, application can be able to ask for resource has (glibc.version >= > 2.20 && JDK.version >= 8u20 && x86_64). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8715) Make allocation tags in the placement spec optional for node-attributes
[
https://issues.apache.org/jira/browse/YARN-8715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naganarasimha G R updated YARN-8715:
Parent Issue: YARN-8766 (was: YARN-3409)
> Make allocation tags in the placement spec optional for node-attributes
> ---
>
> Key: YARN-8715
> URL: https://issues.apache.org/jira/browse/YARN-8715
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
>
> YARN-7863 adds support to specify constraints targeting to node-attributes,
> including the support in distributed shell, but it still needs to specify
> {{allocationTags=numOfContainers}} in the spec. We should make this optional
> as it is not required for node-attribute expressions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8102) Retrospect on having enable and disable flag for Node Attribute
[ https://issues.apache.org/jira/browse/YARN-8102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-8102: Parent Issue: YARN-8766 (was: YARN-3409) > Retrospect on having enable and disable flag for Node Attribute > --- > > Key: YARN-8102 > URL: https://issues.apache.org/jira/browse/YARN-8102 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Major > > Currently node attribute feature is by default enabled. We have to revisit on > the same. > Enabling by default means will try to create store for all cluster > installation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8101) Add UT to verify node-attributes in RM nodes rest API
[ https://issues.apache.org/jira/browse/YARN-8101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-8101: Parent Issue: YARN-8766 (was: YARN-3409) > Add UT to verify node-attributes in RM nodes rest API > - > > Key: YARN-8101 > URL: https://issues.apache.org/jira/browse/YARN-8101 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, restapi >Affects Versions: YARN-3409 >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Minor > > YARN-8092 exposed node-attributes in RM rest api, this task is created to add > UT for that functionality. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7877) Node attributes error handling based on RM validation response
[ https://issues.apache.org/jira/browse/YARN-7877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-7877: Parent Issue: YARN-8766 (was: YARN-3409) > Node attributes error handling based on RM validation response > -- > > Key: YARN-7877 > URL: https://issues.apache.org/jira/browse/YARN-7877 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Major > > When NM reports its distributed attributes to RM, RM needs to do proper > validation of the received attributes, if attributes were not valid or failed > to update, RM needs to notify NM about such failures. Such validation needs > to be also done in NM registration as well. > This JIRA tracks how NM re-act against RM response, probably we can display > some info through UI/metrics/log etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6857) Support REST for Node Attributes configurations
[ https://issues.apache.org/jira/browse/YARN-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-6857: Parent Issue: YARN-8766 (was: YARN-3409) > Support REST for Node Attributes configurations > --- > > Key: YARN-6857 > URL: https://issues.apache.org/jira/browse/YARN-6857 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, capacityscheduler, client >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R >Priority: Major > Attachments: YARN-6857-YARN-3409.001.patch > > > This jira focusses on supporting mapping of Nodes to Attributes through REST -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8766) Support for Node Attribute Functionality, Phase 2
[ https://issues.apache.org/jira/browse/YARN-8766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-8766: Description: This Jira will track all the improvements on the Node Attribute functionality developed in YARN-3409 > Support for Node Attribute Functionality, Phase 2 > - > > Key: YARN-8766 > URL: https://issues.apache.org/jira/browse/YARN-8766 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R >Priority: Major > > This Jira will track all the improvements on the Node Attribute functionality > developed in YARN-3409 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8766) Support for Node Attribute Functionality, Phase 2
Naganarasimha G R created YARN-8766: --- Summary: Support for Node Attribute Functionality, Phase 2 Key: YARN-8766 URL: https://issues.apache.org/jira/browse/YARN-8766 Project: Hadoop YARN Issue Type: Bug Reporter: Naganarasimha G R -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-8766) Support for Node Attribute Functionality, Phase 2
[ https://issues.apache.org/jira/browse/YARN-8766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R reassigned YARN-8766: --- Assignee: Naganarasimha G R > Support for Node Attribute Functionality, Phase 2 > - > > Key: YARN-8766 > URL: https://issues.apache.org/jira/browse/YARN-8766 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8766) Support for Node Attribute Functionality, Phase 2
[ https://issues.apache.org/jira/browse/YARN-8766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-8766: Issue Type: Improvement (was: Bug) > Support for Node Attribute Functionality, Phase 2 > - > > Key: YARN-8766 > URL: https://issues.apache.org/jira/browse/YARN-8766 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7877) Node attributes error handling based on RM validation response
[ https://issues.apache.org/jira/browse/YARN-7877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-7877: Fix Version/s: (was: YARN-3409) > Node attributes error handling based on RM validation response > -- > > Key: YARN-7877 > URL: https://issues.apache.org/jira/browse/YARN-7877 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Major > > When NM reports its distributed attributes to RM, RM needs to do proper > validation of the received attributes, if attributes were not valid or failed > to update, RM needs to notify NM about such failures. Such validation needs > to be also done in NM registration as well. > This JIRA tracks how NM re-act against RM response, probably we can display > some info through UI/metrics/log etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8670) Support scheduling request for SLS input and attributes for Node in SLS
[ https://issues.apache.org/jira/browse/YARN-8670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16611957#comment-16611957 ] Naganarasimha G R commented on YARN-8670: - hi [~Sichen Zhao], please ensure to set target version for desired version to be handled and not Fix version, fix version is updated once its closed > Support scheduling request for SLS input and attributes for Node in SLS > --- > > Key: YARN-8670 > URL: https://issues.apache.org/jira/browse/YARN-8670 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler-load-simulator >Affects Versions: YARN-3409 >Reporter: Sichen zhao >Priority: Major > > YARN-3409 introduces placement constraint, Currently SLS does not support > specify placement constraint. > YARN-8007 support specifying placement constraint for task containers in SLS. > But there are still > some room for improvement: > # YARN-8007 only support placement constraint for the jobs level. In fact, > the more flexible way is support placement constraint for the tasks level. > # In most scenarios, node itself has some characteristics, called attribute, > which is not supported in SLS. So we can add the attribute on Nodes. > # YARN-8007 use the SYNTH as schedulingrequest input. But SYNTH can't create > a large number of specific resource requests. We wanna create a new > schedulingrequest input format(like sis format) for the more authentic input. > We can add some field in sis format, and this is schedulingrequest input > format. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8670) Support scheduling request for SLS input and attributes for Node in SLS
[ https://issues.apache.org/jira/browse/YARN-8670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-8670: Fix Version/s: (was: YARN-3409) > Support scheduling request for SLS input and attributes for Node in SLS > --- > > Key: YARN-8670 > URL: https://issues.apache.org/jira/browse/YARN-8670 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler-load-simulator >Affects Versions: YARN-3409 >Reporter: Sichen zhao >Priority: Major > > YARN-3409 introduces placement constraint, Currently SLS does not support > specify placement constraint. > YARN-8007 support specifying placement constraint for task containers in SLS. > But there are still > some room for improvement: > # YARN-8007 only support placement constraint for the jobs level. In fact, > the more flexible way is support placement constraint for the tasks level. > # In most scenarios, node itself has some characteristics, called attribute, > which is not supported in SLS. So we can add the attribute on Nodes. > # YARN-8007 use the SYNTH as schedulingrequest input. But SYNTH can't create > a large number of specific resource requests. We wanna create a new > schedulingrequest input format(like sis format) for the more authentic input. > We can add some field in sis format, and this is schedulingrequest input > format. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7988) Refactor FSNodeLabelStore code for Node Attributes store support
[ https://issues.apache.org/jira/browse/YARN-7988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-7988: Fix Version/s: (was: YARN-3409) 3.2.0 > Refactor FSNodeLabelStore code for Node Attributes store support > > > Key: YARN-7988 > URL: https://issues.apache.org/jira/browse/YARN-7988 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-7988-YARN-3409.002.patch, > YARN-7988-YARN-3409.003.patch, YARN-7988-YARN-3409.004.patch, > YARN-7988-YARN-3409.005.patch, YARN-7988-YARN-3409.006.patch, > YARN-7988-YARN-3409.007.patch, YARN-7988.001.patch > > > # Abstract out file FileSystemStore operation > # Define EditLog Operartions and Mirror operation > # Support compatibility with old nodelabel store -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7965) NodeAttributeManager add/get API is not working properly
[ https://issues.apache.org/jira/browse/YARN-7965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-7965: Fix Version/s: (was: YARN-3409) 3.2.0 > NodeAttributeManager add/get API is not working properly > > > Key: YARN-7965 > URL: https://issues.apache.org/jira/browse/YARN-7965 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Critical > Fix For: 3.2.0 > > Attachments: YARN-7965-YARN-3409.001.patch, > YARN-7965-YARN-3409.002.patch, YARN-7965-YARN-3409.003.patch, > YARN-7965-YARN-3409.004.patch > > > Fix following issues, > # After add node attributes to the manager, could not retrieve newly added > attributes > # Get cluster attributes API should return empty set when given prefix has > no match > # When an attribute is removed from all nodes, the manager did not remove > this mapping > and add UT -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7892) Revisit NodeAttribute class structure
[ https://issues.apache.org/jira/browse/YARN-7892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-7892: Fix Version/s: (was: YARN-3409) 3.2.0 > Revisit NodeAttribute class structure > - > > Key: YARN-7892 > URL: https://issues.apache.org/jira/browse/YARN-7892 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-7892-YARN-3409.001.patch, > YARN-7892-YARN-3409.002.patch, YARN-7892-YARN-3409.003.WIP.patch, > YARN-7892-YARN-3409.003.patch, YARN-7892-YARN-3409.004.patch, > YARN-7892-YARN-3409.005.patch, YARN-7892-YARN-3409.006.patch, > YARN-7892-YARN-3409.007.patch, YARN-7892-YARN-3409.008.patch, > YARN-7892-YARN-3409.009.patch, YARN-7892-YARN-3409.010.patch > > > In the existing structure, we had kept the type and value along with the > attribute which would create confusion to the user to understand the APIs as > they would not be clear as to what needs to be sent for type and value while > fetching the mappings for node(s). > As well as equals will not make sense when we compare only for prefix and > name where as values for them might be different. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7875) Node Attribute store for storing and recovering attributes
[ https://issues.apache.org/jira/browse/YARN-7875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-7875: Fix Version/s: (was: YARN-3409) 3.2.0 > Node Attribute store for storing and recovering attributes > -- > > Key: YARN-7875 > URL: https://issues.apache.org/jira/browse/YARN-7875 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-7875-WIP.patch, YARN-7875-YARN-3409.001.patch, > YARN-7875-YARN-3409.002.patch, YARN-7875-YARN-3409.003.patch, > YARN-7875-YARN-3409.004.patch, YARN-7875-YARN-3409.005.patch, > YARN-7875-YARN-3409.006.patch, YARN-7875-YARN-3409.007.patch, > YARN-7875-YARN-3409.008.patch > > > Similar to NodeLabelStore need to support NodeAttributeStore for persisting > attributes mapping to Nodes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7871) Node attributes reporting from NM to RM
[ https://issues.apache.org/jira/browse/YARN-7871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-7871: Fix Version/s: (was: YARN-3409) 3.2.0 > Node attributes reporting from NM to RM > > > Key: YARN-7871 > URL: https://issues.apache.org/jira/browse/YARN-7871 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-7871-YARN-3409.001.patch, > YARN-7871-YARN-3409.002.patch, YARN-7871-YARN-3409.003.patch, > YARN-7871-YARN-3409.004.patch > > > Support to initialize proper attribute provider based on user's configuration. > NM collects node attributes from a configured attribute provider and send > them to RM via HB. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7865) Node attributes documentation
[ https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-7865: Fix Version/s: (was: YARN-3409) 3.2.0 > Node attributes documentation > - > > Key: YARN-7865 > URL: https://issues.apache.org/jira/browse/YARN-7865 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation >Reporter: Weiwei Yang >Assignee: Naganarasimha G R >Priority: Major > Labels: node-attributes > Fix For: 3.2.0 > > Attachments: NodeAttributes.html, YARN-7865-YARN-3409.001.patch, > YARN-7865-YARN-3409.002.patch, YARN-7865-YARN-3409.003.patch, > YARN-7865-YARN-3409.004.patch, YARN-7865-YARN-3409.005.patch > > > We need proper docs to introduce how to enable node-attributes how to > configure providers, how to specify script paths, arguments in configuration, > what should be the proper permission of the script and who will run the > script. Also it would be good to add more info to the description of the > configuration properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7863) Modify placement constraints to support node attributes
[ https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-7863: Fix Version/s: (was: YARN-3409) 3.2.0 > Modify placement constraints to support node attributes > --- > > Key: YARN-7863 > URL: https://issues.apache.org/jira/browse/YARN-7863 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-7863-YARN-3409.002.patch, > YARN-7863-YARN-3409.003.patch, YARN-7863-YARN-3409.004.patch, > YARN-7863-YARN-3409.005.patch, YARN-7863-YARN-3409.006.patch, > YARN-7863-YARN-3409.007.patch, YARN-7863-YARN-3409.008.patch, > YARN-7863-YARN-3409.009.patch, YARN-7863-YARN-3409.010.patch, > YARN-7863-YARN-3409.011.patch, YARN-7863.v0.patch > > > This Jira will track to *Modify existing placement constraints to support > node attributes.* -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7856) Validate Node Attributes from NM
[ https://issues.apache.org/jira/browse/YARN-7856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-7856: Fix Version/s: (was: YARN-3409) 3.2.0 > Validate Node Attributes from NM > > > Key: YARN-7856 > URL: https://issues.apache.org/jira/browse/YARN-7856 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, RM >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-7856-YARN-3409.001.patch, > YARN-7856-YARN-3409.002.patch > > > NM needs to do proper validation about the attributes before sending them to > RM, this includes > # a valid prefix is presented > # no duplicate entries > # do not allow two attributes with same prefix/name but different types > This could be an utility class that can be used on both RM/NM sides. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7842) PB changes to carry node-attributes in NM heartbeat
[ https://issues.apache.org/jira/browse/YARN-7842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-7842: Fix Version/s: (was: YARN-3409) 3.2.0 > PB changes to carry node-attributes in NM heartbeat > --- > > Key: YARN-7842 > URL: https://issues.apache.org/jira/browse/YARN-7842 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-7842-YARN-3409.001.patch, > YARN-7842-YARN-3409.002.patch > > > PB changes to carry node-attributes in NM heartbeat. Split from a larger > patch for easier review. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7840) Update PB for prefix support of node attributes
[ https://issues.apache.org/jira/browse/YARN-7840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-7840: Fix Version/s: (was: YARN-3409) 3.2.0 > Update PB for prefix support of node attributes > --- > > Key: YARN-7840 > URL: https://issues.apache.org/jira/browse/YARN-7840 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Weiwei Yang >Assignee: Naganarasimha G R >Priority: Blocker > Fix For: 3.2.0 > > Attachments: YARN-7840-YARN-3409.001.patch, > YARN-7840-YARN-3409.002.patch, YARN-7840-YARN-3409.003.patch, > YARN-7840-YARN-3409.004.patch > > > We need to support prefix (namespace) for node attributes, this will add the > flexibility to provide ability to do proper ACL, avoid naming conflicts etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7757) Refactor NodeLabelsProvider to be more generic and reusable for node attributes providers
[
https://issues.apache.org/jira/browse/YARN-7757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naganarasimha G R updated YARN-7757:
Fix Version/s: (was: YARN-3409)
3.2.0
> Refactor NodeLabelsProvider to be more generic and reusable for node
> attributes providers
> -
>
> Key: YARN-7757
> URL: https://issues.apache.org/jira/browse/YARN-7757
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Blocker
> Fix For: 3.2.0
>
> Attachments: YARN-7757-YARN-3409.001.patch,
> YARN-7757-YARN-3409.002.patch, YARN-7757-YARN-3409.003.patch,
> YARN-7757-YARN-3409.004.patch, YARN-7757-YARN-3409.005.patch,
> YARN-7757-YARN-3409.006.patch,
> nodeLabelsProvider_refactor_class_hierarchy.pdf,
> nodeLabelsProvider_refactor_v2.pdf, nodeLabelsProvider_refactor_v3.pdf
>
>
> Propose to do refactor on {{NodeLabelsProvider}},
> {{AbstractNodeLabelsProvider}} to be more generic, so node attributes
> providers can reuse these interface/abstract classes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7031) Support distributed node attributes
[ https://issues.apache.org/jira/browse/YARN-7031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-7031: Fix Version/s: (was: YARN-3409) 3.2.0 > Support distributed node attributes > --- > > Key: YARN-7031 > URL: https://issues.apache.org/jira/browse/YARN-7031 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Major > Fix For: 3.2.0 > > Attachments: Distributed node attributes v1.pdf, > YARN-7031-YARN-3409.001.patch > > > Allow nodemanagers to push its attributes to RM -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6858) Attribute Manager to store and provide node attributes in RM
[ https://issues.apache.org/jira/browse/YARN-6858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-6858: Fix Version/s: (was: YARN-3409) 3.2.0 > Attribute Manager to store and provide node attributes in RM > > > Key: YARN-6858 > URL: https://issues.apache.org/jira/browse/YARN-6858 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, capacityscheduler, client >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-6858-YARN-3409.001.patch, > YARN-6858-YARN-3409.002.patch, YARN-6858-YARN-3409.003.patch, > YARN-6858-YARN-3409.004.patch, YARN-6858-YARN-3409.005.patch, > YARN-6858-YARN-3409.006.patch, YARN-6858-YARN-3409.007.patch, > YARN-6858-YARN-3409.008.patch, YARN-6858-YARN-3409.009.patch > > > Similar to CommonNodeLabelsManager we need to have a centralized manager for > Node Attributes too. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6856) Support CLI for Node Attributes Mapping
[ https://issues.apache.org/jira/browse/YARN-6856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-6856: Fix Version/s: (was: YARN-3409) 3.2.0 > Support CLI for Node Attributes Mapping > --- > > Key: YARN-6856 > URL: https://issues.apache.org/jira/browse/YARN-6856 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, capacityscheduler, client >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-6856-YARN-3409.001.patch, > YARN-6856-YARN-3409.002.patch, YARN-6856-YARN-3409.004.patch, > YARN-6856-yarn-3409.003.patch, YARN-6856-yarn-3409.004.patch > > > This focuses on the new CLI for the mapping of Node Attributes -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6855) CLI Proto Modifications to support Node Attributes
[ https://issues.apache.org/jira/browse/YARN-6855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-6855: Fix Version/s: (was: YARN-3409) 3.2.0 > CLI Proto Modifications to support Node Attributes > -- > > Key: YARN-6855 > URL: https://issues.apache.org/jira/browse/YARN-6855 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, capacityscheduler, client >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R >Priority: Major > Labels: node-attributes > Fix For: 3.2.0 > > Attachments: YARN-6855-YARN-3409.001.patch, > YARN-6855-YARN-3409.002.patch, YARN-6855-YARN-3409.003.patch, > YARN-6855-YARN-3409.004.patch, YARN-6855-YARN-3409.005.patch, > YARN-6855-YARN-3409.006.patch, YARN-6855-yarn-3409.006.patch > > > This jira focuses only on the proto modifications required for the CLI -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7865) Node attributes documentation
[ https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16598053#comment-16598053 ] Naganarasimha G R commented on YARN-7865: - Thanks [~cheersyang] for the valuable comments, Have addressed it in the latest patch, please check ! > Node attributes documentation > - > > Key: YARN-7865 > URL: https://issues.apache.org/jira/browse/YARN-7865 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation >Reporter: Weiwei Yang >Assignee: Naganarasimha G R >Priority: Major > Attachments: NodeAttributes.html, YARN-7865-YARN-3409.001.patch, > YARN-7865-YARN-3409.002.patch, YARN-7865-YARN-3409.003.patch, > YARN-7865-YARN-3409.004.patch, YARN-7865-YARN-3409.005.patch > > > We need proper docs to introduce how to enable node-attributes how to > configure providers, how to specify script paths, arguments in configuration, > what should be the proper permission of the script and who will run the > script. Also it would be good to add more info to the description of the > configuration properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7865) Node attributes documentation
[ https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-7865: Attachment: YARN-7865-YARN-3409.005.patch > Node attributes documentation > - > > Key: YARN-7865 > URL: https://issues.apache.org/jira/browse/YARN-7865 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation >Reporter: Weiwei Yang >Assignee: Naganarasimha G R >Priority: Major > Attachments: NodeAttributes.html, YARN-7865-YARN-3409.001.patch, > YARN-7865-YARN-3409.002.patch, YARN-7865-YARN-3409.003.patch, > YARN-7865-YARN-3409.004.patch, YARN-7865-YARN-3409.005.patch > > > We need proper docs to introduce how to enable node-attributes how to > configure providers, how to specify script paths, arguments in configuration, > what should be the proper permission of the script and who will run the > script. Also it would be good to add more info to the description of the > configuration properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7865) Node attributes documentation
[ https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-7865: Attachment: (was: NodeAttributes.html) > Node attributes documentation > - > > Key: YARN-7865 > URL: https://issues.apache.org/jira/browse/YARN-7865 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation >Reporter: Weiwei Yang >Assignee: Naganarasimha G R >Priority: Major > Attachments: NodeAttributes.html, YARN-7865-YARN-3409.001.patch, > YARN-7865-YARN-3409.002.patch, YARN-7865-YARN-3409.003.patch, > YARN-7865-YARN-3409.004.patch > > > We need proper docs to introduce how to enable node-attributes how to > configure providers, how to specify script paths, arguments in configuration, > what should be the proper permission of the script and who will run the > script. Also it would be good to add more info to the description of the > configuration properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7865) Node attributes documentation
[ https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-7865: Attachment: NodeAttributes.html > Node attributes documentation > - > > Key: YARN-7865 > URL: https://issues.apache.org/jira/browse/YARN-7865 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation >Reporter: Weiwei Yang >Assignee: Naganarasimha G R >Priority: Major > Attachments: NodeAttributes.html, YARN-7865-YARN-3409.001.patch, > YARN-7865-YARN-3409.002.patch, YARN-7865-YARN-3409.003.patch, > YARN-7865-YARN-3409.004.patch > > > We need proper docs to introduce how to enable node-attributes how to > configure providers, how to specify script paths, arguments in configuration, > what should be the proper permission of the script and who will run the > script. Also it would be good to add more info to the description of the > configuration properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8102) Retrospect on having enable and disable flag for Node Attribute
[ https://issues.apache.org/jira/browse/YARN-8102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16596967#comment-16596967 ] Naganarasimha G R commented on YARN-8102: - Hi [~bibinchundatt] & [~sunilg], I think we had sufficient discussion on this, IMHO its not required and it can be safely configured to local host's tmp where in most of the clusters will have access else we can choose relative path to *_"hadoop.tmp.dir"_* which is what was also used in case of ATSv1 timeline store. I would prefer here to avoid any kind of new configuration being introduced. If agree then we can go ahead and do the required above changes and close this jira. cc [~cheersyang] > Retrospect on having enable and disable flag for Node Attribute > --- > > Key: YARN-8102 > URL: https://issues.apache.org/jira/browse/YARN-8102 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Major > > Currently node attribute feature is by default enabled. We have to revisit on > the same. > Enabling by default means will try to create store for all cluster > installation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7865) Node attributes documentation
[ https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16596963#comment-16596963 ] Naganarasimha G R commented on YARN-7865: - Thanks [~sunilg], for the review, Please find the attached patch addressing your comments. > Node attributes documentation > - > > Key: YARN-7865 > URL: https://issues.apache.org/jira/browse/YARN-7865 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation >Reporter: Weiwei Yang >Assignee: Naganarasimha G R >Priority: Major > Attachments: NodeAttributes.html, YARN-7865-YARN-3409.001.patch, > YARN-7865-YARN-3409.002.patch, YARN-7865-YARN-3409.003.patch, > YARN-7865-YARN-3409.004.patch > > > We need proper docs to introduce how to enable node-attributes how to > configure providers, how to specify script paths, arguments in configuration, > what should be the proper permission of the script and who will run the > script. Also it would be good to add more info to the description of the > configuration properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7865) Node attributes documentation
[ https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-7865: Attachment: YARN-7865-YARN-3409.004.patch > Node attributes documentation > - > > Key: YARN-7865 > URL: https://issues.apache.org/jira/browse/YARN-7865 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation >Reporter: Weiwei Yang >Assignee: Naganarasimha G R >Priority: Major > Attachments: NodeAttributes.html, YARN-7865-YARN-3409.001.patch, > YARN-7865-YARN-3409.002.patch, YARN-7865-YARN-3409.003.patch, > YARN-7865-YARN-3409.004.patch > > > We need proper docs to introduce how to enable node-attributes how to > configure providers, how to specify script paths, arguments in configuration, > what should be the proper permission of the script and who will run the > script. Also it would be good to add more info to the description of the > configuration properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7865) Node attributes documentation
[ https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-7865: Attachment: (was: NodeAttributes.html) > Node attributes documentation > - > > Key: YARN-7865 > URL: https://issues.apache.org/jira/browse/YARN-7865 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation >Reporter: Weiwei Yang >Assignee: Naganarasimha G R >Priority: Major > Attachments: NodeAttributes.html, YARN-7865-YARN-3409.001.patch, > YARN-7865-YARN-3409.002.patch, YARN-7865-YARN-3409.003.patch > > > We need proper docs to introduce how to enable node-attributes how to > configure providers, how to specify script paths, arguments in configuration, > what should be the proper permission of the script and who will run the > script. Also it would be good to add more info to the description of the > configuration properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7865) Node attributes documentation
[ https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-7865: Attachment: NodeAttributes.html > Node attributes documentation > - > > Key: YARN-7865 > URL: https://issues.apache.org/jira/browse/YARN-7865 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation >Reporter: Weiwei Yang >Assignee: Naganarasimha G R >Priority: Major > Attachments: NodeAttributes.html, YARN-7865-YARN-3409.001.patch, > YARN-7865-YARN-3409.002.patch, YARN-7865-YARN-3409.003.patch > > > We need proper docs to introduce how to enable node-attributes how to > configure providers, how to specify script paths, arguments in configuration, > what should be the proper permission of the script and who will run the > script. Also it would be good to add more info to the description of the > configuration properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3409) Support Node Attribute functionality
[ https://issues.apache.org/jira/browse/YARN-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-3409: Attachment: Node-Attributes-Requirements-Design-doc_v2.pdf > Support Node Attribute functionality > > > Key: YARN-3409 > URL: https://issues.apache.org/jira/browse/YARN-3409 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, client, RM >Reporter: Wangda Tan >Assignee: Naganarasimha G R >Priority: Major > Attachments: 3409-apiChanges_v2.pdf (4).pdf, > Constraint-Node-Labels-Requirements-Design-doc_v1.pdf, > Node-Attributes-Requirements-Design-doc_v2.pdf, YARN-3409.WIP.001.patch > > > Specify only one label for each node (IAW, partition a cluster) is a way to > determinate how resources of a special set of nodes could be shared by a > group of entities (like teams, departments, etc.). Partitions of a cluster > has following characteristics: > - Cluster divided to several disjoint sub clusters. > - ACL/priority can apply on partition (Only market team / marke team has > priority to use the partition). > - Percentage of capacities can apply on partition (Market team has 40% > minimum capacity and Dev team has 60% of minimum capacity of the partition). > Attributes are orthogonal to partition, they’re describing features of node’s > hardware/software just for affinity. Some example of attributes: > - glibc version > - JDK version > - Type of CPU (x86_64/i686) > - Type of OS (windows, linux, etc.) > With this, application can be able to ask for resource has (glibc.version >= > 2.20 && JDK.version >= 8u20 && x86_64). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7865) Node attributes documentation
[ https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16595792#comment-16595792 ] Naganarasimha G R commented on YARN-7865: - [~bibinchundatt], Have attached a patch without all whitespace issue fixed and the remaining are required for the mark up to detect as next paragraph. Please Commit without *WHITESPACE=FIX* option if no one has any more comments. > Node attributes documentation > - > > Key: YARN-7865 > URL: https://issues.apache.org/jira/browse/YARN-7865 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation >Reporter: Weiwei Yang >Assignee: Naganarasimha G R >Priority: Major > Attachments: NodeAttributes.html, YARN-7865-YARN-3409.001.patch, > YARN-7865-YARN-3409.002.patch, YARN-7865-YARN-3409.003.patch > > > We need proper docs to introduce how to enable node-attributes how to > configure providers, how to specify script paths, arguments in configuration, > what should be the proper permission of the script and who will run the > script. Also it would be good to add more info to the description of the > configuration properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7865) Node attributes documentation
[ https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-7865: Attachment: YARN-7865-YARN-3409.003.patch > Node attributes documentation > - > > Key: YARN-7865 > URL: https://issues.apache.org/jira/browse/YARN-7865 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation >Reporter: Weiwei Yang >Assignee: Naganarasimha G R >Priority: Major > Attachments: NodeAttributes.html, YARN-7865-YARN-3409.001.patch, > YARN-7865-YARN-3409.002.patch, YARN-7865-YARN-3409.003.patch > > > We need proper docs to introduce how to enable node-attributes how to > configure providers, how to specify script paths, arguments in configuration, > what should be the proper permission of the script and who will run the > script. Also it would be good to add more info to the description of the > configuration properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7865) Node attributes documentation
[ https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16595119#comment-16595119 ] Naganarasimha G R commented on YARN-7865: - Thanks [~bibinchundatt] for the review comments and have addressed all except for the second point as we replace only attributes for the prefix “nm.yarn.io”. Also have addressed the white space and tab issues reported by checkstyle in the latest patch > Node attributes documentation > - > > Key: YARN-7865 > URL: https://issues.apache.org/jira/browse/YARN-7865 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation >Reporter: Weiwei Yang >Assignee: Naganarasimha G R >Priority: Major > Attachments: NodeAttributes.html, YARN-7865-YARN-3409.001.patch, > YARN-7865-YARN-3409.002.patch > > > We need proper docs to introduce how to enable node-attributes how to > configure providers, how to specify script paths, arguments in configuration, > what should be the proper permission of the script and who will run the > script. Also it would be good to add more info to the description of the > configuration properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7865) Node attributes documentation
[ https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-7865: Attachment: YARN-7865-YARN-3409.002.patch > Node attributes documentation > - > > Key: YARN-7865 > URL: https://issues.apache.org/jira/browse/YARN-7865 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation >Reporter: Weiwei Yang >Assignee: Naganarasimha G R >Priority: Major > Attachments: NodeAttributes.html, YARN-7865-YARN-3409.001.patch, > YARN-7865-YARN-3409.002.patch > > > We need proper docs to introduce how to enable node-attributes how to > configure providers, how to specify script paths, arguments in configuration, > what should be the proper permission of the script and who will run the > script. Also it would be good to add more info to the description of the > configuration properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7865) Node attributes documentation
[ https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-7865: Attachment: NodeAttributes.html > Node attributes documentation > - > > Key: YARN-7865 > URL: https://issues.apache.org/jira/browse/YARN-7865 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation >Reporter: Weiwei Yang >Assignee: Naganarasimha G R >Priority: Major > Attachments: NodeAttributes.html, YARN-7865-YARN-3409.001.patch > > > We need proper docs to introduce how to enable node-attributes how to > configure providers, how to specify script paths, arguments in configuration, > what should be the proper permission of the script and who will run the > script. Also it would be good to add more info to the description of the > configuration properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7865) Node attributes documentation
[ https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-7865: Attachment: (was: NodeAttributes.html) > Node attributes documentation > - > > Key: YARN-7865 > URL: https://issues.apache.org/jira/browse/YARN-7865 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation >Reporter: Weiwei Yang >Assignee: Naganarasimha G R >Priority: Major > Attachments: YARN-7865-YARN-3409.001.patch > > > We need proper docs to introduce how to enable node-attributes how to > configure providers, how to specify script paths, arguments in configuration, > what should be the proper permission of the script and who will run the > script. Also it would be good to add more info to the description of the > configuration properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7865) Node attributes documentation
[ https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594540#comment-16594540 ] Naganarasimha G R commented on YARN-7865: - [~sunilg], [~cheersyang] & [~bibinchundatt], Can all of you have a look at the documentation and share if any modifications required ? > Node attributes documentation > - > > Key: YARN-7865 > URL: https://issues.apache.org/jira/browse/YARN-7865 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation >Reporter: Weiwei Yang >Assignee: Naganarasimha G R >Priority: Major > Attachments: NodeAttributes.html, YARN-7865-YARN-3409.001.patch > > > We need proper docs to introduce how to enable node-attributes how to > configure providers, how to specify script paths, arguments in configuration, > what should be the proper permission of the script and who will run the > script. Also it would be good to add more info to the description of the > configuration properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7865) Node attributes documentation
[ https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-7865: Attachment: YARN-7865-YARN-3409.001.patch > Node attributes documentation > - > > Key: YARN-7865 > URL: https://issues.apache.org/jira/browse/YARN-7865 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation >Reporter: Weiwei Yang >Assignee: Naganarasimha G R >Priority: Major > Attachments: NodeAttributes.html, YARN-7865-YARN-3409.001.patch > > > We need proper docs to introduce how to enable node-attributes how to > configure providers, how to specify script paths, arguments in configuration, > what should be the proper permission of the script and who will run the > script. Also it would be good to add more info to the description of the > configuration properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7865) Node attributes documentation
[ https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-7865: Attachment: NodeAttributes.html > Node attributes documentation > - > > Key: YARN-7865 > URL: https://issues.apache.org/jira/browse/YARN-7865 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation >Reporter: Weiwei Yang >Assignee: Naganarasimha G R >Priority: Major > Attachments: NodeAttributes.html > > > We need proper docs to introduce how to enable node-attributes how to > configure providers, how to specify script paths, arguments in configuration, > what should be the proper permission of the script and who will run the > script. Also it would be good to add more info to the description of the > configuration properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7865) Node attributes documentation
[ https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-7865: Attachment: (was: YARN-7865-YARN-3409.001.patch) > Node attributes documentation > - > > Key: YARN-7865 > URL: https://issues.apache.org/jira/browse/YARN-7865 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation >Reporter: Weiwei Yang >Assignee: Naganarasimha G R >Priority: Major > Attachments: NodeAttributes.html > > > We need proper docs to introduce how to enable node-attributes how to > configure providers, how to specify script paths, arguments in configuration, > what should be the proper permission of the script and who will run the > script. Also it would be good to add more info to the description of the > configuration properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7865) Node attributes documentation
[ https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-7865: Attachment: (was: YARN-8721-YARN-3409.001.patch) > Node attributes documentation > - > > Key: YARN-7865 > URL: https://issues.apache.org/jira/browse/YARN-7865 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation >Reporter: Weiwei Yang >Assignee: Naganarasimha G R >Priority: Major > Attachments: YARN-7865-YARN-3409.001.patch > > > We need proper docs to introduce how to enable node-attributes how to > configure providers, how to specify script paths, arguments in configuration, > what should be the proper permission of the script and who will run the > script. Also it would be good to add more info to the description of the > configuration properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7865) Node attributes documentation
[ https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-7865: Attachment: YARN-7865-YARN-3409.001.patch > Node attributes documentation > - > > Key: YARN-7865 > URL: https://issues.apache.org/jira/browse/YARN-7865 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation >Reporter: Weiwei Yang >Assignee: Naganarasimha G R >Priority: Major > Attachments: YARN-7865-YARN-3409.001.patch > > > We need proper docs to introduce how to enable node-attributes how to > configure providers, how to specify script paths, arguments in configuration, > what should be the proper permission of the script and who will run the > script. Also it would be good to add more info to the description of the > configuration properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7865) Node attributes documentation
[ https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-7865: Attachment: YARN-8721-YARN-3409.001.patch > Node attributes documentation > - > > Key: YARN-7865 > URL: https://issues.apache.org/jira/browse/YARN-7865 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation >Reporter: Weiwei Yang >Assignee: Naganarasimha G R >Priority: Major > Attachments: YARN-7865-YARN-3409.001.patch > > > We need proper docs to introduce how to enable node-attributes how to > configure providers, how to specify script paths, arguments in configuration, > what should be the proper permission of the script and who will run the > script. Also it would be good to add more info to the description of the > configuration properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8721) Scheduler should accept nodes which doesnt have an attribute when NodeAttributeType.NE is used
[ https://issues.apache.org/jira/browse/YARN-8721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594432#comment-16594432 ] Naganarasimha G R commented on YARN-8721: - [~sunilg], IIRC idea for the boolean was not to support different type but to have value as empty string and have expression to support "exist" and "not exist" operator. I am fine with achieving it after merge but my concern is whether others like Microsoft will agree during merge! > Scheduler should accept nodes which doesnt have an attribute when > NodeAttributeType.NE is used > -- > > Key: YARN-8721 > URL: https://issues.apache.org/jira/browse/YARN-8721 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Attachments: YARN-8721-YARN-3409.001.patch, > YARN-8721-YARN-3409.002.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8721) Scheduler should accept nodes which doesnt have an attribute when NodeAttributeType.NE is used
[ https://issues.apache.org/jira/browse/YARN-8721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594248#comment-16594248 ] Naganarasimha G R commented on YARN-8721: - Hi [~sunilg], Just had a overview and the patch seems to be fine, but one query i had was to how are we supporting Boolean type attributes ? i.e. we are currently supporting EQ( =) and NE ( != ) only but during requirement discussion there were lot of use cases(from Microsoft) where in they wanted to use exists not exists kind of operation instead of value based comparison. I think this might pop up during merge. Thoughts? > Scheduler should accept nodes which doesnt have an attribute when > NodeAttributeType.NE is used > -- > > Key: YARN-8721 > URL: https://issues.apache.org/jira/browse/YARN-8721 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Attachments: YARN-8721-YARN-3409.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (YARN-8721) Scheduler should accept nodes which doesnt have an attribute when NodeAttributeType.NE is used
[ https://issues.apache.org/jira/browse/YARN-8721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-8721: Comment: was deleted (was: Hi [~sunilg], There are some checkstyle issues in YARN-8718 , can that be addressed along with this patch itself ?) > Scheduler should accept nodes which doesnt have an attribute when > NodeAttributeType.NE is used > -- > > Key: YARN-8721 > URL: https://issues.apache.org/jira/browse/YARN-8721 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Attachments: YARN-8721-YARN-3409.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8721) Scheduler should accept nodes which doesnt have an attribute when NodeAttributeType.NE is used
[ https://issues.apache.org/jira/browse/YARN-8721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594234#comment-16594234 ] Naganarasimha G R commented on YARN-8721: - Hi [~sunilg], There are some checkstyle issues in YARN-8718 , can that be addressed along with this patch itself ? > Scheduler should accept nodes which doesnt have an attribute when > NodeAttributeType.NE is used > -- > > Key: YARN-8721 > URL: https://issues.apache.org/jira/browse/YARN-8721 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Attachments: YARN-8721-YARN-3409.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7863) Modify placement constraints to support node attributes
[ https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593114#comment-16593114 ] Naganarasimha G R commented on YARN-7863: - Yes [~sunilg], as both the issues are related and we would require a test case for the weiwei's scenario too, hence i agree can be covered in other jira. Going ahead and committing the patch ! > Modify placement constraints to support node attributes > --- > > Key: YARN-7863 > URL: https://issues.apache.org/jira/browse/YARN-7863 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Attachments: YARN-7863-YARN-3409.002.patch, > YARN-7863-YARN-3409.003.patch, YARN-7863-YARN-3409.004.patch, > YARN-7863-YARN-3409.005.patch, YARN-7863-YARN-3409.006.patch, > YARN-7863-YARN-3409.007.patch, YARN-7863-YARN-3409.008.patch, > YARN-7863-YARN-3409.009.patch, YARN-7863-YARN-3409.010.patch, > YARN-7863-YARN-3409.011.patch, YARN-7863.v0.patch > > > This Jira will track to *Modify existing placement constraints to support > node attributes.* -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7863) Modify placement constraints to support node attributes
[ https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593109#comment-16593109 ] Naganarasimha G R edited comment on YARN-7863 at 8/27/18 1:58 AM: -- Hi [~sunilg], As discussed some comments can be handled in a separate jira, but still i could see test case coverage is not to that extent there for scheduler side of code. *PlacementConstraintsUtil* : "Evaluation of Placement expression containing Node Attributes". Also i believe i think [~cheersyang]'s comment is also valid, can you take a stab at it ? Lets conclude next one as the final patch so that i can go ahead and commit it to unblock the overall functionality! For all others lets raise separate jira's. was (Author: naganarasimha): Hi [~sunilg], As discussed some comments can be handled in a separate jira, but still i could see test case coverage is not to that extent there from scheduler side of code. *PlacementConstraintsUtil* : "Evaluation of Placement expression containing Node Attributes". Also i believe i think [~cheersyang]'s comment is also valid, can you take a stab at it ? Lets conclude next one as the final patch so that i can go ahead and commit it to unblock the overall functionality! For all others lets raise separate jira's. > Modify placement constraints to support node attributes > --- > > Key: YARN-7863 > URL: https://issues.apache.org/jira/browse/YARN-7863 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Attachments: YARN-7863-YARN-3409.002.patch, > YARN-7863-YARN-3409.003.patch, YARN-7863-YARN-3409.004.patch, > YARN-7863-YARN-3409.005.patch, YARN-7863-YARN-3409.006.patch, > YARN-7863-YARN-3409.007.patch, YARN-7863-YARN-3409.008.patch, > YARN-7863-YARN-3409.009.patch, YARN-7863-YARN-3409.010.patch, > YARN-7863-YARN-3409.011.patch, YARN-7863.v0.patch > > > This Jira will track to *Modify existing placement constraints to support > node attributes.* -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7863) Modify placement constraints to support node attributes
[ https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593109#comment-16593109 ] Naganarasimha G R commented on YARN-7863: - Hi [~sunilg], As discussed some comments can be handled in a separate jira, but still i could see test case coverage is not to that extent there from scheduler side of code. *PlacementConstraintsUtil* : "Evaluation of Placement expression containing Node Attributes". Also i believe i think [~cheersyang]'s comment is also valid, can you take a stab at it ? Lets conclude next one as the final patch so that i can go ahead and commit it to unblock the overall functionality! For all others lets raise separate jira's. > Modify placement constraints to support node attributes > --- > > Key: YARN-7863 > URL: https://issues.apache.org/jira/browse/YARN-7863 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Attachments: YARN-7863-YARN-3409.002.patch, > YARN-7863-YARN-3409.003.patch, YARN-7863-YARN-3409.004.patch, > YARN-7863-YARN-3409.005.patch, YARN-7863-YARN-3409.006.patch, > YARN-7863-YARN-3409.007.patch, YARN-7863-YARN-3409.008.patch, > YARN-7863-YARN-3409.009.patch, YARN-7863-YARN-3409.010.patch, > YARN-7863-YARN-3409.011.patch, YARN-7863.v0.patch > > > This Jira will track to *Modify existing placement constraints to support > node attributes.* -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7863) Modify placement constraints to support node attributes
[ https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16592847#comment-16592847 ] Naganarasimha G R commented on YARN-7863: - Hi [~sunilg], Latest patch does not seem to consider all my [comments|https://issues.apache.org/jira/browse/YARN-7863?focusedCommentId=16589559&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16589559] and also latest patch seems to have some checkstyle issues to be addressed. Can you please take a look at them? > Modify placement constraints to support node attributes > --- > > Key: YARN-7863 > URL: https://issues.apache.org/jira/browse/YARN-7863 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Attachments: YARN-7863-YARN-3409.002.patch, > YARN-7863-YARN-3409.003.patch, YARN-7863-YARN-3409.004.patch, > YARN-7863-YARN-3409.005.patch, YARN-7863-YARN-3409.006.patch, > YARN-7863-YARN-3409.007.patch, YARN-7863-YARN-3409.008.patch, > YARN-7863-YARN-3409.009.patch, YARN-7863-YARN-3409.010.patch, > YARN-7863.v0.patch > > > This Jira will track to *Modify existing placement constraints to support > node attributes.* -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7863) Modify placement constraints to support node attributes
[
https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590410#comment-16590410
]
Naganarasimha G R commented on YARN-7863:
-
Thanks [~cheersyang] for the detailed clarification,
{quote} PCs are not associated with resource, so it's like an extra check after
all other checks are done. Scheduler still calculates how much resource
available in a partition for a given queue, assign resource from a node in this
partition to a request, but if PC is not satisfied then the allocation proposal
will be rejected. et
{quote}
Agree PC's are not directly involved with resource, But IIUC with the new API
:_SchedulingRequest_ for allocation of containers has only PC as the way to
specify partitions which is tied with resource . This is the reason i had my
concerns with the OR. Let me go through the code with new API and understand
whether Partition is handled for all scenarios. I was always under the
impression that there will be a constraint explicitly for the Partition based
on which resources are accounted ( USED, PENDING , RESERVED etc...). But if its
internal to the constraint which is also OR'd with others i am not completely
sure its addressing .
{quote}Partition in PC is not ready, to be honest, I am not sure if everything
is align with existing label-based scheduling. I suggested in YARN-8015 to open
a separate task for further enhance that.
{quote}
Let me too try to understand the PC and new api modifications in detail and
then share my feedback in that Jira and anyway its out of the scope for this
jira.
> Modify placement constraints to support node attributes
> ---
>
> Key: YARN-7863
> URL: https://issues.apache.org/jira/browse/YARN-7863
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Sunil Govindan
>Assignee: Sunil Govindan
>Priority: Major
> Attachments: YARN-7863-YARN-3409.002.patch,
> YARN-7863-YARN-3409.003.patch, YARN-7863-YARN-3409.004.patch,
> YARN-7863-YARN-3409.005.patch, YARN-7863-YARN-3409.006.patch,
> YARN-7863-YARN-3409.007.patch, YARN-7863-YARN-3409.008.patch,
> YARN-7863.v0.patch
>
>
> This Jira will track to *Modify existing placement constraints to support
> node attributes.*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7863) Modify placement constraints to support node attributes
[ https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589559#comment-16589559 ] Naganarasimha G R commented on YARN-7863: - Hi [~sunilg], few other nits in the patch : * NodeAttributesManagerImpl ln no 205: as i was mentioning there is both debug and info log here ,i think we can remove the debug log. * NodeAttributesManagerImpl ln no 212-226 : Here you are sending out complete update of the node collections, and not just the modified NM's only. There are multiple impacts due to this, in large cluster we are unnecessarily sending lot of updates to scheduler secondly removed attributes will not be captured in this way. Where in later is more important * NodeAttributesManagerImpl ln no 212-226 : In general idea earlier was to make use of the AttributeValue by the scheduler so that the converted value is stored and used for comparison . But if we have the flexibility later on to change the scheduler event which is being pushed from NAM to Schedulers then i am fine with the Event being sent out, else i would suggest to send the AttributeValue itself * Test cases for AND and OR are covered ? Though i could see AND not declaratively covered in TestPlacementConstraintParser but better to cover with AND and OR explicitly * PlacementSpec ln no 51: typo "teh" > Modify placement constraints to support node attributes > --- > > Key: YARN-7863 > URL: https://issues.apache.org/jira/browse/YARN-7863 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Attachments: YARN-7863-YARN-3409.002.patch, > YARN-7863-YARN-3409.003.patch, YARN-7863-YARN-3409.004.patch, > YARN-7863-YARN-3409.005.patch, YARN-7863-YARN-3409.006.patch, > YARN-7863-YARN-3409.007.patch, YARN-7863-YARN-3409.008.patch, > YARN-7863.v0.patch > > > This Jira will track to *Modify existing placement constraints to support > node attributes.* -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7863) Modify placement constraints to support node attributes
[
https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589171#comment-16589171
]
Naganarasimha G R commented on YARN-7863:
-
Thanks [~cheersyang], for some clarifications but at the same time was
discussing with Sunil too.
{quote} PC doesn't affect any logic how scheduler selects requests, it is still
how it is handled now. A PC is simply checked twice when 1) creating an
allocation proposal on a node and 2) at commit phase against a specific node.
{quote}
I am not specifying that changes in this Jira alone introducing anything
implications but my point was, earlier when partitions was introduced it was
easy to determine pending resources per partition per queue in the earlier api
using resource request. Now with the New api introduced with PC(not just from
this Jira alone) there is no way to find out for a given queue how much pending
resources are there in each partition it can access. This is because PC can
have an Partition OR'd with Allocation TAG or with this Jira OR'd with
Attributes. Impact of this would be that Cluster admin will not be able to plan
resources for the partition per queue. And also i am not able to envisage the
scenario where in partition needs to be OR'd with Allocation tags or Attributes.
{quote}No, not necessarily. Just to keep the changes clean and incremental, we
can allow this form for now. Because this is the spec we used for distributed
shell. Since we don't have an --allocationTags argument. A "foo=3" is the only
way to specify allocation tags right now.
{quote}
Agree, My bad but the reason i got confused is all the parsing logic for the
Distributed shell's expression is present in "hadoop-yarn-api" project which
more so implies that this the API to be used by other apps too.
{quote}I want to reinforce the phase-target (for the merge), we want
node-attributes can be integrated into PC and support simple operation "=" and
"!=".
{quote}
Yes I concur with you on this, as long as we are able to clearly capture for
others what kind of Java API needs to be used to specify Node attributes.
[~sunilg],
Could there be a simple test case or example which captures how to write a java
api where in node attributes can be specified for a given SchedulingRequest
i.e. without any of these expression DSL's ?
Also i could not see a test case where in CS is handling scheduling of
containers with PC having attributes. So that modifications in
PlacementConstraintsUtil is tested.
> Modify placement constraints to support node attributes
> ---
>
> Key: YARN-7863
> URL: https://issues.apache.org/jira/browse/YARN-7863
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Sunil Govindan
>Assignee: Sunil Govindan
>Priority: Major
> Attachments: YARN-7863-YARN-3409.002.patch,
> YARN-7863-YARN-3409.003.patch, YARN-7863-YARN-3409.004.patch,
> YARN-7863-YARN-3409.005.patch, YARN-7863-YARN-3409.006.patch,
> YARN-7863-YARN-3409.007.patch, YARN-7863-YARN-3409.008.patch,
> YARN-7863.v0.patch
>
>
> This Jira will track to *Modify existing placement constraints to support
> node attributes.*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7863) Modify placement constraints to support node attributes
[
https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589075#comment-16589075
]
Naganarasimha G R commented on YARN-7863:
-
Hi [~sunilg],
Now i am confused when we have everything in one single expression .
* IIUC we can specify allocation tags, attribute expression and partition
label all in single expression ?
* If its the case then how are we going select the outstanding/ pending
requests for a given queue for a given partition, as there could be OR with
allocation tags or attribute expression right ? I think this issue should be
already existing when we have OR ing involved with allocation tags ?
* Based on the test cases in
TestPlacementConstraintParser.testParseNodeAttributeSpec, Allocation tags are
created even if we want to specify attribute expression ? ex :
"xyz,in,rm.yarn.io/foo=true" xyz is allocation tag ? If so users will be
really confused what to pass and what not !
* if i specify "IN" and have attribute expression as
"xyz,in,rm.yarn.io/foo{color:#d04437}*!=*{color}true" is it valid ? In future
when we come with more operators it would not make sense. I would suggest to go
with "," as separator.
I am not sure how many cases would be there where we want to specify all the
constraints in the same expression but we are making the users life complex
with having such a complex DSL.
> Modify placement constraints to support node attributes
> ---
>
> Key: YARN-7863
> URL: https://issues.apache.org/jira/browse/YARN-7863
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Sunil Govindan
>Assignee: Sunil Govindan
>Priority: Major
> Attachments: YARN-7863-YARN-3409.002.patch,
> YARN-7863-YARN-3409.003.patch, YARN-7863-YARN-3409.004.patch,
> YARN-7863-YARN-3409.005.patch, YARN-7863-YARN-3409.006.patch,
> YARN-7863-YARN-3409.007.patch, YARN-7863-YARN-3409.008.patch,
> YARN-7863.v0.patch
>
>
> This Jira will track to *Modify existing placement constraints to support
> node attributes.*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7863) Modify placement constraints to support node attributes
[ https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16585329#comment-16585329 ] Naganarasimha G R commented on YARN-7863: - Thanks [~sunilg], for the latest patch, Some of the things [~cheersyang] mentioned i too had the same issues and had mentioned in my earlier comment like : * _PlacementConstraintParser.NodeConstraintParser_ not to store value (had captured 2nd point) . * _getConstraintNameWithNameSpace()_ defaults the namespace (had captured 4th point) . And seems like many other comments like the following was not addressed or replied yet: * I replaced expression @ _TestPlacementConstraintParser ln no 481,_ with "OR(IN,python=true:NOTIN,java=1.8):IN,os!=centos:NOTIN,env=prod,dev" and i see exception as "_Invalid constraint expression OR(IN,python=true:NOTIN,java=1.8)_". IIUC our branch does not support OR expressions yet, So does it requires a rebase to confirm the same ? * Distributed shell needs to support both specs for the allocation tags and attributes, which i believe can only be achieved having AttributeSpec . Unless we have to seriously meet the deadlines then lets avoid it but if you already have it ready then why not go for it ? * TestPlacementConstraintParser ln no 453 : testParseNodeAttributeSpec has attributeName1, attributeName2; which is unused * ResourceManager ln no 652-654 createNodeAttributesManager should either return NodeAttributesManagerImpl or take parameter as rmContext instead of type casting * findbugs seems to be existent from the previous patch. I concur with most of the [~cheersyang]'s comments except * We need to have a info log or rather an audit to track the attributes being modified but nevertheless there is both info and a debug log of the same message @202 and 205 , but 222 can be avoided all together *TestCases :* * I would love to have a DistributedShell test case with Node attributes in the placement Contstraints. As discussed if required we can take the DS changes from here to a separate patch to make this patch less bulky * And & OR Expressions in the expressions * Test case for covering scheduling part of the Placement expression containing Node Attributes Shall we have a quick huddle some time this week, so that we can finalize on these stuff and then go ahead ? > Modify placement constraints to support node attributes > --- > > Key: YARN-7863 > URL: https://issues.apache.org/jira/browse/YARN-7863 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Attachments: YARN-7863-YARN-3409.002.patch, > YARN-7863-YARN-3409.003.patch, YARN-7863-YARN-3409.004.patch, > YARN-7863-YARN-3409.005.patch, YARN-7863-YARN-3409.006.patch, > YARN-7863.v0.patch > > > This Jira will track to *Modify existing placement constraints to support > node attributes.* -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7863) Modify placement constraints to support node attributes
[ https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16577644#comment-16577644 ] Naganarasimha G R commented on YARN-7863: - Thanks for the patch [~sunilg] and apologies for the delayed review, as i was not acquainted with Placement Constraint part of the code, i required some time in understanding the placement constraints and then in relation how you have modified it. But still could not figure out few of the stuff * Not completely clear sure how the parsing happens for the *And* and *Or* cases which was mentioned by [~cheersyang] in his earlier comment "_-placement_spec OR(IN,python=true:NOTIN,java=1.8):IN,os!=centos:NOTIN,env=prod,dev"_ Just tried to replace the same in your test case but seems like it fails at PlacementConstraintParser.parseNodeAttributeExpression ln no 669. Whether its expected to fail or it should assume it as *And* and proceed? * In PlacementConstraintParser.NodeConstraintParser.parse() PlacementTargets target in ln no 399 seems to include keys also in val. Was not sure why we need to store in this way, would it not be bettert only values here * Can we place both constraints related to attributes and allocation tags together in a single placement spec for distributed shell? As I understand the Placement spec API supports to take multiple type of placement constraints for a single resource/allocation request. We too should support it right ? * NodeConstraintParser.parse seems to assume that attributeNameSpace has "rm.yarn.io/" but IIRC we concluded that user need to specify this explicitly right else exception is thrown right ? some minor comments * TestPlacementConstraintParser ln no 453 : testParseNodeAttributeSpec has attributeName1, attributeName2; which is unused * ResourceManager ln no 652-654 createNodeAttributesManager should either return NodeAttributesManagerImpl or take parameter as rmContext instead of type casting * Seems like build seems to have find bugs and test case failures. Can you please have a look at it ? > Modify placement constraints to support node attributes > --- > > Key: YARN-7863 > URL: https://issues.apache.org/jira/browse/YARN-7863 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Attachments: YARN-7863-YARN-3409.002.patch, > YARN-7863-YARN-3409.003.patch, YARN-7863-YARN-3409.004.patch, > YARN-7863.v0.patch > > > This Jira will track to *Modify existing placement constraints to support > node attributes.* -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8593) Add new RM web service endpoint to get cluster user info
[ https://issues.apache.org/jira/browse/YARN-8593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16561751#comment-16561751 ] Naganarasimha G R commented on YARN-8593: - Hi [~akhilpb], May be can you please share more info on why this api is required ? and is it good to share with what user RM is running ? Even if required is it not good to be known from admin CLI rather than other forms ? > Add new RM web service endpoint to get cluster user info > > > Key: YARN-8593 > URL: https://issues.apache.org/jira/browse/YARN-8593 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: Akhil PB >Assignee: Akhil PB >Priority: Major > Attachments: YARN-8593.001.patch, YARN-8593.002.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7863) Modify placement constraints to support node attributes
[ https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16561134#comment-16561134 ] Naganarasimha G R commented on YARN-7863: - Hi [~sunilg], Patch seems to fail after rebase (tried locally too). Can you please take a look at it ? > Modify placement constraints to support node attributes > --- > > Key: YARN-7863 > URL: https://issues.apache.org/jira/browse/YARN-7863 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Attachments: YARN-7863-YARN-3409.002.patch, > YARN-7863-YARN-3409.003.patch, YARN-7863.v0.patch > > > This Jira will track to *Modify existing placement constraints to support > node attributes.* -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-7863) Modify placement constraints to support node attributes
[ https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R reassigned YARN-7863: --- Assignee: Sunil Govindan (was: Naganarasimha G R) > Modify placement constraints to support node attributes > --- > > Key: YARN-7863 > URL: https://issues.apache.org/jira/browse/YARN-7863 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Attachments: YARN-7863-YARN-3409.002.patch, > YARN-7863-YARN-3409.003.patch, YARN-7863.v0.patch > > > This Jira will track to *Modify existing placement constraints to support > node attributes.* -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-7863) Modify placement constraints to support node attributes
[ https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R reassigned YARN-7863: --- Assignee: Naganarasimha G R (was: Sunil Govindan) > Modify placement constraints to support node attributes > --- > > Key: YARN-7863 > URL: https://issues.apache.org/jira/browse/YARN-7863 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil Govindan >Assignee: Naganarasimha G R >Priority: Major > Attachments: YARN-7863-YARN-3409.002.patch, > YARN-7863-YARN-3409.003.patch, YARN-7863.v0.patch > > > This Jira will track to *Modify existing placement constraints to support > node attributes.* -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8574) Allow dot in attribute values
[ https://issues.apache.org/jira/browse/YARN-8574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16561015#comment-16561015 ] Naganarasimha G R commented on YARN-8574: - Thanks [~sunilg], for rebase. Overall approach looks good to me. I will commit the patch shortly. > Allow dot in attribute values > -- > > Key: YARN-8574 > URL: https://issues.apache.org/jira/browse/YARN-8574 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Minor > Attachments: YARN-8574-YARN-3409.001.patch, > YARN-8574-YARN-3409.002.patch > > > Currently "." is considered as invalid value. Enable the same; -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8574) Allow dot in attribute values
[ https://issues.apache.org/jira/browse/YARN-8574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16559060#comment-16559060 ] Naganarasimha G R edited comment on YARN-8574 at 7/27/18 12:11 AM: --- Hi [~bibinchundatt], You are correct i meant when we start using it as namespace later on. but anyway as we are supporting for internal its as well good to be supported for others too. My only doubt now is how was it NOT failing for earlier mappings for prefix? Have manually triggered the build too... was (Author: naganarasimha): Hi [~bibinchundatt], You are correct i meant when we start using it as namespace later on. but anyway as we are supporting for internal its as well good to be supported for others too. My ownly doubt now is how was it not failing for earlier mappings for prefix? > Allow dot in attribute values > -- > > Key: YARN-8574 > URL: https://issues.apache.org/jira/browse/YARN-8574 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Minor > Attachments: YARN-8574-YARN-3409.001.patch > > > Currently "." is considered as invalid value. Enable the same; -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8574) Allow dot in attribute values
[ https://issues.apache.org/jira/browse/YARN-8574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16559060#comment-16559060 ] Naganarasimha G R commented on YARN-8574: - Hi [~bibinchundatt], You are correct i meant when we start using it as namespace later on. but anyway as we are supporting for internal its as well good to be supported for others too. My ownly doubt now is how was it not failing for earlier mappings for prefix? > Allow dot in attribute values > -- > > Key: YARN-8574 > URL: https://issues.apache.org/jira/browse/YARN-8574 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Minor > Attachments: YARN-8574-YARN-3409.001.patch > > > Currently "." is considered as invalid value. Enable the same; -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8574) Allow dot in attribute values
[ https://issues.apache.org/jira/browse/YARN-8574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16555499#comment-16555499 ] Naganarasimha G R commented on YARN-8574: - Thanks for the patch [~bibinchundatt], my bad almost had forgotten to work upon it. One concern i had is agree that value should be allowed with a dot but not the prefix, so for prefix if we are using the same pattern it qill be a problem > Allow dot in attribute values > -- > > Key: YARN-8574 > URL: https://issues.apache.org/jira/browse/YARN-8574 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Minor > Attachments: YARN-8574-YARN-3409.001.patch > > > Currently "." is considered as invalid value. Enable the same; -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7863) Modify placement constraints to support node attributes
[ https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16553691#comment-16553691 ] Naganarasimha G R commented on YARN-7863: - Have done just a first pass review, few higher level comments : # I think Distributed shell related modifications can be pulled into separate Jira (YARN-8289) as discussed earlier # I could not get in the meeting wrt to Attributes and Partitions/labels, IIUC in ResourceRequest we specify the label expression directly but in SchedulingRequest we can give it as part of PlacementConstraint ? And if so we club it with the attribute expression ? May be we need capture that explicitly over here.can you share a sample request with both of them available? # Attach the doc which captured our conclusions on the expression or capture the essense of it as Jira's descritpion Other minor aspects: * PlacementConstraints.java ln 52- 53, better to use NodeAttribute.PREFIX_DISTRIBUTED & PREFIX_CENTRALIZED * PlacementConstraintParser ln 363-436 NodeConstraintsTokenizer seems to be not used anywhere ? * ResourceManager ln no 652-654 createNodeAttributesManager should either return NodeAttributesManagerImpl or take parameter as rmContext instead of type casting * .bowerrc ln no 3, might not be related to this patch ? * TestPlacementConstraintParser : captures a single test case , we need to capture more examples of capturing the positive and negative expression validation cases * TestPlacementConstraintParser ln no 457 : is this the approach which we finalised ? i thought its much more expressive like the document in PlacementConstraintParser.parsePlacementSpecForAttributes As well change the status of the Jira to patch submitted so that jenkins build will be triggered. > Modify placement constraints to support node attributes > --- > > Key: YARN-7863 > URL: https://issues.apache.org/jira/browse/YARN-7863 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Attachments: YARN-7863-YARN-3409.002.patch, YARN-7863.v0.patch > > > This Jira will track to *Modify existing placement constraints to support > node attributes.* -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7863) Modify placement constraints to support node attributes
[ https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16552201#comment-16552201 ] Naganarasimha G R commented on YARN-7863: - Hi [~sunilg], While i am going through the patch, would you mind converting the status from Inprogress to patch submitted so that the jenkins build is run ? > Modify placement constraints to support node attributes > --- > > Key: YARN-7863 > URL: https://issues.apache.org/jira/browse/YARN-7863 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Attachments: YARN-7863-YARN-3409.002.patch, YARN-7863.v0.patch > > > This Jira will track to *Modify existing placement constraints to support > node attributes.* -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8480) Add boolean option for resources
[ https://issues.apache.org/jira/browse/YARN-8480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16552196#comment-16552196 ] Naganarasimha G R commented on YARN-8480: - Sorry for pitching in late, ran through the discussions in the Jira and I concur with [~leftnoteasy], I do not see the need for providing 2 approaches to address the same scenario. Understand there would be some initial hickups in using Node Attributes in FS but i think that would be the better approach than having 2 different approaches for FS and CS to address the same scenario. If we wear the user's cap and see it would be good to have a single approach to address a scenario irrespective of the scheduler. If we start addressing scenario's differently for FS and CS then there will 2 parallel systems within YARN which is not desirable. [~templedf],[~snemeth], May be if you guys can share all the features which you want from Node Attributes to address then we can confirm to what extent we have already covered in Phase 1. As far as i see if its only exist and not exist then we do already cover. > Add boolean option for resources > > > Key: YARN-8480 > URL: https://issues.apache.org/jira/browse/YARN-8480 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Daniel Templeton >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-8480.001.patch, YARN-8480.002.patch > > > Make it possible to define a resource with a boolean value. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7863) Modify placement constraints to support node attributes
[ https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16529118#comment-16529118 ] Naganarasimha G R commented on YARN-7863: - [~sunilg], Few queries before going into the detail ... 1. who is doing the validation of the resource request? 2. If there are any open resource request for specified attribute, do we allow removing of that attribute from Attribute Manager ? 3. Confused little with the logic (may be earlier code to yours), i was always of the opinion that the partition is the first to be compared and then with tag or attribute here it seems to first check for the allocation tag and then for partition and then attribute. should it not be like under a given parition check any nodes having allocation tags and/or attributes satisfying the user's request? Some comments : 1. PlacementConstraintsUtil, ln no 145-156 : Scheduler node needs to be having map of NodeAttributeKey -> NodeAttribute value to make it O(1), as well this will more useful when we go other types and operator based comparison happens. Or to consider for current scenario atleast it would be better to override NodeAttribute's equals method. > Modify placement constraints to support node attributes > --- > > Key: YARN-7863 > URL: https://issues.apache.org/jira/browse/YARN-7863 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Attachments: YARN-7863.v0.patch > > > This Jira will track to *Modify existing placement constraints to support > node attributes.* -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7863) Modify placement constraints to support node attributes
[ https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16529111#comment-16529111 ] Naganarasimha G R edited comment on YARN-7863 at 7/1/18 2:19 PM: - Thanks for the patch [~sunilg], It gives a very good idea what you plan to do in scheduler for node attributes. At the high level overview approach seems to be fine. Hope you can change the status of the Jira to Patch available so that Jenkins run can happen. Also please modify the attached patch format so that it jenkins runs against YARN-3409 branch was (Author: naganarasimha): Thanks for the patch [~sunilg], It gives a very good idea what you plan to do in scheduler for node attributes. At the high level overview approach seems to be fine. Hope you can change the status of the Jira to Patch available so that Jenkins run can happen. > Modify placement constraints to support node attributes > --- > > Key: YARN-7863 > URL: https://issues.apache.org/jira/browse/YARN-7863 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Attachments: YARN-7863.v0.patch > > > This Jira will track to *Modify existing placement constraints to support > node attributes.* -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7863) Modify placement constraints to support node attributes
[ https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16529111#comment-16529111 ] Naganarasimha G R commented on YARN-7863: - Thanks for the patch [~sunilg], It gives a very good idea what you plan to do in scheduler for node attributes. At the high level overview approach seems to be fine. Hope you can change the status of the Jira to Patch available so that Jenkins run can happen. > Modify placement constraints to support node attributes > --- > > Key: YARN-7863 > URL: https://issues.apache.org/jira/browse/YARN-7863 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Attachments: YARN-7863.v0.patch > > > This Jira will track to *Modify existing placement constraints to support > node attributes.* -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8103) Add CLI interface to query node attributes
[ https://issues.apache.org/jira/browse/YARN-8103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525758#comment-16525758 ] Naganarasimha G R commented on YARN-8103: - Thanks for the contribution [~bibinchundatt] and additional review by [~cheersyang]. I have committed the patch to branch YARN-3409. > Add CLI interface to query node attributes > --- > > Key: YARN-8103 > URL: https://issues.apache.org/jira/browse/YARN-8103 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Major > Attachments: YARN-8103-YARN-3409.001.patch, > YARN-8103-YARN-3409.002.patch, YARN-8103-YARN-3409.003.patch, > YARN-8103-YARN-3409.004.patch, YARN-8103-YARN-3409.005.patch, > YARN-8103-YARN-3409.006.patch, YARN-8103-YARN-3409.WIP.patch > > > YARN-8100 will add API interface for querying the attributes. CLI interface > for querying node attributes for each nodes and list all attributes in > cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8103) Add CLI interface to query node attributes
[ https://issues.apache.org/jira/browse/YARN-8103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16524084#comment-16524084 ] Naganarasimha G R commented on YARN-8103: - Thanks for the updated patch [~bibinchundatt], latest patch LGTM. If no further comments from [~cheersyang] and [~sunil.gov...@gmail.com], will go ahead and commit tomorrow EOD... As discussed sorting based on need can be handled as additional improvement task. > Add CLI interface to query node attributes > --- > > Key: YARN-8103 > URL: https://issues.apache.org/jira/browse/YARN-8103 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Major > Attachments: YARN-8103-YARN-3409.001.patch, > YARN-8103-YARN-3409.002.patch, YARN-8103-YARN-3409.003.patch, > YARN-8103-YARN-3409.004.patch, YARN-8103-YARN-3409.005.patch, > YARN-8103-YARN-3409.006.patch, YARN-8103-YARN-3409.WIP.patch > > > YARN-8100 will add API interface for querying the attributes. CLI interface > for querying node attributes for each nodes and list all attributes in > cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8103) Add CLI interface to query node attributes
[ https://issues.apache.org/jira/browse/YARN-8103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16520880#comment-16520880 ] Naganarasimha G R commented on YARN-8103: - Thanks for the patch [~bibinchundatt], Sorry my bad had done the validation wrong in "equals" and everything else is fine except for checkstyle issues having some unused imports. Also one small suggestion. would it be good to sort the attribute listing based one the attribute prefix/ name and listing it ? (in all places, nodecli, cluster cli and nodeattributecli). Hope others can also take a look as its almost ready to go in ? > Add CLI interface to query node attributes > --- > > Key: YARN-8103 > URL: https://issues.apache.org/jira/browse/YARN-8103 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Major > Attachments: YARN-8103-YARN-3409.001.patch, > YARN-8103-YARN-3409.002.patch, YARN-8103-YARN-3409.003.patch, > YARN-8103-YARN-3409.004.patch, YARN-8103-YARN-3409.005.patch, > YARN-8103-YARN-3409.WIP.patch > > > YARN-8100 will add API interface for querying the attributes. CLI interface > for querying node attributes for each nodes and list all attributes in > cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8103) Add CLI interface to query node attributes
[ https://issues.apache.org/jira/browse/YARN-8103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16517924#comment-16517924 ] Naganarasimha G R commented on YARN-8103: - IMHO, its better to be in multiple lines as it will be more readable and also when value comes in to picture it might be of any length. _"1-3 attributes per node"_ : cannot be pre determined based on use case different systems might have different number of attributes per node and also depends on how much system internal attributes we are coming up with. And obviously in the demo we will be using few attributes per node so its better to discuss in this Jira than waiting for the demo. > Add CLI interface to query node attributes > --- > > Key: YARN-8103 > URL: https://issues.apache.org/jira/browse/YARN-8103 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Major > Attachments: YARN-8103-YARN-3409.001.patch, > YARN-8103-YARN-3409.002.patch, YARN-8103-YARN-3409.003.patch, > YARN-8103-YARN-3409.004.patch, YARN-8103-YARN-3409.WIP.patch > > > YARN-8100 will add API interface for querying the attributes. CLI interface > for querying node attributes for each nodes and list all attributes in > cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8103) Add CLI interface to query node attributes
[ https://issues.apache.org/jira/browse/YARN-8103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16516100#comment-16516100 ] Naganarasimha G R commented on YARN-8103: - Thanks for the patch [~bibinchundatt] Some of my comments seems to be still not addressed: * NodeAttributesCLI ln no 187: i think its better to use null here instead of "handler" for better readability * TestNodeAttributesCLI ln no 405: testListAttributes is encapsulating NodesToAttributes tests too, may be it can captured as a different case * NodeCLI ln no 317 : IMO it would better to wrap each attribute in a new line ? And some of the [~cheersyang] comments needs to be discussed (though i am not more towards numeric form for to but we can discuss other better options) and this comment needs to be handled * line 588: buildNodeLabelsMapFromStr -> buildNodeAttributesListFromStr Apart from this many of the Checkstyle issues still seems to be valid and needs to be addressed. > Add CLI interface to query node attributes > --- > > Key: YARN-8103 > URL: https://issues.apache.org/jira/browse/YARN-8103 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Major > Attachments: YARN-8103-YARN-3409.001.patch, > YARN-8103-YARN-3409.002.patch, YARN-8103-YARN-3409.003.patch, > YARN-8103-YARN-3409.WIP.patch > > > YARN-8100 will add API interface for querying the attributes. CLI interface > for querying node attributes for each nodes and list all attributes in > cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8103) Add CLI interface to query node attributes
[ https://issues.apache.org/jira/browse/YARN-8103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16501238#comment-16501238 ] Naganarasimha G R edited comment on YARN-8103 at 6/5/18 3:53 AM: - Thanks for the patch and apologies for the delayed response. Major Comments : * I agree with the approach of not storing and updating the attributes in RMNode and instead request NodeAttributesManager to share the information. But was also wondering whether the API(RMNodeImpl.getAllNodeAttributes) is useful based on current scenarios ? All the callers are eventually converting it into a set of attributes and utilizing it. So i would prefer to change the api to just return the set of attributes applicable on a node and when needed let the caller take care of sorting based on the prefix( which is anyway not a current scenario) Few other comments: * hadoop-yarn/bin/yarn ln no 58: i think "client" was missing from earlier which we need to add it * NodeAttributesCLI ln no 195: i think its better to use null here instead of "handler" for better readability * NodeAttributesCLI ln no 88,96 : unused variables * TestNodeAttributesCLI ln no 405: testListAttributes is encapsulating NodesToAttributes tests too, may be it can captured as a different case * NodeAttributesCLI ln no 355 : HashSet => HashSet * NodeAttributesCLI ln no 394 : HashSet => HashSet * NodeCLI ln no 349: formatNodeAttribute : instead of having new formatting here why not update NodeAttributePBImpl.toString ? * NodeCLI ln no 317 : IMO it would better to wrap each attribute in a new line ? * TestClusterCLI : does not capture any case for listing of attributes (basically need to capture multiline) * BuilderUtils ln no 211 : space is required before Set Some of the findbugs and checkstyle seems to be valid can you have a look at them ? Kudos NodeAttributesCLI has been handled well ! was (Author: naganarasimha): Thanks for the patch and apologies for the delayed response. Major Comments : * I agree with the approach of not storing and updating the attributes in RMNode and instead request NodeAttributesManager to share the information. But was also wondering whether the API(RMNodeImpl.getAllNodeAttributes) is useful based on current scenarios ? All the callers are eventually converting it into a set of attributes and utilizing it. So i would prefer to change the api to just return the set of attributes applicable on a node and when needed let the caller take care of sorting based on the prefix( which is anyway not a current scenario) Few other comments: * hadoop-yarn/bin/yarn ln no 58: i think "client" was missing from earlier which we need to add it * NodeAttributesCLI ln no 195: i think its better to use null here instead of "handler" for better readability * NodeAttributesCLI ln no 88,96 : unused variables * TestNodeAttributesCLI ln no 405: testListAttributes is encapsulating NodesToAttributes tests too, may be it can captured as a different case Some of the findbugs and checkstyle seems to be valid can you have a look at them ? Kudos NodeAttributesCLI has been handled well ! > Add CLI interface to query node attributes > --- > > Key: YARN-8103 > URL: https://issues.apache.org/jira/browse/YARN-8103 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Major > Attachments: YARN-8103-YARN-3409.001.patch, > YARN-8103-YARN-3409.002.patch, YARN-8103-YARN-3409.WIP.patch > > > YARN-8100 will add API interface for querying the attributes. CLI interface > for querying node attributes for each nodes and list all attributes in > cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8103) Add CLI interface to query node attributes
[ https://issues.apache.org/jira/browse/YARN-8103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16501238#comment-16501238 ] Naganarasimha G R commented on YARN-8103: - Thanks for the patch and apologies for the delayed response. Major Comments : * I agree with the approach of not storing and updating the attributes in RMNode and instead request NodeAttributesManager to share the information. But was also wondering whether the API(RMNodeImpl.getAllNodeAttributes) is useful based on current scenarios ? All the callers are eventually converting it into a set of attributes and utilizing it. So i would prefer to change the api to just return the set of attributes applicable on a node and when needed let the caller take care of sorting based on the prefix( which is anyway not a current scenario) Few other comments: * hadoop-yarn/bin/yarn ln no 58: i think "client" was missing from earlier which we need to add it * NodeAttributesCLI ln no 195: i think its better to use null here instead of "handler" for better readability * NodeAttributesCLI ln no 88,96 : unused variables * TestNodeAttributesCLI ln no 405: testListAttributes is encapsulating NodesToAttributes tests too, may be it can captured as a different case Some of the findbugs and checkstyle seems to be valid can you have a look at them ? Kudos NodeAttributesCLI has been handled well ! > Add CLI interface to query node attributes > --- > > Key: YARN-8103 > URL: https://issues.apache.org/jira/browse/YARN-8103 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Major > Attachments: YARN-8103-YARN-3409.001.patch, > YARN-8103-YARN-3409.002.patch, YARN-8103-YARN-3409.WIP.patch > > > YARN-8100 will add API interface for querying the attributes. CLI interface > for querying node attributes for each nodes and list all attributes in > cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6578) Return container resource utilization from NM ContainerStatus call
[
https://issues.apache.org/jira/browse/YARN-6578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477463#comment-16477463
]
Naganarasimha G R edited comment on YARN-6578 at 5/16/18 1:56 PM:
--
[~cheersyang] & [~fly_in_gis],
I went through the approach, but was wondering its not covering the scenario
with resource profiles, if multiple resource types are supported then this
would be like not a proper solution in terms of the Object
{{ResourceUtilization}} being returned & changes in {{ContainerStatusPBImpl}}
by storing only cpu, vmem and pmem as concatenated string.
Was wondering whether we need to return {{ResourceInformation}} and store
resource type information and individual resource values as separate key value
pairs in the {{ContainerStatusPBImpl.containerAttributes}} map ?
cc [~sunil.gov...@gmail.com]
was (Author: naganarasimha):
[~cheersyang] & [~fly_in_gis],
I went through the approach, but was wondering its not covering the scenario
with resource profiles, if multiple resource types are supported then this
would be like not a proper solution in terms of the Object
{{ResourceUtilization}} being returned & changes in{{ContainerStatusPBImpl}} by
storing only .
Was wondering whether we need to return {{ResourceInformation }}and store
resource type information and individual resource values as separate key value
pairs in the {{ContainerStatusPBImpl.containerAttributes}} map ?
cc [~sunil.gov...@gmail.com]
> Return container resource utilization from NM ContainerStatus call
> --
>
> Key: YARN-6578
> URL: https://issues.apache.org/jira/browse/YARN-6578
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Yang Wang
>Assignee: Yang Wang
>Priority: Major
> Attachments: YARN-6578.001.patch, YARN-6578.002.patch,
> YARN-6578.003.patch
>
>
> When the applicationMaster wants to change(increase/decrease) resources of an
> allocated container, resource utilization is an important reference indicator
> for decision making. So, when AM call NMClient.getContainerStatus, resource
> utilization needs to be returned.
> Also container resource utilization need to report to RM to make better
> scheduling.
> So put resource utilization in ContainerStatus.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6578) Return container resource utilization from NM ContainerStatus call
[
https://issues.apache.org/jira/browse/YARN-6578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477463#comment-16477463
]
Naganarasimha G R commented on YARN-6578:
-
[~cheersyang] & [~fly_in_gis],
I went through the approach, but was wondering its not covering the scenario
with resource profiles, if multiple resource types are supported then this
would be like not a proper solution in terms of the Object
{{ResourceUtilization}} being returned & changes in{{ContainerStatusPBImpl}} by
storing only .
Was wondering whether we need to return {{ResourceInformation }}and store
resource type information and individual resource values as separate key value
pairs in the {{ContainerStatusPBImpl.containerAttributes}} map ?
cc [~sunil.gov...@gmail.com]
> Return container resource utilization from NM ContainerStatus call
> --
>
> Key: YARN-6578
> URL: https://issues.apache.org/jira/browse/YARN-6578
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Yang Wang
>Assignee: Yang Wang
>Priority: Major
> Attachments: YARN-6578.001.patch, YARN-6578.002.patch,
> YARN-6578.003.patch
>
>
> When the applicationMaster wants to change(increase/decrease) resources of an
> allocated container, resource utilization is an important reference indicator
> for decision making. So, when AM call NMClient.getContainerStatus, resource
> utilization needs to be returned.
> Also container resource utilization need to report to RM to make better
> scheduling.
> So put resource utilization in ContainerStatus.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8289) Modify distributedshell to support Node Attributes
[ https://issues.apache.org/jira/browse/YARN-8289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16475361#comment-16475361 ] Naganarasimha G R commented on YARN-8289: - [~sunil.gov...@gmail.com], Given that you had already working on the scheduler patch and testing it already with DS shell, would you like to take this Jira up? > Modify distributedshell to support Node Attributes > -- > > Key: YARN-8289 > URL: https://issues.apache.org/jira/browse/YARN-8289 > Project: Hadoop YARN > Issue Type: Sub-task > Components: distributed-shell >Affects Versions: YARN-3409 >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R >Priority: Major > > Modifications required in Distributed shell to support NodeAttributes -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8289) Modify distributedshell to support Node Attributes
[ https://issues.apache.org/jira/browse/YARN-8289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16475264#comment-16475264 ] Naganarasimha G R commented on YARN-8289: - Hi [~cheersyang], I have not completely thought through but felt this is kind of required for demo and kind of similar to yours YARN-7745. But would start once atleast YARN-7863 WIP is available. > Modify distributedshell to support Node Attributes > -- > > Key: YARN-8289 > URL: https://issues.apache.org/jira/browse/YARN-8289 > Project: Hadoop YARN > Issue Type: Sub-task > Components: distributed-shell >Affects Versions: YARN-3409 >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R >Priority: Major > > Modifications required in Distributed shell to support NodeAttributes -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8289) Modify distributedshell to support Node Attributes
[ https://issues.apache.org/jira/browse/YARN-8289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474472#comment-16474472 ] Naganarasimha G R commented on YARN-8289: - Based on the Scheduler API need to modify Distributed shell to support NodeAttributes > Modify distributedshell to support Node Attributes > -- > > Key: YARN-8289 > URL: https://issues.apache.org/jira/browse/YARN-8289 > Project: Hadoop YARN > Issue Type: Sub-task > Components: distributed-shell >Affects Versions: YARN-3409 >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R >Priority: Major > > Modifications required in Distributed shell to support NodeAttributes -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8289) Modify distributedshell to support Node Attributes
Naganarasimha G R created YARN-8289: --- Summary: Modify distributedshell to support Node Attributes Key: YARN-8289 URL: https://issues.apache.org/jira/browse/YARN-8289 Project: Hadoop YARN Issue Type: Sub-task Components: distributed-shell Affects Versions: YARN-3409 Reporter: Naganarasimha G R Assignee: Naganarasimha G R Modifications required in Distributed shell to support NodeAttributes -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7863) Modify placement constraints to support node attributes
[ https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474464#comment-16474464 ] Naganarasimha G R commented on YARN-7863: - Thanks [~sunilg], Would support you in the review of it ! > Modify placement constraints to support node attributes > --- > > Key: YARN-7863 > URL: https://issues.apache.org/jira/browse/YARN-7863 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil G >Assignee: Sunil G >Priority: Major > > This Jira will track to *Modify existing placement constraints to support > node attributes.* -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8288) Fix wrong number of table columns in Resource Model doc
[ https://issues.apache.org/jira/browse/YARN-8288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474364#comment-16474364 ] Naganarasimha G R commented on YARN-8288: - Thanks [~cheersyang] , i agree with you. Committing this patch. > Fix wrong number of table columns in Resource Model doc > --- > > Key: YARN-8288 > URL: https://issues.apache.org/jira/browse/YARN-8288 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Major > Attachments: YARN-8288.001.patch, after.jpg, before.jpg > > > In resource model doc, resource-types.xml and node-resource.xml description > table has wrong number of columns defined, see > [https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceModel.html] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8288) Fix wrong number of table columns in Resource Model doc
[ https://issues.apache.org/jira/browse/YARN-8288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474102#comment-16474102 ] Naganarasimha G R commented on YARN-8288: - [~cheersyang], should it not be that we are putting the default value under "Value" column and push the description to the actual "Description" column ? > Fix wrong number of table columns in Resource Model doc > --- > > Key: YARN-8288 > URL: https://issues.apache.org/jira/browse/YARN-8288 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Major > Attachments: YARN-8288.001.patch, after.jpg, before.jpg > > > In resource model doc, resource-types.xml and node-resource.xml description > table has wrong number of columns defined, see > [https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceModel.html] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7892) Revisit NodeAttribute class structure
[ https://issues.apache.org/jira/browse/YARN-7892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474099#comment-16474099 ] Naganarasimha G R commented on YARN-7892: - Thanks for the review [~bibinchundatt] , [~sunilg] & [~leftnoteasy] and the commit by [~bibinchundatt]. > Revisit NodeAttribute class structure > - > > Key: YARN-7892 > URL: https://issues.apache.org/jira/browse/YARN-7892 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R >Priority: Major > Fix For: YARN-3409 > > Attachments: YARN-7892-YARN-3409.001.patch, > YARN-7892-YARN-3409.002.patch, YARN-7892-YARN-3409.003.WIP.patch, > YARN-7892-YARN-3409.003.patch, YARN-7892-YARN-3409.004.patch, > YARN-7892-YARN-3409.005.patch, YARN-7892-YARN-3409.006.patch, > YARN-7892-YARN-3409.007.patch, YARN-7892-YARN-3409.008.patch, > YARN-7892-YARN-3409.009.patch, YARN-7892-YARN-3409.010.patch > > > In the existing structure, we had kept the type and value along with the > attribute which would create confusion to the user to understand the APIs as > they would not be clear as to what needs to be sent for type and value while > fetching the mappings for node(s). > As well as equals will not make sense when we compare only for prefix and > name where as values for them might be different. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6578) Return container resource utilization from NM ContainerStatus call
[ https://issues.apache.org/jira/browse/YARN-6578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16471395#comment-16471395 ] Naganarasimha G R commented on YARN-6578: - Sorry for the delayed response [~cheersyang] & [~yangwang166], Was occupied for the past two days. Will try to conclude my review by today ! > Return container resource utilization from NM ContainerStatus call > -- > > Key: YARN-6578 > URL: https://issues.apache.org/jira/browse/YARN-6578 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Yang Wang >Assignee: Yang Wang >Priority: Major > Attachments: YARN-6578.001.patch, YARN-6578.002.patch, > YARN-6578.003.patch > > > When the applicationMaster wants to change(increase/decrease) resources of an > allocated container, resource utilization is an important reference indicator > for decision making. So, when AM call NMClient.getContainerStatus, resource > utilization needs to be returned. > Also container resource utilization need to report to RM to make better > scheduling. > So put resource utilization in ContainerStatus. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
