[jira] [Updated] (YARN-7205) Log improvements for the ResourceUtils

[Sunil G (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sunil G updated YARN-7205:
--
Attachment: YARN-7205.004.patch

Updating v4 patch. Thanks [~templedf], please help to check the latest patch

> Log improvements for the ResourceUtils
> --
>
> Key: YARN-7205
> URL: https://issues.apache.org/jira/browse/YARN-7205
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager, resourcemanager
>Reporter: Jian He
>Assignee: Sunil G
> Attachments: YARN-7205.001.patch, YARN-7205.002.patch, 
> YARN-7205.003.patch, YARN-7205.004.patch
>
>
> I've seen below logs printed at the service client console after the merge, 
> can this be moved to debug level log ? cc  [~sunilg], [~leftnoteasy]
> {code}
> 17/09/15 10:26:32 INFO conf.Configuration: resource-types.xml not found
> 17/09/15 10:26:32 INFO resource.ResourceUtils: Unable to find 
> 'resource-types.xml'. Falling back to memory and vcores as resources.
> 17/09/15 10:26:32 INFO resource.ResourceUtils: Adding resource type - name = 
> memory-mb, units = Mi, type = COUNTABLE
> 17/09/15 10:26:32 INFO resource.ResourceUtils: Adding resource type - name = 
> vcores, units = , type = COUNTABLE
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7296) [Resource Profiles] convertToProtoFormat(Resource r) is not setting all resources

[Sunil G (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198202#comment-16198202
 ] 

Sunil G commented on YARN-7296:
---

+1 for v2 patch. Committing shortly.

> [Resource Profiles] convertToProtoFormat(Resource r) is not setting all 
> resources
> -
>
> Key: YARN-7296
> URL: https://issues.apache.org/jira/browse/YARN-7296
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Affects Versions: 3.1.0
>Reporter: lovekesh bansal
> Fix For: 3.1.0
>
> Attachments: YARN-7296_YARN-3926.001.patch, 
> YARN-7296_trunk.001.patch, YARN-7296_trunk.002.patch
>
>
> convertToProtoFormat(Resource r) is not setting all resources in a couple of 
> files i.e. ProfileCapabilityPBImpl.java



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7272) Enable timeline collector fault tolerance

[Rohith Sharma K S (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198201#comment-16198201
 ] 

Rohith Sharma K S commented on YARN-7272:
-

thanks for clarifying doubts!
bq. Is there a specific concern about using leveldb to implement the WAL for 
transient persistence? 
We don't have any concerns for using leveldb. Given delete operation can be 
performed, I would also highly recommend for using level db.  

> Enable timeline collector fault tolerance
> -
>
> Key: YARN-7272
> URL: https://issues.apache.org/jira/browse/YARN-7272
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineclient, timelinereader, timelineserver
>Reporter: Vrushali C
>Assignee: Rohith Sharma K S
>
> If a NM goes down and along with it the timeline collector aux service for a 
> running yarn app, we would like that yarn app to re-establish connection with 
> a new timeline collector. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7312) [Resource Profiles] getMaximumAllocationPerQueue to account for all resources

[Sunil G (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sunil G updated YARN-7312:
--
Issue Type: Sub-task  (was: Task)
Parent: YARN-7069

> [Resource Profiles] getMaximumAllocationPerQueue to account for all resources
> -
>
> Key: YARN-7312
> URL: https://issues.apache.org/jira/browse/YARN-7312
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Affects Versions: 3.1.0
>Reporter: lovekesh bansal
> Fix For: 3.1.0
>
> Attachments: YARN-7312_trunk.001.patch
>
>
> getMaximumAllocationPerQueue has memory and vcores hardcoded. So it should be 
> made generic for all resources.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-7312) [Resource Profiles] getMaximumAllocationPerQueue to account for all resources

[lovekesh bansal (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198186#comment-16198186
 ] 

lovekesh bansal edited comment on YARN-7312 at 10/10/17 5:40 AM:
-

V1 Patch uploaded for umbrella YARN-7069.


was (Author: lovekesh.bansal):
V1 Patch uploaded.

> [Resource Profiles] getMaximumAllocationPerQueue to account for all resources
> -
>
> Key: YARN-7312
> URL: https://issues.apache.org/jira/browse/YARN-7312
> Project: Hadoop YARN
>  Issue Type: Task
>Affects Versions: 3.1.0
>Reporter: lovekesh bansal
> Fix For: 3.1.0
>
> Attachments: YARN-7312_trunk.001.patch
>
>
> getMaximumAllocationPerQueue has memory and vcores hardcoded. So it should be 
> made generic for all resources.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7312) [Resource Profiles] getMaximumAllocationPerQueue to account for all resources

[lovekesh bansal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

lovekesh bansal updated YARN-7312:
--
Attachment: YARN-7312_trunk.001.patch

V1 Patch uploaded.

> [Resource Profiles] getMaximumAllocationPerQueue to account for all resources
> -
>
> Key: YARN-7312
> URL: https://issues.apache.org/jira/browse/YARN-7312
> Project: Hadoop YARN
>  Issue Type: Task
>Affects Versions: 3.1.0
>Reporter: lovekesh bansal
> Fix For: 3.1.0
>
> Attachments: YARN-7312_trunk.001.patch
>
>
> getMaximumAllocationPerQueue has memory and vcores hardcoded. So it should be 
> made generic for all resources.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7312) [Resource Profiles] getMaximumAllocationPerQueue to account for all resources

[lovekesh bansal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

lovekesh bansal updated YARN-7312:
--
Affects Version/s: 3.1.0

> [Resource Profiles] getMaximumAllocationPerQueue to account for all resources
> -
>
> Key: YARN-7312
> URL: https://issues.apache.org/jira/browse/YARN-7312
> Project: Hadoop YARN
>  Issue Type: Task
>Affects Versions: 3.1.0
>Reporter: lovekesh bansal
> Fix For: 3.1.0
>
>
> getMaximumAllocationPerQueue has memory and vcores hardcoded. So it should be 
> made generic for all resources.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7312) [Resource Profiles] getMaximumAllocationPerQueue to account for all resources

[lovekesh bansal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

lovekesh bansal updated YARN-7312:
--
Fix Version/s: 3.1.0

> [Resource Profiles] getMaximumAllocationPerQueue to account for all resources
> -
>
> Key: YARN-7312
> URL: https://issues.apache.org/jira/browse/YARN-7312
> Project: Hadoop YARN
>  Issue Type: Task
>Affects Versions: 3.1.0
>Reporter: lovekesh bansal
> Fix For: 3.1.0
>
>
> getMaximumAllocationPerQueue has memory and vcores hardcoded. So it should be 
> made generic for all resources.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-7312) [Resource Profiles] getMaximumAllocationPerQueue to account for all resources

[lovekesh bansal (JIRA)]

lovekesh bansal created YARN-7312:
-

 Summary: [Resource Profiles] getMaximumAllocationPerQueue to 
account for all resources
 Key: YARN-7312
 URL: https://issues.apache.org/jira/browse/YARN-7312
 Project: Hadoop YARN
  Issue Type: Task
Reporter: lovekesh bansal


getMaximumAllocationPerQueue has memory and vcores hardcoded. So it should be 
made generic for all resources.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-2162) add ability in Fair Scheduler to optionally configure maxResources in terms of percentage

[Yufei Gu (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198176#comment-16198176
 ] 

Yufei Gu commented on YARN-2162:


[~wangda], these will be fixed by YARN-7270.

> add ability in Fair Scheduler to optionally configure maxResources in terms 
> of percentage
> -
>
> Key: YARN-2162
> URL: https://issues.apache.org/jira/browse/YARN-2162
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: fairscheduler, scheduler
>Reporter: Ashwin Shankar
>Assignee: Yufei Gu
>  Labels: scheduler
> Fix For: 2.9.0, 3.0.0, 3.1.0
>
> Attachments: YARN-2162.001.patch, YARN-2162.002.patch, 
> YARN-2162.003.patch, YARN-2162.004.patch, YARN-2162.005.patch, 
> YARN-2162.006.patch, YARN-2162.007.patch, YARN-2162.008.patch, 
> YARN-2162.branch-2.010.patch, YARN-2162.branch-3.0.009.patch, 
> test-400nm-200app-2k_NODE_UPDATE.timecost.svg
>
>
> minResources and maxResources in fair scheduler configs are expressed in 
> terms of absolute numbers X mb, Y vcores. 
> As a result, when we expand or shrink our hadoop cluster, we need to 
> recalculate and change minResources/maxResources accordingly, which is pretty 
> inconvenient.
> We can circumvent this problem if we can optionally configure these 
> properties in terms of percentage of cluster capacity. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7307) Revisit resource-types.xml loading behaviors

[Wangda Tan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wangda Tan updated YARN-7307:
-
Fix Version/s: (was: 3.1.0)

> Revisit resource-types.xml loading behaviors
> 
>
> Key: YARN-7307
> URL: https://issues.apache.org/jira/browse/YARN-7307
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager, resourcemanager
>Reporter: Wangda Tan
>Assignee: Sunil G
>Priority: Blocker
>
> Existing feature requires every client has a resource-types.xml in order to 
> use multiple resource types, should we allow client/AM update supported 
> resource types via Yarn APIs?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7307) Revisit resource-types.xml loading behaviors

[Wangda Tan (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198126#comment-16198126
 ] 

Wangda Tan commented on YARN-7307:
--

[~andrew.wang], since the whole feature isn't backported to 3.0 yet, I would 
keep it blocking 3.1.0 release for now. Once this is committed, we can consider 
to backport the whole feature to 3.0

> Revisit resource-types.xml loading behaviors
> 
>
> Key: YARN-7307
> URL: https://issues.apache.org/jira/browse/YARN-7307
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager, resourcemanager
>Reporter: Wangda Tan
>Assignee: Sunil G
>Priority: Blocker
>
> Existing feature requires every client has a resource-types.xml in order to 
> use multiple resource types, should we allow client/AM update supported 
> resource types via Yarn APIs?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7237) Cleanup usages of ResourceProfiles

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198129#comment-16198129
 ] 

Hudson commented on YARN-7237:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13059 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/13059/])
YARN-7237. Cleanup usages of ResourceProfiles. (Wangda Tan) (wangda: rev 
5cfbad68c0ef6e70d248d4989f9889576d5fcf7e)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/resources/resource-profiles.json
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterService.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestProfileCapability.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNM.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/profiles/sample-profiles-2.json
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/resource/MockResourceProfileManager.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/ResourceProfilesManagerImpl.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestUtils.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ProfileCapability.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/resource/TestResourceProfiles.java
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerWithMultiResourceTypes.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/profiles/sample-profiles-1.json


> Cleanup usages of ResourceProfiles
> --
>
> Key: YARN-7237
> URL: https://issues.apache.org/jira/browse/YARN-7237
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager, resourcemanager
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Critical
> Fix For: 3.1.0
>
> Attachments: YARN-7237.001.patch, YARN-7237.002.patch, 
> YARN-7237.003.patch, YARN-7237.004.patch, YARN-7237.005.patch, 
> YARN-7237.006.patch, YARN-7237.007.patch
>
>
> While doing tests, there're a couple of issues:
> 1) When use {{ProfileCapability#getProfileCapabilityOverride}}, it does 
> overwrite of whatever specified in resource-profiles.json when value >= 0. 
> Which is different from javadocs of {{ProfileCapability}} 
> bq. For example, if you have a resource profile "small" that maps to <4096M, 
> 2 cores, 1 gpu> and you set the capability override to <8192M, 0 cores, 0 
> gpu>, then the actual resource allocation on the ResourceManager will be 
> <8192M, 2 cores, 1 gpu>
> To me, the correct behavior should do overwrite when value > 0. The reason 
> is, by default resource value will be set to 0, For example, assume we have a 
> profile {{"a" = (mem=3, vcore=5, res_1=7)}}, and create a 
> capability-overwrite (capability = new resource(8). The final result should 
> be (mem=8, vcore=5, res_1=7), instead of (mem=8, vcore=0, res_1=0).
> 2) ResourceProfileManager now loads minimum/maximum profile from config file 
> (resource-profiles.json), to me this is not correct because minimum/maximum 
> allocation for each resource types are already specified inside 
> {{resource-types.xml}}. We should always use 
> 

[jira] [Updated] (YARN-7307) Revisit resource-types.xml loading behaviors

[Wangda Tan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wangda Tan updated YARN-7307:
-
Target Version/s: 3.1.0

> Revisit resource-types.xml loading behaviors
> 
>
> Key: YARN-7307
> URL: https://issues.apache.org/jira/browse/YARN-7307
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager, resourcemanager
>Reporter: Wangda Tan
>Assignee: Sunil G
>Priority: Blocker
>
> Existing feature requires every client has a resource-types.xml in order to 
> use multiple resource types, should we allow client/AM update supported 
> resource types via Yarn APIs?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7307) Revisit resource-types.xml loading behaviors

[Wangda Tan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wangda Tan updated YARN-7307:
-
Fix Version/s: 3.1.0

> Revisit resource-types.xml loading behaviors
> 
>
> Key: YARN-7307
> URL: https://issues.apache.org/jira/browse/YARN-7307
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager, resourcemanager
>Reporter: Wangda Tan
>Assignee: Sunil G
>Priority: Blocker
> Fix For: 3.1.0
>
>
> Existing feature requires every client has a resource-types.xml in order to 
> use multiple resource types, should we allow client/AM update supported 
> resource types via Yarn APIs?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6620) [YARN-6223] NM Java side code changes to support isolate GPU devices by using CGroups

[Wangda Tan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wangda Tan updated YARN-6620:
-
Attachment: YARN-6620.017.patch

Attached ver.017 patch fixed unit tests, rest of unit tests should not related. 

I would leave white space warnings as-is because this is directly captured from 
nvidia-smi output. 

> [YARN-6223] NM Java side code changes to support isolate GPU devices by using 
> CGroups
> -
>
> Key: YARN-6620
> URL: https://issues.apache.org/jira/browse/YARN-6620
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-6620.001.patch, YARN-6620.002.patch, 
> YARN-6620.003.patch, YARN-6620.004.patch, YARN-6620.005.patch, 
> YARN-6620.006-WIP.patch, YARN-6620.007.patch, YARN-6620.008.patch, 
> YARN-6620.009.patch, YARN-6620.010.patch, YARN-6620.011.patch, 
> YARN-6620.012.patch, YARN-6620.013.patch, YARN-6620.014.patch, 
> YARN-6620.015.patch, YARN-6620.016.patch, YARN-6620.017.patch
>
>
> This JIRA plan to add support of:
> 1) GPU configuration for NodeManagers
> 2) Isolation in CGroups. (Java side).
> 3) NM restart and recovery allocated GPU devices



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-7237) Cleanup usages of ResourceProfiles

[Wangda Tan (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198111#comment-16198111
 ] 

Wangda Tan edited comment on YARN-7237 at 10/10/17 3:39 AM:


Test failures are caused by YARN-2162.

I just pushed this patch to trunk, thanks [~sunilg] and [~templedf] for review. 
Let's move rest of cleanups to YARN-7292.


was (Author: leftnoteasy):
I just pushed this patch to trunk, thanks [~sunilg] and [~templedf] for review. 
Let's move rest of cleanups to YARN-7292.

> Cleanup usages of ResourceProfiles
> --
>
> Key: YARN-7237
> URL: https://issues.apache.org/jira/browse/YARN-7237
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager, resourcemanager
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Critical
> Fix For: 3.1.0
>
> Attachments: YARN-7237.001.patch, YARN-7237.002.patch, 
> YARN-7237.003.patch, YARN-7237.004.patch, YARN-7237.005.patch, 
> YARN-7237.006.patch, YARN-7237.007.patch
>
>
> While doing tests, there're a couple of issues:
> 1) When use {{ProfileCapability#getProfileCapabilityOverride}}, it does 
> overwrite of whatever specified in resource-profiles.json when value >= 0. 
> Which is different from javadocs of {{ProfileCapability}} 
> bq. For example, if you have a resource profile "small" that maps to <4096M, 
> 2 cores, 1 gpu> and you set the capability override to <8192M, 0 cores, 0 
> gpu>, then the actual resource allocation on the ResourceManager will be 
> <8192M, 2 cores, 1 gpu>
> To me, the correct behavior should do overwrite when value > 0. The reason 
> is, by default resource value will be set to 0, For example, assume we have a 
> profile {{"a" = (mem=3, vcore=5, res_1=7)}}, and create a 
> capability-overwrite (capability = new resource(8). The final result should 
> be (mem=8, vcore=5, res_1=7), instead of (mem=8, vcore=0, res_1=0).
> 2) ResourceProfileManager now loads minimum/maximum profile from config file 
> (resource-profiles.json), to me this is not correct because minimum/maximum 
> allocation for each resource types are already specified inside 
> {{resource-types.xml}}. We should always use 
> {{ResourceUtils#getResourceTypesMinimum/MaximumAllocation}} to get from 
> resource-types.xml and yarn-site.xml. This value will be added to profiles so 
> client can get these configs.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-2162) add ability in Fair Scheduler to optionally configure maxResources in terms of percentage

[Wangda Tan (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198100#comment-16198100
 ] 

Wangda Tan commented on YARN-2162:
--

[~templedf]/[~yufeigu], it looks like this patch broke a lot of unit tests, 
could you check: 
https://issues.apache.org/jira/browse/YARN-2162?focusedCommentId=16193467=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16193467?

> add ability in Fair Scheduler to optionally configure maxResources in terms 
> of percentage
> -
>
> Key: YARN-2162
> URL: https://issues.apache.org/jira/browse/YARN-2162
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: fairscheduler, scheduler
>Reporter: Ashwin Shankar
>Assignee: Yufei Gu
>  Labels: scheduler
> Fix For: 2.9.0, 3.0.0, 3.1.0
>
> Attachments: YARN-2162.001.patch, YARN-2162.002.patch, 
> YARN-2162.003.patch, YARN-2162.004.patch, YARN-2162.005.patch, 
> YARN-2162.006.patch, YARN-2162.007.patch, YARN-2162.008.patch, 
> YARN-2162.branch-2.010.patch, YARN-2162.branch-3.0.009.patch, 
> test-400nm-200app-2k_NODE_UPDATE.timecost.svg
>
>
> minResources and maxResources in fair scheduler configs are expressed in 
> terms of absolute numbers X mb, Y vcores. 
> As a result, when we expand or shrink our hadoop cluster, we need to 
> recalculate and change minResources/maxResources accordingly, which is pretty 
> inconvenient.
> We can circumvent this problem if we can optionally configure these 
> properties in terms of percentage of cluster capacity. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7198) Add jsvc support for RegistryDNS

[Jian He (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198097#comment-16198097
 ] 

Jian He commented on YARN-7198:
---

[~aw], any comments on the patch ?

> Add jsvc support for RegistryDNS
> 
>
> Key: YARN-7198
> URL: https://issues.apache.org/jira/browse/YARN-7198
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn-native-services
>Reporter: Billie Rinaldi
>Assignee: Billie Rinaldi
>Priority: Critical
> Attachments: YARN-7198-yarn-native-services.01.patch, 
> YARN-7198-yarn-native-services.02.patch, 
> YARN-7198-yarn-native-services.03.patch, 
> YARN-7198-yarn-native-services.04.patch, 
> YARN-7198-yarn-native-services.05.patch
>
>
> RegistryDNS should have jsvc support and be managed through the shell 
> scripts, rather than being started manually. See original comments on 
> YARN-7191.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Arun Suresh (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198077#comment-16198077
 ] 

Arun Suresh edited comment on YARN-4511 at 10/10/17 3:08 AM:
-

[~haibochen], took a quick look at the patch.
With regard to the changes to the {{swapContainer}} method, the original 
intent, was as you guessed to swap the internal container objects of two 
RMContainers. Generally, in the context of container updates, the new container 
is killed off immediately after the swap. We assume this swap happens somewhat 
transactional-ly(given the locking order etc.), and allows us to side-step 
having to deal with intermediate events the original RMContainer might have to 
deal with during the update. Also, in the context of container updates, 
generally, a container stays on the same node - so was surprised that you added 
a newNode and oldNode arguments to the method, since I felt it does not make 
sense. Maybe if you could point me to how you are going to ultimately use the 
changes - I might have more context ?


was (Author: asuresh):
[~haibochen], took a quick look at the patch.
With regard to the changes to the {{swapContainer}} method, the original 
intent, was as you guessed to swap the internal container objects of two 
RMContainers. Generally, in the context of container updates, the new container 
is killed off immediately after the swap. We assume this swap happens somewhat 
transactional-ly(given the locking order etc.), and allows us to side-step 
having to deal with intermediate events the original RMContainer might have to 
deal with during the update. Also, in the context of container updates, 
generally, the a container stays on the same node - so was surprised that you 
added a newNode and oldNode arguments to the method, since I felt it does not 
make sense. Maybe if you could point me to how you are going to ultimately use 
the changes - I might have more context ?

> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Arun Suresh (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198077#comment-16198077
 ] 

Arun Suresh commented on YARN-4511:
---

[~haibochen], took a quick look at the patch.
With regard to the changes to the {{swapContainer}} method, the original 
intent, was as you guessed to swap the internal container objects of two 
RMContainers. Generally, in the context of container updates, the new container 
is killed off immediately after the swap. We assume this swap happens somewhat 
transitionally (given the locking order etc.), and allows us to side-step 
having to deal with intermediate events the original RMContainer might have to 
deal with during the update. Also, in the context of container updates, 
generally, the a container stays on the same node - so was surprised that you 
added a newNode and oldNode arguments to the method, since I felt it does not 
make sense. Maybe if you could point me to how you are going to ultimately use 
the changes - I might have more context ?

> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Arun Suresh (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198077#comment-16198077
 ] 

Arun Suresh edited comment on YARN-4511 at 10/10/17 3:07 AM:
-

[~haibochen], took a quick look at the patch.
With regard to the changes to the {{swapContainer}} method, the original 
intent, was as you guessed to swap the internal container objects of two 
RMContainers. Generally, in the context of container updates, the new container 
is killed off immediately after the swap. We assume this swap happens somewhat 
transactional-ly(given the locking order etc.), and allows us to side-step 
having to deal with intermediate events the original RMContainer might have to 
deal with during the update. Also, in the context of container updates, 
generally, the a container stays on the same node - so was surprised that you 
added a newNode and oldNode arguments to the method, since I felt it does not 
make sense. Maybe if you could point me to how you are going to ultimately use 
the changes - I might have more context ?


was (Author: asuresh):
[~haibochen], took a quick look at the patch.
With regard to the changes to the {{swapContainer}} method, the original 
intent, was as you guessed to swap the internal container objects of two 
RMContainers. Generally, in the context of container updates, the new container 
is killed off immediately after the swap. We assume this swap happens somewhat 
transitionally (given the locking order etc.), and allows us to side-step 
having to deal with intermediate events the original RMContainer might have to 
deal with during the update. Also, in the context of container updates, 
generally, the a container stays on the same node - so was surprised that you 
added a newNode and oldNode arguments to the method, since I felt it does not 
make sense. Maybe if you could point me to how you are going to ultimately use 
the changes - I might have more context ?

> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7309) TestClientRMService#testUpdateApplicationPriorityRequest and TestClientRMService#testUpdatePriorityAndKillAppWithZeroClusterResource test functionality not supported by

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198073#comment-16198073
 ] 

Hadoop QA commented on YARN-7309:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
38s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 32m 
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
18m 16s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
56s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
18m 29s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 62m 26s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  1m 
 6s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}151m 17s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter |
|   | 
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer
 |
|   | 
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing |
|   | hadoop.yarn.server.resourcemanager.TestApplicationMasterService |
|   | 
hadoop.yarn.server.resourcemanager.scheduler.fair.TestContinuousScheduling |
|   | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppAttempt |
|   | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler |
|   | hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart |
| Timed out junit tests | 
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationCleanup |
|   | 
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:71bbb86 |
| JIRA Issue | YARN-7309 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12891172/YARN-7309.001.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux a4738d48c3de 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build 

[jira] [Updated] (YARN-7311) TestRMWebServicesReservation doesn't really test fair scheduler

[Yufei Gu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yufei Gu updated YARN-7311:
---
Attachment: YARN-7311.WIP.patch

Post WIP patch. All CS tests pass. All No Auth Fair Scheduler tests pass. Most 
Simple Auth Fair Scheduler failed. 

> TestRMWebServicesReservation doesn't really test fair scheduler
> ---
>
> Key: YARN-7311
> URL: https://issues.apache.org/jira/browse/YARN-7311
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: fairscheduler, reservation system
>Reporter: Yufei Gu
> Attachments: YARN-7311.WIP.patch
>
>
> YARN-4248 introduced the REST API for submit/update/delete Reservations. 
> Class {{TestRMWebServicesReservation}} intends to test both FS and CS. The 
> test cases designed for fair scheduler actually test capacity scheduler. The 
> following code in method {{configureServlets}} shows it sets the scheduler to 
> CS even test cases are for fair scheduler.
> {code}
>   conf.setClass(YarnConfiguration.RM_SCHEDULER, CapacityScheduler.class,
>   ResourceScheduler.class);
>   CapacitySchedulerConfiguration csconf =
>   new CapacitySchedulerConfiguration(conf);
>   String[] queues = { "default", "dedicated" };
>   csconf.setQueues("root", queues);
>   csconf.setCapacity("root.default", 50.0f);
>   csconf.setCapacity("root.dedicated", 50.0f);
>   csconf.setReservable("root.dedicated", true);
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7127) Merge yarn-native-service branch into trunk

[Jian He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jian He updated YARN-7127:
--
Summary: Merge yarn-native-service branch into trunk  (was: Run 
yarn-native-services branch against trunk)

> Merge yarn-native-service branch into trunk
> ---
>
> Key: YARN-7127
> URL: https://issues.apache.org/jira/browse/YARN-7127
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-7127.01.patch, YARN-7127.02.patch, 
> YARN-7127.03.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7310) TestAMRMProxy#testAMRMProxyE2E fails with FairScheduler

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198049#comment-16198049
 ] 

Hadoop QA commented on YARN-7310:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}  
9m 57s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
17s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m 48s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 30m 54s{color} 
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
24s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 73m 41s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.yarn.client.api.impl.TestAMRMClient 
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:71bbb86 |
| JIRA Issue | YARN-7310 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12891176/YARN-7310.001.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 933f75ccd772 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 82cd85d |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| unit | 
https://builds.apache.org/job/PreCommit-YARN-Build/17837/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/17837/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/17837/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> 

[jira] [Commented] (YARN-7308) TestApplicationACLs fails with FairScheduler

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198025#comment-16198025
 ] 

Hadoop QA commented on YARN-7308:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 14m 
50s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}  
9m 50s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m  
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m  5s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 47m 49s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
21s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}103m 53s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.resourcemanager.TestApplicationMasterService |
|   | 
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer
 |
|   | hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter |
|   | hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart |
|   | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler |
|   | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppAttempt |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:71bbb86 |
| JIRA Issue | YARN-7308 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12891169/YARN-7308.001.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 45d08139f63f 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / f9ff17f |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| unit | 

[jira] [Commented] (YARN-7237) Cleanup usages of ResourceProfiles

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198018#comment-16198018
 ] 

Hadoop QA commented on YARN-7237:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 11 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  3m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
14m  0s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  5m 
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
31s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  6m 
51s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
1m  3s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch 
generated 2 new + 183 unchanged - 3 fixed = 185 total (was 186) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  3m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m  1s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  6m 
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
40s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
50s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m  
0s{color} | {color:green} hadoop-yarn-server-common in the patch passed. 
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 52m  0s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 31m 12s{color} 
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
28s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}178m 27s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter |
|   | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler |
|   | hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart |
|   | 

[jira] [Commented] (YARN-7202) End-to-end UT for api-server

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198013#comment-16198013
 ] 

Hadoop QA commented on YARN-7202:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 18m 
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
|| || || || {color:brown} yarn-native-services Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m 
34s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
12s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
10s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
13s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}  
7m 52s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
19s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m  
8s{color} | {color:green} yarn-native-services passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
11s{color} | {color:red} hadoop-yarn-services-api in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  0m 
12s{color} | {color:red} hadoop-yarn-services-api in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  0m 12s{color} 
| {color:red} hadoop-yarn-services-api in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m  8s{color} | {color:orange} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api:
 The patch generated 2 new + 2 unchanged - 1 fixed = 4 total (was 3) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red}  0m 
12s{color} | {color:red} hadoop-yarn-services-api in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}  
8m 42s{color} | {color:green} patch has no errors when building and testing our 
client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
11s{color} | {color:red} hadoop-yarn-services-api in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m  
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  0m 12s{color} 
| {color:red} hadoop-yarn-services-api in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
16s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 50m 46s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:71bbb86 |
| JIRA Issue | YARN-7202 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12891174/YARN-7202.yarn-native-services.010.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  xml  findbugs  checkstyle  |
| uname | Linux 181f123b50eb 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | yarn-native-services / 4993b8a |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| mvninstall | 

[jira] [Commented] (YARN-7202) End-to-end UT for api-server

[Eric Yang (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198012#comment-16198012
 ] 

Eric Yang commented on YARN-7202:
-

[~jianhe] PUT method throws YarnException for both Service Not Found, and 
general Yarn Framework errors in ServiceClient.  When YarnException is surfaced 
to client, it will indicate Service Not Found, in fact, it may have failed due 
to backend errors.  This is the reason that later patch was modified to show 
only BAD_REQUEST or INTERNAL_SERVER_ERROR only to make avoid misdirect user.  I 
revert it back to your original intent.  We can probably throw a file not found 
exception in ServiceClient to make YarnException more precise of the intent.

> End-to-end UT for api-server
> 
>
> Key: YARN-7202
> URL: https://issues.apache.org/jira/browse/YARN-7202
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jian He
>Assignee: Eric Yang
> Attachments: YARN-7202.yarn-native-services.001.patch, 
> YARN-7202.yarn-native-services.002.patch, 
> YARN-7202.yarn-native-services.003.patch, 
> YARN-7202.yarn-native-services.004.patch, 
> YARN-7202.yarn-native-services.005.patch, 
> YARN-7202.yarn-native-services.006.patch, 
> YARN-7202.yarn-native-services.007.patch, 
> YARN-7202.yarn-native-services.008.patch, 
> YARN-7202.yarn-native-services.009.patch, 
> YARN-7202.yarn-native-services.010.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7202) End-to-end UT for api-server

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197999#comment-16197999
 ] 

Hadoop QA commented on YARN-7202:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
|| || || || {color:brown} yarn-native-services Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 
18s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
16s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
14s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
20s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}  
9m 37s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
27s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
14s{color} | {color:green} yarn-native-services passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
14s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m  9s{color} | {color:orange} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api:
 The patch generated 1 new + 4 unchanged - 0 fixed = 5 total (was 4) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 33s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
37s{color} | {color:red} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api
 generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  0m 20s{color} 
| {color:red} hadoop-yarn-services-api in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
25s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 45m 32s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | 
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api
 |
|  |  Write to static field 
org.apache.hadoop.yarn.service.webapp.ApiServer.SERVICE_CLIENT from instance 
method 
org.apache.hadoop.yarn.service.webapp.ApiServer.setServiceClient(ServiceClient) 
 At ApiServer.java:from instance method 
org.apache.hadoop.yarn.service.webapp.ApiServer.setServiceClient(ServiceClient) 
 At ApiServer.java:[line 88] |
| Failed junit tests | hadoop.yarn.service.TestApiServer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:71bbb86 |
| JIRA Issue | YARN-7202 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12891173/YARN-7202.yarn-native-services.009.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  

[jira] [Commented] (YARN-6620) [YARN-6223] NM Java side code changes to support isolate GPU devices by using CGroups

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198004#comment-16198004
 ] 

Hadoop QA commented on YARN-6620:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 15 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m 31s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m  
3s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m 
24s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
1m 25s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch 
generated 83 new + 489 unchanged - 24 fixed = 572 total (was 513) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green}  0m 
 0s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green}  0m 
31s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch 523 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 44s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  5m 
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m  
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
44s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  2m 55s{color} 
| {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 17m  7s{color} 
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
37s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}111m 17s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.util.resource.TestResourceUtils |
|   | hadoop.yarn.server.nodemanager.scheduler.TestDistributedScheduler |
|   | 

[jira] [Updated] (YARN-7311) TestRMWebServicesReservation doesn't really test fair scheduler

[Yufei Gu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yufei Gu updated YARN-7311:
---
Description: 
YARN-4248 introduced the REST API for submit/update/delete Reservations. Class 
{{TestRMWebServicesReservation}} intends to test both FS and CS. The test cases 
designed for fair scheduler actually test capacity scheduler. The following 
code in method {{configureServlets}} shows it set the scheduler to CS even test 
cases are for fair scheduler.
{code}
  conf.setClass(YarnConfiguration.RM_SCHEDULER, CapacityScheduler.class,
  ResourceScheduler.class);
  CapacitySchedulerConfiguration csconf =
  new CapacitySchedulerConfiguration(conf);
  String[] queues = { "default", "dedicated" };
  csconf.setQueues("root", queues);
  csconf.setCapacity("root.default", 50.0f);
  csconf.setCapacity("root.dedicated", 50.0f);
  csconf.setReservable("root.dedicated", true);
{code}

  was:
YARN-4248 introduced the REST API for submit/update/delete Reservations. Class 
{{TestRMWebServicesReservation}} intends to test both FS and CS. The test cases 
designed for fair scheduler test capacity scheduler instead. The following code 
in method {{configureServlets}} shows it set the scheduler to CS even test 
cases are for fair scheduler.
{code}
  conf.setClass(YarnConfiguration.RM_SCHEDULER, CapacityScheduler.class,
  ResourceScheduler.class);
  CapacitySchedulerConfiguration csconf =
  new CapacitySchedulerConfiguration(conf);
  String[] queues = { "default", "dedicated" };
  csconf.setQueues("root", queues);
  csconf.setCapacity("root.default", 50.0f);
  csconf.setCapacity("root.dedicated", 50.0f);
  csconf.setReservable("root.dedicated", true);
{code}


> TestRMWebServicesReservation doesn't really test fair scheduler
> ---
>
> Key: YARN-7311
> URL: https://issues.apache.org/jira/browse/YARN-7311
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: fairscheduler, reservation system
>Reporter: Yufei Gu
>
> YARN-4248 introduced the REST API for submit/update/delete Reservations. 
> Class {{TestRMWebServicesReservation}} intends to test both FS and CS. The 
> test cases designed for fair scheduler actually test capacity scheduler. The 
> following code in method {{configureServlets}} shows it set the scheduler to 
> CS even test cases are for fair scheduler.
> {code}
>   conf.setClass(YarnConfiguration.RM_SCHEDULER, CapacityScheduler.class,
>   ResourceScheduler.class);
>   CapacitySchedulerConfiguration csconf =
>   new CapacitySchedulerConfiguration(conf);
>   String[] queues = { "default", "dedicated" };
>   csconf.setQueues("root", queues);
>   csconf.setCapacity("root.default", 50.0f);
>   csconf.setCapacity("root.dedicated", 50.0f);
>   csconf.setReservable("root.dedicated", true);
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7311) TestRMWebServicesReservation doesn't really test fair scheduler

[Yufei Gu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yufei Gu updated YARN-7311:
---
Description: 
YARN-4248 introduced the REST API for submit/update/delete Reservations. Class 
{{TestRMWebServicesReservation}} intends to test both FS and CS. The test cases 
designed for fair scheduler actually test capacity scheduler. The following 
code in method {{configureServlets}} shows it sets the scheduler to CS even 
test cases are for fair scheduler.
{code}
  conf.setClass(YarnConfiguration.RM_SCHEDULER, CapacityScheduler.class,
  ResourceScheduler.class);
  CapacitySchedulerConfiguration csconf =
  new CapacitySchedulerConfiguration(conf);
  String[] queues = { "default", "dedicated" };
  csconf.setQueues("root", queues);
  csconf.setCapacity("root.default", 50.0f);
  csconf.setCapacity("root.dedicated", 50.0f);
  csconf.setReservable("root.dedicated", true);
{code}

  was:
YARN-4248 introduced the REST API for submit/update/delete Reservations. Class 
{{TestRMWebServicesReservation}} intends to test both FS and CS. The test cases 
designed for fair scheduler actually test capacity scheduler. The following 
code in method {{configureServlets}} shows it set the scheduler to CS even test 
cases are for fair scheduler.
{code}
  conf.setClass(YarnConfiguration.RM_SCHEDULER, CapacityScheduler.class,
  ResourceScheduler.class);
  CapacitySchedulerConfiguration csconf =
  new CapacitySchedulerConfiguration(conf);
  String[] queues = { "default", "dedicated" };
  csconf.setQueues("root", queues);
  csconf.setCapacity("root.default", 50.0f);
  csconf.setCapacity("root.dedicated", 50.0f);
  csconf.setReservable("root.dedicated", true);
{code}


> TestRMWebServicesReservation doesn't really test fair scheduler
> ---
>
> Key: YARN-7311
> URL: https://issues.apache.org/jira/browse/YARN-7311
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: fairscheduler, reservation system
>Reporter: Yufei Gu
>
> YARN-4248 introduced the REST API for submit/update/delete Reservations. 
> Class {{TestRMWebServicesReservation}} intends to test both FS and CS. The 
> test cases designed for fair scheduler actually test capacity scheduler. The 
> following code in method {{configureServlets}} shows it sets the scheduler to 
> CS even test cases are for fair scheduler.
> {code}
>   conf.setClass(YarnConfiguration.RM_SCHEDULER, CapacityScheduler.class,
>   ResourceScheduler.class);
>   CapacitySchedulerConfiguration csconf =
>   new CapacitySchedulerConfiguration(conf);
>   String[] queues = { "default", "dedicated" };
>   csconf.setQueues("root", queues);
>   csconf.setCapacity("root.default", 50.0f);
>   csconf.setCapacity("root.dedicated", 50.0f);
>   csconf.setReservable("root.dedicated", true);
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7311) TestRMWebServicesReservation doesn't really test fair scheduler

[Yufei Gu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yufei Gu updated YARN-7311:
---
Description: 
YARN-4248 introduced the REST API for submit/update/delete Reservations. Class 
{{TestRMWebServicesReservation}} intends to test both FS and CS. The test cases 
designed for fair scheduler test capacity scheduler instead. The following code 
in method {{configureServlets}} shows it set the scheduler to CS even test 
cases are for fair scheduler.
{code}
  conf.setClass(YarnConfiguration.RM_SCHEDULER, CapacityScheduler.class,
  ResourceScheduler.class);
  CapacitySchedulerConfiguration csconf =
  new CapacitySchedulerConfiguration(conf);
  String[] queues = { "default", "dedicated" };
  csconf.setQueues("root", queues);
  csconf.setCapacity("root.default", 50.0f);
  csconf.setCapacity("root.dedicated", 50.0f);
  csconf.setReservable("root.dedicated", true);
{code}

  was:
Class {{TestRMWebServicesReservation}} intends to test both FS and CS. The test 
cases designed for fair scheduler test capacity scheduler instead. The 
following code in method {{configureServlets}} shows it set the scheduler to CS 
even test cases are for fair scheduler.
{code}
  conf.setClass(YarnConfiguration.RM_SCHEDULER, CapacityScheduler.class,
  ResourceScheduler.class);
  CapacitySchedulerConfiguration csconf =
  new CapacitySchedulerConfiguration(conf);
  String[] queues = { "default", "dedicated" };
  csconf.setQueues("root", queues);
  csconf.setCapacity("root.default", 50.0f);
  csconf.setCapacity("root.dedicated", 50.0f);
  csconf.setReservable("root.dedicated", true);
{code}


> TestRMWebServicesReservation doesn't really test fair scheduler
> ---
>
> Key: YARN-7311
> URL: https://issues.apache.org/jira/browse/YARN-7311
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: fairscheduler, reservation system
>Reporter: Yufei Gu
>
> YARN-4248 introduced the REST API for submit/update/delete Reservations. 
> Class {{TestRMWebServicesReservation}} intends to test both FS and CS. The 
> test cases designed for fair scheduler test capacity scheduler instead. The 
> following code in method {{configureServlets}} shows it set the scheduler to 
> CS even test cases are for fair scheduler.
> {code}
>   conf.setClass(YarnConfiguration.RM_SCHEDULER, CapacityScheduler.class,
>   ResourceScheduler.class);
>   CapacitySchedulerConfiguration csconf =
>   new CapacitySchedulerConfiguration(conf);
>   String[] queues = { "default", "dedicated" };
>   csconf.setQueues("root", queues);
>   csconf.setCapacity("root.default", 50.0f);
>   csconf.setCapacity("root.dedicated", 50.0f);
>   csconf.setReservable("root.dedicated", true);
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-7311) TestRMWebServicesReservation doesn't really test fair scheduler

[Yufei Gu (JIRA)]

Yufei Gu created YARN-7311:
--

 Summary: TestRMWebServicesReservation doesn't really test fair 
scheduler
 Key: YARN-7311
 URL: https://issues.apache.org/jira/browse/YARN-7311
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: fairscheduler, reservation system
Reporter: Yufei Gu


Class {{TestRMWebServicesReservation}} intends to test both FS and CS. The test 
cases designed for fair scheduler test capacity scheduler instead. The 
following code in method {{configureServlets}} shows it set the scheduler to CS 
even test cases are for fair scheduler.
{code}
  conf.setClass(YarnConfiguration.RM_SCHEDULER, CapacityScheduler.class,
  ResourceScheduler.class);
  CapacitySchedulerConfiguration csconf =
  new CapacitySchedulerConfiguration(conf);
  String[] queues = { "default", "dedicated" };
  csconf.setQueues("root", queues);
  csconf.setCapacity("root.default", 50.0f);
  csconf.setCapacity("root.dedicated", 50.0f);
  csconf.setReservable("root.dedicated", true);
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197989#comment-16197989
 ] 

Hadoop QA commented on YARN-4511:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 20m  
7s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 6 new or modified test 
files. {color} |
|| || || || {color:brown} YARN-1011 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  5m 
58s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 
40s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m 
40s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
26s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
21s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
16m 12s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
2s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
1s{color} | {color:green} YARN-1011 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m 
55s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
2m 22s{color} | {color:orange} root: The patch generated 4 new + 565 unchanged 
- 2 fixed = 569 total (was 567) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 17s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m 
45s{color} | {color:red} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 generated 8 new + 0 unchanged - 0 fixed = 8 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
43s{color} | {color:red} 
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager
 generated 2 new + 297 unchanged - 0 fixed = 299 total (was 297) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 51m 11s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  1m 57s{color} 
| {color:red} hadoop-sls in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
50s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}189m 36s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | 
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 |
|  |  Increment of volatile field 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode.numGuaranteedContainers
 in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode.allocateContainer(RMContainer,
 boolean)  At SchedulerNode.java:in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode.allocateContainer(RMContainer,
 boolean)  At SchedulerNode.java:[line 182] |
|  

[jira] [Commented] (YARN-7307) Revisit resource-types.xml loading behaviors

[Andrew Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197973#comment-16197973
 ] 

Andrew Wang commented on YARN-7307:
---

Hi Wangda, Sunil, should this be targeted at 3.0.0?

> Revisit resource-types.xml loading behaviors
> 
>
> Key: YARN-7307
> URL: https://issues.apache.org/jira/browse/YARN-7307
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager, resourcemanager
>Reporter: Wangda Tan
>Assignee: Sunil G
>Priority: Blocker
>
> Existing feature requires every client has a resource-types.xml in order to 
> use multiple resource types, should we allow client/AM update supported 
> resource types via Yarn APIs?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7310) TestAMRMProxy#testAMRMProxyE2E fails with FairScheduler

[Robert Kanter (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter updated YARN-7310:

Attachment: YARN-7310.001.patch

The test was looking for 2 containers to be assigned, but only 1 was because 
FairScheduler only does one container per heartbeat unless you turn on "assign 
multiple", which the 001 patch does to fix it.

> TestAMRMProxy#testAMRMProxyE2E fails with FairScheduler
> ---
>
> Key: YARN-7310
> URL: https://issues.apache.org/jira/browse/YARN-7310
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: test
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: YARN-7310.001.patch
>
>
> {{TestAMRMProxy#testAMRMProxyE2E}} fails with FairScheduler:
> {noformat}
> [ERROR] 
> testAMRMProxyE2E(org.apache.hadoop.yarn.client.api.impl.TestAMRMProxy)  Time 
> elapsed: 29.047 s  <<< FAILURE!
> java.lang.AssertionError: expected:<2> but was:<1>
>   at 
> org.apache.hadoop.yarn.client.api.impl.TestAMRMProxy.testAMRMProxyE2E(TestAMRMProxy.java:124)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-7310) TestAMRMProxy#testAMRMProxyE2E fails with FairScheduler

[Robert Kanter (JIRA)]

Robert Kanter created YARN-7310:
---

 Summary: TestAMRMProxy#testAMRMProxyE2E fails with FairScheduler
 Key: YARN-7310
 URL: https://issues.apache.org/jira/browse/YARN-7310
 Project: Hadoop YARN
  Issue Type: Bug
  Components: test
Reporter: Robert Kanter
Assignee: Robert Kanter


{{TestAMRMProxy#testAMRMProxyE2E}} fails with FairScheduler:
{noformat}
[ERROR] testAMRMProxyE2E(org.apache.hadoop.yarn.client.api.impl.TestAMRMProxy)  
Time elapsed: 29.047 s  <<< FAILURE!
java.lang.AssertionError: expected:<2> but was:<1>
at 
org.apache.hadoop.yarn.client.api.impl.TestAMRMProxy.testAMRMProxyE2E(TestAMRMProxy.java:124)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7202) End-to-end UT for api-server

[Eric Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Yang updated YARN-7202:

Attachment: YARN-7202.yarn-native-services.010.patch

> End-to-end UT for api-server
> 
>
> Key: YARN-7202
> URL: https://issues.apache.org/jira/browse/YARN-7202
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jian He
>Assignee: Eric Yang
> Attachments: YARN-7202.yarn-native-services.001.patch, 
> YARN-7202.yarn-native-services.002.patch, 
> YARN-7202.yarn-native-services.003.patch, 
> YARN-7202.yarn-native-services.004.patch, 
> YARN-7202.yarn-native-services.005.patch, 
> YARN-7202.yarn-native-services.006.patch, 
> YARN-7202.yarn-native-services.007.patch, 
> YARN-7202.yarn-native-services.008.patch, 
> YARN-7202.yarn-native-services.009.patch, 
> YARN-7202.yarn-native-services.010.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-7202) End-to-end UT for api-server

[Eric Yang (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197942#comment-16197942
 ] 

Eric Yang edited comment on YARN-7202 at 10/10/17 12:35 AM:


After discussion with Jian, we agree to revert back to Patch 03, and removed 
changes in PUT method.  This unit test case only contains negative test cases 
to simulate exception that will throw from ServiceClient.  There is no full 
integration test case for positive case.  The positive integration test will be 
introduced in YARN-7215.


was (Author: eyang):
After discussion with Jian, we agree to revert back to Patch 03, and removed 
changes in PUT method.  This unit test case only contains negative test cases 
to simulate exception that will throw from ServiceClient.  There is no full 
integration test case for positive case.  The positive integration test will be 
included in YARN-7015.

> End-to-end UT for api-server
> 
>
> Key: YARN-7202
> URL: https://issues.apache.org/jira/browse/YARN-7202
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jian He
>Assignee: Eric Yang
> Attachments: YARN-7202.yarn-native-services.001.patch, 
> YARN-7202.yarn-native-services.002.patch, 
> YARN-7202.yarn-native-services.003.patch, 
> YARN-7202.yarn-native-services.004.patch, 
> YARN-7202.yarn-native-services.005.patch, 
> YARN-7202.yarn-native-services.006.patch, 
> YARN-7202.yarn-native-services.007.patch, 
> YARN-7202.yarn-native-services.008.patch, 
> YARN-7202.yarn-native-services.009.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7202) End-to-end UT for api-server

[Eric Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Yang updated YARN-7202:

Attachment: YARN-7202.yarn-native-services.009.patch

After discussion with Jian, we agree to revert back to Patch 03, and removed 
changes in PUT method.  This unit test case only contains negative test cases 
to simulate exception that will throw from ServiceClient.  There is no full 
integration test case for positive case.  The positive integration test will be 
included in YARN-7015.

> End-to-end UT for api-server
> 
>
> Key: YARN-7202
> URL: https://issues.apache.org/jira/browse/YARN-7202
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jian He
>Assignee: Eric Yang
> Attachments: YARN-7202.yarn-native-services.001.patch, 
> YARN-7202.yarn-native-services.002.patch, 
> YARN-7202.yarn-native-services.003.patch, 
> YARN-7202.yarn-native-services.004.patch, 
> YARN-7202.yarn-native-services.005.patch, 
> YARN-7202.yarn-native-services.006.patch, 
> YARN-7202.yarn-native-services.007.patch, 
> YARN-7202.yarn-native-services.008.patch, 
> YARN-7202.yarn-native-services.009.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7309) TestClientRMService#testUpdateApplicationPriorityRequest and TestClientRMService#testUpdatePriorityAndKillAppWithZeroClusterResource test functionality not supported by Fa

[Robert Kanter (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter updated YARN-7309:

Attachment: YARN-7309.001.patch

Simple patch to skip the tests with a clear message when using FairScheduler.

> TestClientRMService#testUpdateApplicationPriorityRequest and 
> TestClientRMService#testUpdatePriorityAndKillAppWithZeroClusterResource test 
> functionality not supported by FairScheduler
> --
>
> Key: YARN-7309
> URL: https://issues.apache.org/jira/browse/YARN-7309
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: test
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: YARN-7309.001.patch
>
>
> {{TestClientRMService#testUpdateApplicationPriorityRequest}} and 
> {{TestClientRMService#testUpdatePriorityAndKillAppWithZeroClusterResource}} 
> test functionality (i.e. Application Priorities) not supported by 
> FairScheduler.  We should skip these two tests when using FairScheduler or 
> they'll fail.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-7309) TestClientRMService#testUpdateApplicationPriorityRequest and TestClientRMService#testUpdatePriorityAndKillAppWithZeroClusterResource test functionality not supported by Fa

[Robert Kanter (JIRA)]

Robert Kanter created YARN-7309:
---

 Summary: TestClientRMService#testUpdateApplicationPriorityRequest 
and TestClientRMService#testUpdatePriorityAndKillAppWithZeroClusterResource 
test functionality not supported by FairScheduler
 Key: YARN-7309
 URL: https://issues.apache.org/jira/browse/YARN-7309
 Project: Hadoop YARN
  Issue Type: Bug
  Components: test
Reporter: Robert Kanter
Assignee: Robert Kanter


{{TestClientRMService#testUpdateApplicationPriorityRequest}} and 
{{TestClientRMService#testUpdatePriorityAndKillAppWithZeroClusterResource}} 
test functionality (i.e. Application Priorities) not supported by 
FairScheduler.  We should skip these two tests when using FairScheduler or 
they'll fail.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7308) TestApplicationACLs fails with FairScheduler

[Robert Kanter (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter updated YARN-7308:

Attachment: YARN-7308.001.patch

The 001 patch configures a queue allocations file when using FairScheduler, 
along with a couple other minor changes.

> TestApplicationACLs fails with FairScheduler
> 
>
> Key: YARN-7308
> URL: https://issues.apache.org/jira/browse/YARN-7308
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: test
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: YARN-7308.001.patch
>
>
> {{TestApplicationACLs}} fails when using FairScheduler:
> {noformat}
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 98.389 sec 
> <<< FAILURE! - in 
> org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
> testApplicationACLs(org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs)
>   Time elapsed: 94.563 sec  <<< FAILURE!
> java.lang.AssertionError: App State is not correct (timeout). 
> expected: but was:
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:284)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs.verifyInvalidQueueWithAcl(TestApplicationACLs.java:422)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs.testApplicationACLs(TestApplicationACLs.java:186)
> {noformat}
> There's a bunch of messages like this in the output:
> {noformat}
> 2017-10-09 17:00:54,572 INFO  [main] resourcemanager.MockRM 
> (MockRM.java:waitForState(277)) - App : application_1507593559080_0006 State 
> is : ACCEPTED Waiting for state : FAILED
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-7308) TestApplicationACLs fails with FairScheduler

[Robert Kanter (JIRA)]

Robert Kanter created YARN-7308:
---

 Summary: TestApplicationACLs fails with FairScheduler
 Key: YARN-7308
 URL: https://issues.apache.org/jira/browse/YARN-7308
 Project: Hadoop YARN
  Issue Type: Bug
  Components: test
Reporter: Robert Kanter
Assignee: Robert Kanter


{{TestApplicationACLs}} fails when using FairScheduler:
{noformat}
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 98.389 sec <<< 
FAILURE! - in org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
testApplicationACLs(org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs)
  Time elapsed: 94.563 sec  <<< FAILURE!
java.lang.AssertionError: App State is not correct (timeout). expected: 
but was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at 
org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:284)
at 
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs.verifyInvalidQueueWithAcl(TestApplicationACLs.java:422)
at 
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs.testApplicationACLs(TestApplicationACLs.java:186)
{noformat}

There's a bunch of messages like this in the output:
{noformat}
2017-10-09 17:00:54,572 INFO  [main] resourcemanager.MockRM 
(MockRM.java:waitForState(277)) - App : application_1507593559080_0006 State is 
: ACCEPTED Waiting for state : FAILED
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6620) [YARN-6223] NM Java side code changes to support isolate GPU devices by using CGroups

[Wangda Tan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wangda Tan updated YARN-6620:
-
Attachment: YARN-6620.016.patch

> [YARN-6223] NM Java side code changes to support isolate GPU devices by using 
> CGroups
> -
>
> Key: YARN-6620
> URL: https://issues.apache.org/jira/browse/YARN-6620
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-6620.001.patch, YARN-6620.002.patch, 
> YARN-6620.003.patch, YARN-6620.004.patch, YARN-6620.005.patch, 
> YARN-6620.006-WIP.patch, YARN-6620.007.patch, YARN-6620.008.patch, 
> YARN-6620.009.patch, YARN-6620.010.patch, YARN-6620.011.patch, 
> YARN-6620.012.patch, YARN-6620.013.patch, YARN-6620.014.patch, 
> YARN-6620.015.patch, YARN-6620.016.patch
>
>
> This JIRA plan to add support of:
> 1) GPU configuration for NodeManagers
> 2) Isolation in CGroups. (Java side).
> 3) NM restart and recovery allocated GPU devices



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-3661) Basic Federation UI

[JIRA]

[ 
https://issues.apache.org/jira/browse/YARN-3661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197863#comment-16197863
 ] 

Íñigo Goiri edited comment on YARN-3661 at 10/9/17 11:28 PM:
-

[~giovanni.fumarola], what is the situation with the timeout in branch-2?
I'm also getting the error with:
{code}
ClientResponse response =
performCall(RMWSConsts.RM_WEB_SERVICE_PATH + RMWSConsts.SCHEDULER_LOGS,
RMWSConsts.TIME, "1", null, HTTPMethods.POST);
{code}
As it does not return {{STATUS_BADREQUEST}}.

Not sure this has anything to do with the patch.
Do you mind double checking?

The test seems to fail in branch-2 too.


was (Author: elgoiri):
[~giovanni.fumarola], what is the situation with the timeout in branch-2?
I'm also getting the error with:
{code}
ClientResponse response =
performCall(RMWSConsts.RM_WEB_SERVICE_PATH + RMWSConsts.SCHEDULER_LOGS,
RMWSConsts.TIME, "1", null, HTTPMethods.POST);
{code}
As it does not return {{STATUS_BADREQUEST}}.

Not sure this has anything to do with the patch.
Do you mind double checking?

> Basic Federation UI 
> 
>
> Key: YARN-3661
> URL: https://issues.apache.org/jira/browse/YARN-3661
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager, resourcemanager
>Reporter: Giovanni Matteo Fumarola
>Assignee: Íñigo Goiri
> Attachments: YARN-3661-000.patch, YARN-3661-001.patch, 
> YARN-3661-002.patch, YARN-3661-003.patch, YARN-3661-004.patch, 
> YARN-3661-005.patch, YARN-3661-006.patch, YARN-3661-007.patch, 
> YARN-3661-008.patch, YARN-3661-009.patch, YARN-3661-010.patch, 
> YARN-3661-011.patch, YARN-3661-012.patch, YARN-3661-013.patch, 
> YARN-3661-014.patch, YARN-3661-015.patch, YARN-3661-branch-2.001.patch
>
>
> The UIs provided by each RM, provide a correct "local" view of what is 
> running in a sub-cluster. In the context of federation we need new 
> UIs that can track load, jobs, users across sub-clusters.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-3661) Basic Federation UI

[JIRA]

[ 
https://issues.apache.org/jira/browse/YARN-3661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197863#comment-16197863
 ] 

Íñigo Goiri commented on YARN-3661:
---

[~giovanni.fumarola], what is the situation with the timeout in branch-2?
I'm also getting the error with:
{code}
ClientResponse response =
performCall(RMWSConsts.RM_WEB_SERVICE_PATH + RMWSConsts.SCHEDULER_LOGS,
RMWSConsts.TIME, "1", null, HTTPMethods.POST);
{code}
As it does not return {{STATUS_BADREQUEST}}.

Not sure this has anything to do with the patch.
Do you mind double checking?

> Basic Federation UI 
> 
>
> Key: YARN-3661
> URL: https://issues.apache.org/jira/browse/YARN-3661
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager, resourcemanager
>Reporter: Giovanni Matteo Fumarola
>Assignee: Íñigo Goiri
> Attachments: YARN-3661-000.patch, YARN-3661-001.patch, 
> YARN-3661-002.patch, YARN-3661-003.patch, YARN-3661-004.patch, 
> YARN-3661-005.patch, YARN-3661-006.patch, YARN-3661-007.patch, 
> YARN-3661-008.patch, YARN-3661-009.patch, YARN-3661-010.patch, 
> YARN-3661-011.patch, YARN-3661-012.patch, YARN-3661-013.patch, 
> YARN-3661-014.patch, YARN-3661-015.patch, YARN-3661-branch-2.001.patch
>
>
> The UIs provided by each RM, provide a correct "local" view of what is 
> running in a sub-cluster. In the context of federation we need new 
> UIs that can track load, jobs, users across sub-clusters.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7202) End-to-end UT for api-server

[Jian He (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197860#comment-16197860
 ] 

Jian He commented on YARN-7202:
---

confused, isn't this code not part of this patch ? I did clean before build.
Can you try compile only this patch with the dependency removed? 
{code}
[ERROR] 
/home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api/src/test/java/org/apache/hadoop/yarn/service/ServiceClientTest.java:[29,45]
 cannot find symbol
[ERROR]   symbol:   class YarnSolrClientTest
{code}

bq.  The new code enables the logic to be handled more accurately and provide 
better status report to down stream program weather the failure is caused by a 
bad request, record not found, or undetermined server errors. The previous 
implementation does not allow certain error to be surfaced correctly.
Confused. I'm seeing the opposite. 
Before the patch, it directly returns the status code based on whatever failure 
conditions it is inside the method which is accurate.
After the patch, it only returns INTERNAL_SERVER_ERROR or BAD_REQUEST.  e.g. 
the NOT_FOUND  status code for stopService method is instead converted to the 
BAD_REQUEST status code.  Why is this converstion needed? 

> End-to-end UT for api-server
> 
>
> Key: YARN-7202
> URL: https://issues.apache.org/jira/browse/YARN-7202
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jian He
>Assignee: Eric Yang
> Attachments: YARN-7202.yarn-native-services.001.patch, 
> YARN-7202.yarn-native-services.002.patch, 
> YARN-7202.yarn-native-services.003.patch, 
> YARN-7202.yarn-native-services.004.patch, 
> YARN-7202.yarn-native-services.005.patch, 
> YARN-7202.yarn-native-services.006.patch, 
> YARN-7202.yarn-native-services.007.patch, 
> YARN-7202.yarn-native-services.008.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7286) Add support for docker to have no capabilities

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197816#comment-16197816
 ] 

Hadoop QA commented on YARN-7286:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  4m 
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
50s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m  1s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
18s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  6m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m 24s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
45s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 48s{color} 
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
32s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 96m 59s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.nodemanager.scheduler.TestDistributedScheduler |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:71bbb86 |
| JIRA Issue | YARN-7286 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12891124/YARN-7286.003.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  xml  findbugs  checkstyle  |
| uname | Linux b196c7a505a2 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 7938208 |
| Default Java | 

[jira] [Commented] (YARN-3661) Basic Federation UI

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-3661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197812#comment-16197812
 ] 

Hadoop QA commented on YARN-3661:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 12m 
51s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
 9s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
44s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
48s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
41s{color} | {color:green} branch-2 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
37s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
28s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  2m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  1m  8s{color} 
| {color:red} hadoop-yarn-server-router in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m  
9s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
23s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 42m 11s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.router.webapp.TestRouterWebServicesREST |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:eaf5c66 |
| JIRA Issue | YARN-3661 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12891138/YARN-3661-branch-2.001.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 68b4c4ab53cb 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | branch-2 / 

[jira] [Commented] (YARN-7307) Revisit resource-types.xml loading behaviors

[Wangda Tan (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197807#comment-16197807
 ] 

Wangda Tan commented on YARN-7307:
--

Separated this JIRA from YARN-7292, [~sunilg] mentioned that he's working on 
this, so assign to Sunil.

> Revisit resource-types.xml loading behaviors
> 
>
> Key: YARN-7307
> URL: https://issues.apache.org/jira/browse/YARN-7307
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager, resourcemanager
>Reporter: Wangda Tan
>Assignee: Sunil G
>Priority: Blocker
>
> Existing feature requires every client has a resource-types.xml in order to 
> use multiple resource types, should we allow client/AM update supported 
> resource types via Yarn APIs?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7292) Revisit Resource Profile Behavior

[Wangda Tan (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197805#comment-16197805
 ] 

Wangda Tan commented on YARN-7292:
--

Moved resource-types.xml related behavior changes to YARN-7307

> Revisit Resource Profile Behavior
> -
>
> Key: YARN-7292
> URL: https://issues.apache.org/jira/browse/YARN-7292
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager, resourcemanager
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Blocker
>
> Had discussions with [~templedf], [~vvasudev], [~sunilg] offline. There're a 
> couple of resource profile related behaviors might need to be updated:
> 1) Configure resource profile in server side or client side: 
> Currently resource profile can be only configured centrally:
> - Advantages:
> A given resource profile has a the same meaning in the cluster. It won’t 
> change when we run different apps in different configurations. A job can run 
> under Amazon’s G2.8X can also run on YARN with G2.8X profile. A side benefit 
> is YARN scheduler can potentially do better bin packing.
> - Disadvantages: 
> Hard for applications to add their own resource profiles. 
> 2) Do we really need mandatory resource profiles such as 
> minimum/maximum/default? 
> 3) Should we send resource profile name inside ResourceRequest, or should 
> client/AM translate it to resource and set it to the existing resource 
> fields? 
> 4) Related to above, should we allow resource overrides or client/AM should 
> send final resource to RM?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7292) Revisit Resource Profile Behavior

[Wangda Tan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wangda Tan updated YARN-7292:
-
Description: 
Had discussions with [~templedf], [~vvasudev], [~sunilg] offline. There're a 
couple of resource profile related behaviors might need to be updated:

1) Configure resource profile in server side or client side: 

Currently resource profile can be only configured centrally:
- Advantages:
A given resource profile has a the same meaning in the cluster. It won’t change 
when we run different apps in different configurations. A job can run under 
Amazon’s G2.8X can also run on YARN with G2.8X profile. A side benefit is YARN 
scheduler can potentially do better bin packing.
- Disadvantages: 
Hard for applications to add their own resource profiles. 

2) Do we really need mandatory resource profiles such as 
minimum/maximum/default? 

3) Should we send resource profile name inside ResourceRequest, or should 
client/AM translate it to resource and set it to the existing resource fields? 

4) Related to above, should we allow resource overrides or client/AM should 
send final resource to RM?

  was:
Had discussions with [~templedf], [~vvasudev], [~sunilg] offline. There're a 
couple of resource profile related behaviors might need to be updated:

1) Configure resource profile in server side or client side: 

Currently resource profile can be only configured centrally:
- Advantages:
A given resource profile has a the same meaning in the cluster. It won’t change 
when we run different apps in different configurations. A job can run under 
Amazon’s G2.8X can also run on YARN with G2.8X profile. A side benefit is YARN 
scheduler can potentially do better bin packing.
- Disadvantages: 
Hard for applications to add their own resource profiles. 

2) Do we really need mandatory resource profiles such as 
minimum/maximum/default? 

3) Existing feature requires every client has a resource-types.xml in order to 
use multiple resource types, should we allow client/AM update supported 
resource types via Yarn APIs?

4) Should we send resource profile name inside ResourceRequest, or should 
client/AM translate it to resource and set it to the existing resource fields? 

5) Related to above, should we allow resource overrides or client/AM should 
send final resource to RM?


> Revisit Resource Profile Behavior
> -
>
> Key: YARN-7292
> URL: https://issues.apache.org/jira/browse/YARN-7292
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager, resourcemanager
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Blocker
>
> Had discussions with [~templedf], [~vvasudev], [~sunilg] offline. There're a 
> couple of resource profile related behaviors might need to be updated:
> 1) Configure resource profile in server side or client side: 
> Currently resource profile can be only configured centrally:
> - Advantages:
> A given resource profile has a the same meaning in the cluster. It won’t 
> change when we run different apps in different configurations. A job can run 
> under Amazon’s G2.8X can also run on YARN with G2.8X profile. A side benefit 
> is YARN scheduler can potentially do better bin packing.
> - Disadvantages: 
> Hard for applications to add their own resource profiles. 
> 2) Do we really need mandatory resource profiles such as 
> minimum/maximum/default? 
> 3) Should we send resource profile name inside ResourceRequest, or should 
> client/AM translate it to resource and set it to the existing resource 
> fields? 
> 4) Related to above, should we allow resource overrides or client/AM should 
> send final resource to RM?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-3661) Basic Federation UI

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-3661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197804#comment-16197804
 ] 

Hadoop QA commented on YARN-3661:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 17m 
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
33s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
32s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
42s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
40s{color} | {color:green} branch-2 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
34s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
25s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  2m 
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  1m  1s{color} 
| {color:red} hadoop-yarn-server-router in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
10s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
24s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 44m 47s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.router.webapp.TestRouterWebServicesREST |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:eaf5c66 |
| JIRA Issue | YARN-3661 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12891138/YARN-3661-branch-2.001.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux b69ffa531afe 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | branch-2 / 

[jira] [Created] (YARN-7307) Revisit resource-types.xml loading behaviors

[Wangda Tan (JIRA)]

Wangda Tan created YARN-7307:


 Summary: Revisit resource-types.xml loading behaviors
 Key: YARN-7307
 URL: https://issues.apache.org/jira/browse/YARN-7307
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Wangda Tan
Assignee: Sunil G


Existing feature requires every client has a resource-types.xml in order to use 
multiple resource types, should we allow client/AM update supported resource 
types via Yarn APIs?




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7307) Revisit resource-types.xml loading behaviors

[Wangda Tan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wangda Tan updated YARN-7307:
-
Priority: Blocker  (was: Major)

> Revisit resource-types.xml loading behaviors
> 
>
> Key: YARN-7307
> URL: https://issues.apache.org/jira/browse/YARN-7307
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager, resourcemanager
>Reporter: Wangda Tan
>Assignee: Sunil G
>Priority: Blocker
>
> Existing feature requires every client has a resource-types.xml in order to 
> use multiple resource types, should we allow client/AM update supported 
> resource types via Yarn APIs?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6620) [YARN-6223] NM Java side code changes to support isolate GPU devices by using CGroups

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197798#comment-16197798
 ] 

Hadoop QA commented on YARN-6620:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red}  0m  5s{color} 
| {color:red} YARN-6620 does not apply to trunk. Rebase required? Wrong Branch? 
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-6620 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12891148/YARN-6620.015.patch |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/17830/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> [YARN-6223] NM Java side code changes to support isolate GPU devices by using 
> CGroups
> -
>
> Key: YARN-6620
> URL: https://issues.apache.org/jira/browse/YARN-6620
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-6620.001.patch, YARN-6620.002.patch, 
> YARN-6620.003.patch, YARN-6620.004.patch, YARN-6620.005.patch, 
> YARN-6620.006-WIP.patch, YARN-6620.007.patch, YARN-6620.008.patch, 
> YARN-6620.009.patch, YARN-6620.010.patch, YARN-6620.011.patch, 
> YARN-6620.012.patch, YARN-6620.013.patch, YARN-6620.014.patch, 
> YARN-6620.015.patch
>
>
> This JIRA plan to add support of:
> 1) GPU configuration for NodeManagers
> 2) Isolation in CGroups. (Java side).
> 3) NM restart and recovery allocated GPU devices



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6620) [YARN-6223] NM Java side code changes to support isolate GPU devices by using CGroups

[Wangda Tan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wangda Tan updated YARN-6620:
-
Attachment: YARN-6620.015.patch

Attached ver.015 patch. Major change is:
- Fixed a bug in 014 which prevent GPU resources added to resource-types.xml
- Update resource mapping of container once resources allocated, added unit 
test for this as well.

> [YARN-6223] NM Java side code changes to support isolate GPU devices by using 
> CGroups
> -
>
> Key: YARN-6620
> URL: https://issues.apache.org/jira/browse/YARN-6620
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-6620.001.patch, YARN-6620.002.patch, 
> YARN-6620.003.patch, YARN-6620.004.patch, YARN-6620.005.patch, 
> YARN-6620.006-WIP.patch, YARN-6620.007.patch, YARN-6620.008.patch, 
> YARN-6620.009.patch, YARN-6620.010.patch, YARN-6620.011.patch, 
> YARN-6620.012.patch, YARN-6620.013.patch, YARN-6620.014.patch, 
> YARN-6620.015.patch
>
>
> This JIRA plan to add support of:
> 1) GPU configuration for NodeManagers
> 2) Isolation in CGroups. (Java side).
> 3) NM restart and recovery allocated GPU devices



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7251) Misc changes to YARN-5734

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197791#comment-16197791
 ] 

Hudson commented on YARN-7251:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13057 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/13057/])
YARN-7251. Misc changes to YARN-5734 (jhung: rev 
09c5dfe937f0570cd9494b34d210df2d5f0737a7)
* (edit) hadoop-yarn-project/hadoop-yarn/bin/yarn
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestZKConfigurationStore.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesConfigurationMutation.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/SchedConfCLI.java
* (edit) hadoop-yarn-project/hadoop-yarn/bin/yarn.cmd
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnCommands.md
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestLeveldbConfigurationStore.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/dao/SchedConfUpdateInfo.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestSchedConfCLI.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestMutableCSConfigurationProvider.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/dao/QueueConfigInfo.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerRest.md
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/MutableCSConfigurationProvider.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java


> Misc changes to YARN-5734
> -
>
> Key: YARN-7251
> URL: https://issues.apache.org/jira/browse/YARN-7251
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
> Fix For: YARN-5734
>
> Attachments: YARN-7251-YARN-5734.001.patch, 
> YARN-7251-YARN-5734.002.patch, YARN-7251-YARN-5734.003-v2.patch, 
> YARN-7251-YARN-5734.004-v2.patch, YARN-7251-YARN-5734.005-v2.patch, 
> YARN-7251-YARN-5734.006-final.patch
>
>
> Documentation/style changes to YARN-5734 before merge.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6575) Support global configuration mutation in MutableConfProvider

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197783#comment-16197783
 ] 

Hudson commented on YARN-6575:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13057 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/13057/])
YARN-6575. Support global configuration mutation in MutableConfProvider. 
(jhung: rev e566fd8b58a6570593b21597aa5294d7b37cb388)
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/SchedConfUpdateInfo.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/MutableCSConfigurationProvider.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestConfigurationMutationACLPolicies.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/MutableConfScheduler.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/MutableConfigurationProvider.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/QueueAdminConfigurationMutationACLPolicy.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/DefaultConfigurationMutationACLPolicy.java
* (delete) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/QueueConfigsUpdateInfo.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesConfigurationMutation.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/ConfigurationMutationACLPolicy.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestMutableCSConfigurationProvider.java


> Support global configuration mutation in MutableConfProvider
> 
>
> Key: YARN-6575
> URL: https://issues.apache.org/jira/browse/YARN-6575
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
> Fix For: YARN-5734
>
> Attachments: YARN-6575-YARN-5734.001.patch, 
> YARN-6575-YARN-5734.002.patch, YARN-6575-YARN-5734.003.patch
>
>
> Right now mutating configs assumes they are only queue configs. Support 
> should be added to mutate global scheduler configs.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6322) Disable queue refresh when configuration mutation is enabled

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197786#comment-16197786
 ] 

Hudson commented on YARN-6322:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13057 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/13057/])
YARN-6322: Disable queue refresh when configuration mutation is enabled. 
(jhung: rev 4c8b208adb8c7639628676387e275856cf0842c9)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/MutableConfigurationProvider.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestMutableCSConfigurationProvider.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/MutableConfScheduler.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/MutableCSConfigurationProvider.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java


> Disable queue refresh when configuration mutation is enabled
> 
>
> Key: YARN-6322
> URL: https://issues.apache.org/jira/browse/YARN-6322
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
> Fix For: YARN-5734
>
> Attachments: YARN-6322-YARN-5734.001.patch, 
> YARN-6322-YARN-5734.002.patch, YARN-6322-YARN-5734.003.patch, 
> YARN-6322-YARN-5734.004.patch
>
>
> When configuration mutation is enabled, the configuration store is the source 
> of truth. Calling {{-refreshQueues}} won't work as intended, so we should 
> just disable this.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5946) Create YarnConfigurationStore interface and InMemoryConfigurationStore class

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197779#comment-16197779
 ] 

Hudson commented on YARN-5946:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13057 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/13057/])
YARN-5946: Create YarnConfigurationStore interface and (jhung: rev 
e3579a8c3b1dc58a38859b189973be5a2d23f730)
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/InMemoryConfigurationStore.java
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestYarnConfigurationStore.java
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/YarnConfigurationStore.java


> Create YarnConfigurationStore interface and InMemoryConfigurationStore class
> 
>
> Key: YARN-5946
> URL: https://issues.apache.org/jira/browse/YARN-5946
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
> Fix For: YARN-5734
>
> Attachments: YARN-5946-YARN-5734.002.patch, 
> YARN-5946-YARN-5734.003.patch, YARN-5946-YARN-5734.004.patch, 
> YARN-5946.001.patch
>
>
> This class provides the interface to persist YARN configurations in a backing 
> store.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7046) Add closing logic to configuration store

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197789#comment-16197789
 ] 

Hudson commented on YARN-7046:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13057 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/13057/])
YARN-7046. Add closing logic to configuration store (jhung: rev 
d6622daaa3c95a5040d06f91dc0c793bafa70f95)
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestLeveldbConfigurationStore.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/YarnConfigurationStore.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/ConfigurationStoreBaseTest.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/MutableConfigurationProvider.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/LeveldbConfigurationStore.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/MutableCSConfigurationProvider.java


> Add closing logic to configuration store
> 
>
> Key: YARN-7046
> URL: https://issues.apache.org/jira/browse/YARN-7046
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
> Fix For: YARN-5734
>
> Attachments: YARN-7046-YARN-5734.001.patch, 
> YARN-7046-YARN-5734.002.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5734) OrgQueue for easy CapacityScheduler queue configuration management

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197792#comment-16197792
 ] 

Hudson commented on YARN-5734:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13057 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/13057/])
YARN-7251. Misc changes to YARN-5734 (jhung: rev 
09c5dfe937f0570cd9494b34d210df2d5f0737a7)
* (edit) hadoop-yarn-project/hadoop-yarn/bin/yarn
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestZKConfigurationStore.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesConfigurationMutation.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/SchedConfCLI.java
* (edit) hadoop-yarn-project/hadoop-yarn/bin/yarn.cmd
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnCommands.md
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestLeveldbConfigurationStore.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/dao/SchedConfUpdateInfo.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestSchedConfCLI.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestMutableCSConfigurationProvider.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/dao/QueueConfigInfo.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerRest.md
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/MutableCSConfigurationProvider.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java


> OrgQueue for easy CapacityScheduler queue configuration management
> --
>
> Key: YARN-5734
> URL: https://issues.apache.org/jira/browse/YARN-5734
> Project: Hadoop YARN
>  Issue Type: New Feature
>Reporter: Min Shen
>Assignee: Min Shen
> Attachments: 
> OrgQueueAPI-BasedSchedulerConfigurationManagement_v2.pdf, 
> OrgQueue_API-Based_Config_Management_v1.pdf, OrgQueue_Design_v0.pdf, 
> YARN-5734-YARN-5734.001.patch
>
>
> The current xml based configuration mechanism in CapacityScheduler makes it 
> very inconvenient to apply any changes to the queue configurations. We saw 2 
> main drawbacks in the file based configuration mechanism:
> # This makes it very inconvenient to automate queue configuration updates. 
> For example, in our cluster setup, we leverage the queue mapping feature from 
> YARN-2411 to route users to their dedicated organization queues. It could be 
> extremely cumbersome to keep updating the config file to manage the very 
> dynamic mapping between users to organizations.
> # Even a user has the admin permission on one specific queue, that user is 
> unable to make any queue configuration changes to resize the subqueues, 
> changing queue ACLs, or creating new queues. All these operations need to be 
> performed in a centralized manner by the cluster administrators.
> With these current limitations, we realized the need of a more flexible 
> configuration mechanism that allows queue 

[jira] [Commented] (YARN-5947) Create LeveldbConfigurationStore class using Leveldb as backing store

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197785#comment-16197785
 ] 

Hudson commented on YARN-5947:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13057 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/13057/])
YARN-5947: Create LeveldbConfigurationStore class using Leveldb as (jhung: rev 
74ba6ffa0b601badc68366685dcf993a0844520a)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/YarnConfigurationStore.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/MutableCSConfigurationProvider.java
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/LeveldbConfigurationStore.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/MutableConfigurationProvider.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestYarnConfigurationStore.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java


> Create LeveldbConfigurationStore class using Leveldb as backing store
> -
>
> Key: YARN-5947
> URL: https://issues.apache.org/jira/browse/YARN-5947
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
> Fix For: YARN-5734
>
> Attachments: YARN-5947-YARN-5734.001.patch, 
> YARN-5947-YARN-5734.002.patch, YARN-5947-YARN-5734.003.patch, 
> YARN-5947-YARN-5734.004.patch, YARN-5947.001.patch
>
>
> LeveldbConfigurationStore will extend YarnConfigurationStore for storing 
> scheduler configuration in a LevelDB.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5953) Create CLI for changing YARN configurations

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197784#comment-16197784
 ] 

Hudson commented on YARN-5953:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13057 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/13057/])
YARN-5953:Create CLI for changing YARN configurations. (Jonathan Hung (jhung: 
rev 916bdbd6bedbc75000bb74e16c1e2ee692a4912c)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesConfigurationMutation.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* (edit) hadoop-yarn-project/hadoop-yarn/bin/yarn.cmd
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/MutableConfScheduler.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/SchedConfUpdateInfo.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/QueueAdminConfigurationMutationACLPolicy.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestMutableCSConfigurationProvider.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/ConfigurationMutationACLPolicy.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestConfigurationMutationACLPolicies.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/QueueConfigInfo.java
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestSchedConfCLI.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/MutableConfigurationProvider.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/util/YarnWebServiceUtils.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/DefaultConfigurationMutationACLPolicy.java
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/SchedConfCLI.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/MutableCSConfigurationProvider.java
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/dao/package-info.java
* (edit) hadoop-yarn-project/hadoop-yarn/bin/yarn
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java
YARN-5953 addendum: Move QueueConfigInfo and SchedConfUpdateInfo to (jhung: rev 
04f9e80bb2662c51c2fc53c2edc10c660fd71a9e)
* (delete) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/SchedConfUpdateInfo.java
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/dao/SchedConfUpdateInfo.java
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/dao/QueueConfigInfo.java
* (delete) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/QueueConfigInfo.java


> Create CLI for changing YARN configurations
> ---
>
> Key: YARN-5953
> URL: https://issues.apache.org/jira/browse/YARN-5953
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
> Fix For: YARN-5734
>
> Attachments: YARN-5953-YARN-5734.001.patch, 
> YARN-5953-YARN-5734.002.patch, 

[jira] [Commented] (YARN-5951) Changes to allow CapacityScheduler to use configuration store

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197778#comment-16197778
 ] 

Hudson commented on YARN-5951:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13057 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/13057/])
YARN-5951. Changes to allow CapacityScheduler to use configuration store 
(jhung: rev ef59cbe08aabe9844159cb4192d6345e4e3caab2)
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/FileBasedCSConfigurationProvider.java
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/CSConfigurationProvider.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/package-info.java


> Changes to allow CapacityScheduler to use configuration store
> -
>
> Key: YARN-5951
> URL: https://issues.apache.org/jira/browse/YARN-5951
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
> Fix For: YARN-5734
>
> Attachments: YARN-5951-YARN-5734.001.patch, 
> YARN-5951-YARN-5734.002.patch, YARN-5951-YARN-5734.003.patch, 
> YARN-5951-YARN-5734.004.patch
>
>
> EDIT: changing this ticket. Found that the CapacityStoreConfigurationProvider 
> is not necessary, since we can just grab a Configuration object from 
> StoreConfigurationProvider with type "SCHEDULER" and create a 
> CapacitySchedulerConfiguration from it.
> This ticket will track changes needed for integrating other components to be 
> used by the capacity scheduler.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7238) Documentation for API based scheduler configuration management

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197790#comment-16197790
 ] 

Hudson commented on YARN-7238:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13057 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/13057/])
YARN-7238. Documentation for API based scheduler configuration (jhung: rev 
a3075de9ab2fc9a97debaab6fc838ea07b1cc77e)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerRest.md
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnCommands.md
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md


> Documentation for API based scheduler configuration management
> --
>
> Key: YARN-7238
> URL: https://issues.apache.org/jira/browse/YARN-7238
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
> Fix For: YARN-5734
>
> Attachments: YARN-7238-YARN-5734.001.patch, 
> YARN-7238-YARN-5734.002.patch
>
>
> Documentation for configurations to set / how to use scheduler configuration 
> mutation API.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7252) Removing queue then failing over results in exception

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197793#comment-16197793
 ] 

Hudson commented on YARN-7252:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13057 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/13057/])
YARN-7252. Removing queue then failing over results in exception (jhung: rev 
1d36b53ab6d9bb1d9144101e424c24371343c5bf)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerQueueManager.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestZKConfigurationStore.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerContext.java


> Removing queue then failing over results in exception
> -
>
> Key: YARN-7252
> URL: https://issues.apache.org/jira/browse/YARN-7252
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Critical
> Fix For: YARN-5734
>
> Attachments: YARN-7252-YARN-5734.001.patch, 
> YARN-7252-YARN-5734.002.patch
>
>
> Scenario: rm1 and rm2, starting configuration with root.default, root.a. rm1 
> is active. First, put root.a into STOPPED state, then remove it. Then put rm1 
> in standby and rm2 in active. Here's the exception: {noformat}Operation 
> failed: Error on refreshAll during transition to Active
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:315)
>   at 
> org.apache.hadoop.ha.protocolPB.HAServiceProtocolServerSideTranslatorPB.transitionToActive(HAServiceProtocolServerSideTranslatorPB.java:107)
>   at 
> org.apache.hadoop.ha.proto.HAServiceProtocolProtos$HAServiceProtocolService$2.callBlockingMethod(HAServiceProtocolProtos.java:4460)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
>   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
>   at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
>   at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
>   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
> Caused by: org.apache.hadoop.ha.ServiceFailedException: RefreshAll operation 
> failed
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshAll(AdminService.java:747)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:307)
>   ... 10 more
> Caused by: java.io.IOException: Failed to re-init queues : root.a is deleted 
> from the new capacity scheduler configuration, but the queue is not yet in 
> stopped state. Current State : RUNNING
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:436)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshQueues(AdminService.java:405)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshAll(AdminService.java:736)
>   ... 11 more
> Caused by: java.io.IOException: root.a is deleted from the new capacity 
> scheduler configuration, but the queue is not yet in stopped state. Current 
> State : RUNNING
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.validateQueueHierarchy(CapacitySchedulerQueueManager.java:312)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.reinitializeQueues(CapacitySchedulerQueueManager.java:174)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitializeQueues(CapacityScheduler.java:648)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:432)
>   ... 13 more{noformat}
> Seems rm2 does not think root.a was STOPPED, so when it can't find root.a and 
> sees it is deleted, it throws exception.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For 

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197788#comment-16197788
 ] 

Hudson commented on YARN-6840:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13057 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/13057/])
YARN-6840. Implement zookeeper based store for scheduler configuration (jhung: 
rev ff39c0de206a4fce1f0e8a416357a7a8261f8634)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/MutableConfScheduler.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/InMemoryConfigurationStore.java
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestZKConfigurationStore.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/ZKConfigurationStore.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/LeveldbConfigurationStore.java
* (delete) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestYarnConfigurationStore.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/ConfigurationStoreBaseTest.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/MutableCSConfigurationProvider.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestInMemoryConfigurationStore.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateVersionIncompatibleException.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/MutableConfigurationProvider.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestMutableCSConfigurationProvider.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/YarnConfigurationStore.java


> Implement zookeeper based store for scheduler configuration updates
> ---
>
> Key: YARN-6840
> URL: https://issues.apache.org/jira/browse/YARN-6840
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Jonathan Hung
> Fix For: YARN-5734
>
> Attachments: YARN-6840-YARN-5734.001.patch, 
> YARN-6840-YARN-5734.002.patch, YARN-6840-YARN-5734.003.patch, 
> YARN-6840-YARN-5734.004.patch, YARN-6840-YARN-5734.005.patch, 
> YARN-6840-YARN-5734.006.patch, 

[jira] [Commented] (YARN-5948) Implement MutableConfigurationManager for handling storage into configuration store

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197780#comment-16197780
 ] 

Hudson commented on YARN-5948:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13057 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/13057/])
YARN-5948. Implement MutableConfigurationManager for handling storage (jhung: 
rev ef686635c1aa0cd4bd52ca212a95769f2532aa1a)
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/MutableCSConfigurationProvider.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/YarnConfigurationStoreFactory.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/MutableConfigurationProvider.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/CSConfigurationProvider.java
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestMutableCSConfigurationProvider.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java


> Implement MutableConfigurationManager for handling storage into configuration 
> store
> ---
>
> Key: YARN-5948
> URL: https://issues.apache.org/jira/browse/YARN-5948
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
> Fix For: YARN-5734
>
> Attachments: YARN-5948-YARN-5734.002.patch, 
> YARN-5948-YARN-5734.003.patch, YARN-5948-YARN-5734.004.patch, 
> YARN-5948-YARN-5734.005.patch, YARN-5948-YARN-5734.006.patch, 
> YARN-5948-YARN-5734.007.patch, YARN-5948-YARN-5734.008.patch, 
> YARN-5948.001.patch
>
>
> The MutableConfigurationManager will take REST calls with desired client 
> configuration changes and call YarnConfigurationStore methods to store these 
> changes in the backing store.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7024) Fix issues on recovery in LevelDB store

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197787#comment-16197787
 ] 

Hudson commented on YARN-7024:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13057 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/13057/])
YARN-7024: Fix issues on recovery in LevelDB store. Contributed by (jhung: rev 
4d8abd84f40a6124e7b05b34c14b4035309623ef)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/InMemoryConfigurationStore.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/LeveldbConfigurationStore.java


> Fix issues on recovery in LevelDB store
> ---
>
> Key: YARN-7024
> URL: https://issues.apache.org/jira/browse/YARN-7024
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
> Fix For: YARN-5734
>
> Attachments: YARN-7024-YARN-5734.001.patch
>
>
> On recovery, we get the pending mutations not yet committed to store. Upon 
> confirming them, we remove them in memory while iterating, which may cause 
> issues in the future.
> Also, when getting the pending mutations which were logged but not committed 
> on recovery, we don't update the txnId in memory.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5949) Add pluggable configuration ACL policy interface and implementation

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197782#comment-16197782
 ] 

Hudson commented on YARN-5949:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13057 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/13057/])
YARN-5949. Add pluggable configuration ACL policy interface and (jhung: rev 
a4e62530469e4c3d5b339a06adeac2146fc15fa5)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/QueueAdminConfigurationMutationACLPolicy.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestMutableCSConfigurationProvider.java
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/DefaultConfigurationMutationACLPolicy.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/MutableCSConfigurationProvider.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/MutableConfigurationProvider.java
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/ConfigurationMutationACLPolicy.java
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestConfigurationMutationACLPolicies.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/MutableConfScheduler.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/ConfigurationMutationACLPolicyFactory.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java


> Add pluggable configuration ACL policy interface and implementation
> ---
>
> Key: YARN-5949
> URL: https://issues.apache.org/jira/browse/YARN-5949
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
> Fix For: YARN-5734
>
> Attachments: YARN-5949-YARN-5734.001.patch, 
> YARN-5949-YARN-5734.002.patch, YARN-5949-YARN-5734.003.patch, 
> YARN-5949-YARN-5734.004.patch, YARN-5949-YARN-5734.005.patch
>
>
> This will allow different policies to customize how/if configuration changes 
> should be applied (for example, a policy might restrict whether a 
> configuration change by a certain user is allowed). This will be enforced by 
> the MutableCSConfigurationProvider.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5952) Create REST API for changing YARN scheduler configurations

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197781#comment-16197781
 ] 

Hudson commented on YARN-5952:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13057 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/13057/])
YARN-5952. Create REST API for changing YARN scheduler configurations. (jhung: 
rev d8a5136b1f3761166a08969eda7a7f66d1578d7e)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestMutableCSConfigurationProvider.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/MutableCSConfigurationProvider.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/InMemoryConfigurationStore.java
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/MutableConfScheduler.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/MutableConfigurationProvider.java
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/QueueConfigsUpdateInfo.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/QueueConfigInfo.java
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesConfigurationMutation.java


> Create REST API for changing YARN scheduler configurations
> --
>
> Key: YARN-5952
> URL: https://issues.apache.org/jira/browse/YARN-5952
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
> Fix For: YARN-5734
>
> Attachments: YARN-5952-YARN-5734.003.patch, 
> YARN-5952-YARN-5734.004.patch, YARN-5952-YARN-5734.005.patch, 
> YARN-5952-YARN-5734.006.patch, YARN-5952-YARN-5734.007.patch, 
> YARN-5952-YARN-5734.008.patch, YARN-5952.001.patch, YARN-5952.002.patch
>
>
> Based on the design in YARN-5734.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Resolved] (YARN-7304) Merge YARN-5734 branch to branch-3.0

[Xuan Gong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuan Gong resolved YARN-7304.
-
Resolution: Fixed

> Merge YARN-5734 branch to branch-3.0
> 
>
> Key: YARN-7304
> URL: https://issues.apache.org/jira/browse/YARN-7304
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Fix For: 3.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7304) Merge YARN-5734 branch to branch-3.0

[Xuan Gong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuan Gong updated YARN-7304:

Fix Version/s: 3.0.0

> Merge YARN-5734 branch to branch-3.0
> 
>
> Key: YARN-7304
> URL: https://issues.apache.org/jira/browse/YARN-7304
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Fix For: 3.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7304) Merge YARN-5734 branch to branch-3.0

[Xuan Gong (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197776#comment-16197776
 ] 

Xuan Gong commented on YARN-7304:
-

Merged

> Merge YARN-5734 branch to branch-3.0
> 
>
> Key: YARN-7304
> URL: https://issues.apache.org/jira/browse/YARN-7304
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7303) Merge YARN-5734 branch to trunk branch

[Xuan Gong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuan Gong updated YARN-7303:

Fix Version/s: 3.1.0

> Merge YARN-5734 branch to trunk branch
> --
>
> Key: YARN-7303
> URL: https://issues.apache.org/jira/browse/YARN-7303
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Fix For: 3.1.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7224) Support GPU isolation for docker container

[Wangda Tan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wangda Tan updated YARN-7224:
-
Attachment: YARN-7224.002-wip.patch

Attached ver.2 work-in-progress patch. Major change of this patch is I found 
cgroup + docker doesn't work under newer docker version which uses {{runc}} as 
default runtime. Setting {{--cgroup-parent}} to a cgroup which include any 
{{devices.deny}} causes docker container cannot be launched.

Instead this patch passes allowed GPU devices via {{--device}} to docker launch 
command. Tested this patch in a centos 7 machine with 2 GPU devices, it works 
fine. There're some cleanups need to be done and more unit tests need to be 
added. Marked as WIP.

> Support GPU isolation for docker container
> --
>
> Key: YARN-7224
> URL: https://issues.apache.org/jira/browse/YARN-7224
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-7224.001.patch, YARN-7224.002-wip.patch
>
>
> YARN-6620 added support of GPU isolation in NM side, which only supports 
> non-docker containers. We need to add support to help docker containers 
> launched by YARN can utilize GPUs.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Haibo Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197757#comment-16197757
 ] 

Haibo Chen commented on YARN-4511:
--

Uploaded a new patch that based on Wangda's review
1) removes allowedResourceForOverAllocation() from SchedulerNode 
2) passes SchedulerNode from SchedulerAppAttempt to 
ContainerUpdateContext.swapContainers() in the parameter to avoid plumbing 
context.
3) consolidates two separate maps of  into one
 
[~asuresh], can you give some insight into the swapContainer change and a look 
at the patch ?

> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Haibo Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haibo Chen updated YARN-4511:
-
Attachment: YARN-4511-YARN-1011.03.patch

> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7303) Merge YARN-5734 branch to trunk branch

[Xuan Gong (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197754#comment-16197754
 ] 

Xuan Gong commented on YARN-7303:
-

Merged YARN-5734 branch to trunk branch.

> Merge YARN-5734 branch to trunk branch
> --
>
> Key: YARN-7303
> URL: https://issues.apache.org/jira/browse/YARN-7303
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Resolved] (YARN-7303) Merge YARN-5734 branch to trunk branch

[Xuan Gong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuan Gong resolved YARN-7303.
-
Resolution: Fixed

> Merge YARN-5734 branch to trunk branch
> --
>
> Key: YARN-7303
> URL: https://issues.apache.org/jira/browse/YARN-7303
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-3661) Basic Federation UI

[JIRA]

 [ 
https://issues.apache.org/jira/browse/YARN-3661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri updated YARN-3661:
--
Attachment: YARN-3661-branch-2.001.patch

> Basic Federation UI 
> 
>
> Key: YARN-3661
> URL: https://issues.apache.org/jira/browse/YARN-3661
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager, resourcemanager
>Reporter: Giovanni Matteo Fumarola
>Assignee: Íñigo Goiri
> Attachments: YARN-3661-000.patch, YARN-3661-001.patch, 
> YARN-3661-002.patch, YARN-3661-003.patch, YARN-3661-004.patch, 
> YARN-3661-005.patch, YARN-3661-006.patch, YARN-3661-007.patch, 
> YARN-3661-008.patch, YARN-3661-009.patch, YARN-3661-010.patch, 
> YARN-3661-011.patch, YARN-3661-012.patch, YARN-3661-013.patch, 
> YARN-3661-014.patch, YARN-3661-015.patch, YARN-3661-branch-2.001.patch
>
>
> The UIs provided by each RM, provide a correct "local" view of what is 
> running in a sub-cluster. In the context of federation we need new 
> UIs that can track load, jobs, users across sub-clusters.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-3661) Basic Federation UI

[JIRA]

 [ 
https://issues.apache.org/jira/browse/YARN-3661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri updated YARN-3661:
--
Attachment: (was: YARN-3661-015-branch-2.patch)

> Basic Federation UI 
> 
>
> Key: YARN-3661
> URL: https://issues.apache.org/jira/browse/YARN-3661
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager, resourcemanager
>Reporter: Giovanni Matteo Fumarola
>Assignee: Íñigo Goiri
> Attachments: YARN-3661-000.patch, YARN-3661-001.patch, 
> YARN-3661-002.patch, YARN-3661-003.patch, YARN-3661-004.patch, 
> YARN-3661-005.patch, YARN-3661-006.patch, YARN-3661-007.patch, 
> YARN-3661-008.patch, YARN-3661-009.patch, YARN-3661-010.patch, 
> YARN-3661-011.patch, YARN-3661-012.patch, YARN-3661-013.patch, 
> YARN-3661-014.patch, YARN-3661-015.patch
>
>
> The UIs provided by each RM, provide a correct "local" view of what is 
> running in a sub-cluster. In the context of federation we need new 
> UIs that can track load, jobs, users across sub-clusters.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-7306) Set default RPC timeout to 5 minutes

[Yufei Gu (JIRA)]

Yufei Gu created YARN-7306:
--

 Summary: Set default RPC timeout to 5 minutes
 Key: YARN-7306
 URL: https://issues.apache.org/jira/browse/YARN-7306
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: nodemanager, resourcemanager
Reporter: Yufei Gu
Assignee: Yufei Gu


The motivation is well described in HADOOP-11252. 
{quote}
The RPC client has a default timeout set to 0 when no timeout is passed in. 
This means that the network connection created will not timeout when used to 
write data. The issue has shown in YARN-2578 and HDFS-4858. Timeouts for writes 
then fall back to the tcp level retry (configured via tcp_retries2) and 
timeouts between the 15-30 minutes. Which is too long for a default behaviour.
{quote}
However, HADOOP-11252 didn't set the default value to a meaningful timeout(it 
is zero, which means infinity). User will still hit this issue by default. 
Maybe we should set the default value to  a meaningful one.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-3661) Basic Federation UI

[JIRA]

 [ 
https://issues.apache.org/jira/browse/YARN-3661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri updated YARN-3661:
--
Attachment: YARN-3661-015-branch-2.patch

> Basic Federation UI 
> 
>
> Key: YARN-3661
> URL: https://issues.apache.org/jira/browse/YARN-3661
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager, resourcemanager
>Reporter: Giovanni Matteo Fumarola
>Assignee: Íñigo Goiri
> Attachments: YARN-3661-000.patch, YARN-3661-001.patch, 
> YARN-3661-002.patch, YARN-3661-003.patch, YARN-3661-004.patch, 
> YARN-3661-005.patch, YARN-3661-006.patch, YARN-3661-007.patch, 
> YARN-3661-008.patch, YARN-3661-009.patch, YARN-3661-010.patch, 
> YARN-3661-011.patch, YARN-3661-012.patch, YARN-3661-013.patch, 
> YARN-3661-014.patch, YARN-3661-015-branch-2.patch, YARN-3661-015.patch
>
>
> The UIs provided by each RM, provide a correct "local" view of what is 
> running in a sub-cluster. In the context of federation we need new 
> UIs that can track load, jobs, users across sub-clusters.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-4006) YARN ATSv1 Alternate Kerberos HTTP Authentication Changes

[Vrushali C (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vrushali C updated YARN-4006:
-
Summary: YARN ATSv1 Alternate Kerberos HTTP Authentication Changes  (was: 
YARN ATS Alternate Kerberos HTTP Authentication Changes)

> YARN ATSv1 Alternate Kerberos HTTP Authentication Changes
> -
>
> Key: YARN-4006
> URL: https://issues.apache.org/jira/browse/YARN-4006
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: security, timelineserver
>Affects Versions: 2.5.0, 2.6.0, 2.7.0, 2.5.1, 2.6.1, 2.8.0, 2.7.1, 2.7.2
>Reporter: Greg Senia
>Priority: Blocker
> Attachments: YARN-4006-branch-trunk.patch, 
> YARN-4006-branch2.6.0.patch, sample-ats-alt-auth.patch
>
>
> When attempting to use The Hadoop Alternate Authentication Classes. They do 
> not exactly work with what was built with YARN-1935.
> I went ahead and made the following changes to support using a Custom 
> AltKerberos DelegationToken custom class.
> Changes to: TimelineAuthenticationFilterInitializer.class
> {code}
>String authType = filterConfig.get(AuthenticationFilter.AUTH_TYPE);
> LOG.info("AuthType Configured: "+authType);
> if (authType.equals(PseudoAuthenticationHandler.TYPE)) {
>   filterConfig.put(AuthenticationFilter.AUTH_TYPE,
>   PseudoDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: PseudoDelegationTokenAuthenticationHandler");
> } else if (authType.equals(KerberosAuthenticationHandler.TYPE) || 
> (UserGroupInformation.isSecurityEnabled() && 
> conf.get("hadoop.security.authentication").equals(KerberosAuthenticationHandler.TYPE)))
>  {
>   if (!(authType.equals(KerberosAuthenticationHandler.TYPE))) {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
>   authType);
> LOG.info("AuthType: "+authType);
>   } else {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
>   KerberosDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: KerberosDelegationTokenAuthenticationHandler");
>   } 
>   // Resolve _HOST into bind address
>   String bindAddress = conf.get(HttpServer2.BIND_ADDRESS);
>   String principal =
>   filterConfig.get(KerberosAuthenticationHandler.PRINCIPAL);
>   if (principal != null) {
> try {
>   principal = SecurityUtil.getServerPrincipal(principal, bindAddress);
> } catch (IOException ex) {
>   throw new RuntimeException(
>   "Could not resolve Kerberos principal name: " + ex.toString(), 
> ex);
> }
> filterConfig.put(KerberosAuthenticationHandler.PRINCIPAL,
> principal);
>   }
> }
>  {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4754) Too many connection opened to TimelineServer while publishing entities

[Subru Krishnan (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197704#comment-16197704
 ] 

Subru Krishnan commented on YARN-4754:
--

[~rohithsharma], [~varun_saxena], [~Ying Zhang], is this still a problem? If 
so, are we on track for 2.9.0 (27th Oct)? Thanks!

> Too many connection opened to TimelineServer while publishing entities
> --
>
> Key: YARN-4754
> URL: https://issues.apache.org/jira/browse/YARN-4754
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Rohith Sharma K S
>Priority: Critical
> Attachments: ConnectionLeak.rar
>
>
> It is observed that there are too many connections are kept opened to 
> TimelineServer while publishing entities via SystemMetricsPublisher. This 
> cause sometimes resource shortage for other process or RM itself
> {noformat}
> tcp0  0 10.18.99.110:3999   10.18.214.60:59265  
> ESTABLISHED 115302/java 
> tcp0  0 10.18.99.110:25001  :::*LISTEN
>   115302/java 
> tcp0  0 10.18.99.110:25002  :::*LISTEN
>   115302/java 
> tcp0  0 10.18.99.110:25003  :::*LISTEN
>   115302/java 
> tcp0  0 10.18.99.110:25004  :::*LISTEN
>   115302/java 
> tcp0  0 10.18.99.110:25005  :::*LISTEN
>   115302/java 
> tcp1  0 10.18.99.110:48866  10.18.99.110:8188   
> CLOSE_WAIT  115302/java 
> tcp1  0 10.18.99.110:48137  10.18.99.110:8188   
> CLOSE_WAIT  115302/java 
> tcp1  0 10.18.99.110:47553  10.18.99.110:8188   
> CLOSE_WAIT  115302/java 
> tcp1  0 10.18.99.110:48424  10.18.99.110:8188   
> CLOSE_WAIT  115302/java 
> tcp1  0 10.18.99.110:48139  10.18.99.110:8188   
> CLOSE_WAIT  115302/java 
> tcp1  0 10.18.99.110:48096  10.18.99.110:8188   
> CLOSE_WAIT  115302/java 
> tcp1  0 10.18.99.110:47558  10.18.99.110:8188   
> CLOSE_WAIT  115302/java 
> tcp1  0 10.18.99.110:49270  10.18.99.110:8188   
> CLOSE_WAIT  115302/java 
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4006) YARN ATS Alternate Kerberos HTTP Authentication Changes

[Subru Krishnan (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197703#comment-16197703
 ] 

Subru Krishnan commented on YARN-4006:
--

[~gss2002], [~aw], [~lmccay], is this still a problem? If so, are we on track 
for 2.9.0 (27th Oct)? Thanks!

> YARN ATS Alternate Kerberos HTTP Authentication Changes
> ---
>
> Key: YARN-4006
> URL: https://issues.apache.org/jira/browse/YARN-4006
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: security, timelineserver
>Affects Versions: 2.5.0, 2.6.0, 2.7.0, 2.5.1, 2.6.1, 2.8.0, 2.7.1, 2.7.2
>Reporter: Greg Senia
>Priority: Blocker
> Attachments: YARN-4006-branch-trunk.patch, 
> YARN-4006-branch2.6.0.patch, sample-ats-alt-auth.patch
>
>
> When attempting to use The Hadoop Alternate Authentication Classes. They do 
> not exactly work with what was built with YARN-1935.
> I went ahead and made the following changes to support using a Custom 
> AltKerberos DelegationToken custom class.
> Changes to: TimelineAuthenticationFilterInitializer.class
> {code}
>String authType = filterConfig.get(AuthenticationFilter.AUTH_TYPE);
> LOG.info("AuthType Configured: "+authType);
> if (authType.equals(PseudoAuthenticationHandler.TYPE)) {
>   filterConfig.put(AuthenticationFilter.AUTH_TYPE,
>   PseudoDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: PseudoDelegationTokenAuthenticationHandler");
> } else if (authType.equals(KerberosAuthenticationHandler.TYPE) || 
> (UserGroupInformation.isSecurityEnabled() && 
> conf.get("hadoop.security.authentication").equals(KerberosAuthenticationHandler.TYPE)))
>  {
>   if (!(authType.equals(KerberosAuthenticationHandler.TYPE))) {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
>   authType);
> LOG.info("AuthType: "+authType);
>   } else {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
>   KerberosDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: KerberosDelegationTokenAuthenticationHandler");
>   } 
>   // Resolve _HOST into bind address
>   String bindAddress = conf.get(HttpServer2.BIND_ADDRESS);
>   String principal =
>   filterConfig.get(KerberosAuthenticationHandler.PRINCIPAL);
>   if (principal != null) {
> try {
>   principal = SecurityUtil.getServerPrincipal(principal, bindAddress);
> } catch (IOException ex) {
>   throw new RuntimeException(
>   "Could not resolve Kerberos principal name: " + ex.toString(), 
> ex);
> }
> filterConfig.put(KerberosAuthenticationHandler.PRINCIPAL,
> principal);
>   }
> }
>  {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-2919) Potential race between renew and cancel in DelegationTokenRenwer

[Subru Krishnan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-2919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Subru Krishnan updated YARN-2919:
-
Target Version/s: 3.1.0  (was: 2.9.0)

> Potential race between renew and cancel in DelegationTokenRenwer 
> -
>
> Key: YARN-2919
> URL: https://issues.apache.org/jira/browse/YARN-2919
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.6.0
>Reporter: Karthik Kambatla
>Assignee: Naganarasimha G R
>Priority: Critical
> Attachments: YARN-2919.002.patch, YARN-2919.003.patch, 
> YARN-2919.004.patch, YARN-2919.005.patch, YARN-2919.20141209-1.patch
>
>
> YARN-2874 fixes a deadlock in DelegationTokenRenewer, but there is still a 
> race because of which a renewal in flight isn't interrupted by a cancel. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-2919) Potential race between renew and cancel in DelegationTokenRenwer

[Subru Krishnan (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-2919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197699#comment-16197699
 ] 

Subru Krishnan commented on YARN-2919:
--

Moving target version to 3.1.0 due to lack of activity. Feel free to revert if 
you have any concerns.

> Potential race between renew and cancel in DelegationTokenRenwer 
> -
>
> Key: YARN-2919
> URL: https://issues.apache.org/jira/browse/YARN-2919
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.6.0
>Reporter: Karthik Kambatla
>Assignee: Naganarasimha G R
>Priority: Critical
> Attachments: YARN-2919.002.patch, YARN-2919.003.patch, 
> YARN-2919.004.patch, YARN-2919.005.patch, YARN-2919.20141209-1.patch
>
>
> YARN-2874 fixes a deadlock in DelegationTokenRenewer, but there is still a 
> race because of which a renewal in flight isn't interrupted by a cancel. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Resolved] (YARN-7305) Merge YARN-5734 branch to branch-2

[Xuan Gong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuan Gong resolved YARN-7305.
-
   Resolution: Fixed
Fix Version/s: 2.9.0

> Merge YARN-5734 branch to branch-2
> --
>
> Key: YARN-7305
> URL: https://issues.apache.org/jira/browse/YARN-7305
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Fix For: 2.9.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7305) Merge YARN-5734 branch to branch-2

[Xuan Gong (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197696#comment-16197696
 ] 

Xuan Gong commented on YARN-7305:
-

Merged YARN-5734-branch-2 to branch-2.


> Merge YARN-5734 branch to branch-2
> --
>
> Key: YARN-7305
> URL: https://issues.apache.org/jira/browse/YARN-7305
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6523) RM requires large memory in sending out security tokens as part of Node Heartbeat in large cluster

[Subru Krishnan (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197691#comment-16197691
 ] 

Subru Krishnan commented on YARN-6523:
--

Thanks [~maniraj...@gmail.com], [~Naganarasimha], [~jlowe] for working on this. 
This looks to be a serious performance bottleneck especially considering that 
Federation (YARN-2915) is now part of 2.9.0. Can we accelerate this and get it 
resolved by 27th October?

> RM requires large memory in sending out security tokens as part of Node 
> Heartbeat in large cluster
> --
>
> Key: YARN-6523
> URL: https://issues.apache.org/jira/browse/YARN-6523
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: RM
>Affects Versions: 2.8.0, 2.7.3
>Reporter: Naganarasimha G R
>Assignee: Manikandan R
>Priority: Critical
>
> Currently as part of heartbeat response RM sets all application's tokens 
> though all applications might not be active on the node. On top of it 
> NodeHeartbeatResponsePBImpl converts tokens for each app into 
> SystemCredentialsForAppsProto. Hence for each node and each heartbeat too 
> many SystemCredentialsForAppsProto objects were getting created.
> We hit a OOM while testing for 2000 concurrent apps on 500 nodes cluster with 
> 8GB RAM configured for RM



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7286) Add support for docker to have no capabilities

[Eric Badger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Badger updated YARN-7286:
--
Attachment: YARN-7286.003.patch

Thanks for the review, [~jlowe]! Uploading a new patch that updates the 
yarn-default documentation and fixes the checkstyle warnings.

> Add support for docker to have no capabilities
> --
>
> Key: YARN-7286
> URL: https://issues.apache.org/jira/browse/YARN-7286
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: YARN-7286.001.patch, YARN-7286.002.patch, 
> YARN-7286.003.patch
>
>
> Support for controlling capabilities was introduced in YARN-4258. However, it 
> does not allow for the capabilities list to be NULL, since {{getStrings()}} 
> will treat an empty value the same as it treats an unset property. So, a NULL 
> list will actually give the default capabilities list.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7124) CLONE - Log aggregation deletes/renames while file is open

[Subru Krishnan (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197684#comment-16197684
 ] 

Subru Krishnan commented on YARN-7124:
--

[~jlowe], [~daryn], [~djp], do you guys have a port for YARN-7083 for 
trunk/branch-2 as this is a blocker for 2.9.0 release? Thanks!

> CLONE - Log aggregation deletes/renames while file is open
> --
>
> Key: YARN-7124
> URL: https://issues.apache.org/jira/browse/YARN-7124
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 2.8.2
>Reporter: Daryn Sharp
>Priority: Critical
>
> YARN-6288 changes the log aggregation writer to be an autoclosable.  
> Unfortunately the try-with-resources block for the writer will either rename 
> or delete the log while open.
> Assuming the NM's behavior is correct, deleting open files only results in 
> ominous WARNs in the nodemanager log and increases the rate of logging in the 
> NN when the implicit try-with-resource close fails.  These red herrings 
> complicate debugging efforts.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org