[jira] [Commented] (YARN-6575) Support global configuration mutation in MutableConfProvider

2017-10-09 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197783#comment-16197783
 ] 

Hudson commented on YARN-6575:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13057 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/13057/])
YARN-6575. Support global configuration mutation in MutableConfProvider. 
(jhung: rev e566fd8b58a6570593b21597aa5294d7b37cb388)
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/SchedConfUpdateInfo.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/MutableCSConfigurationProvider.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestConfigurationMutationACLPolicies.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/MutableConfScheduler.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/MutableConfigurationProvider.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/QueueAdminConfigurationMutationACLPolicy.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/DefaultConfigurationMutationACLPolicy.java
* (delete) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/QueueConfigsUpdateInfo.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesConfigurationMutation.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/ConfigurationMutationACLPolicy.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestMutableCSConfigurationProvider.java


> Support global configuration mutation in MutableConfProvider
> 
>
> Key: YARN-6575
> URL: https://issues.apache.org/jira/browse/YARN-6575
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
> Fix For: YARN-5734
>
> Attachments: YARN-6575-YARN-5734.001.patch, 
> YARN-6575-YARN-5734.002.patch, YARN-6575-YARN-5734.003.patch
>
>
> Right now mutating configs assumes they are only queue configs. Support 
> should be added to mutate global scheduler configs.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6575) Support global configuration mutation in MutableConfProvider

2017-06-05 Thread Jonathan Hung (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16037857#comment-16037857
 ] 

Jonathan Hung commented on YARN-6575:
-

Awesome, thanks [~xgong]!

> Support global configuration mutation in MutableConfProvider
> 
>
> Key: YARN-6575
> URL: https://issues.apache.org/jira/browse/YARN-6575
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
> Fix For: YARN-5734
>
> Attachments: YARN-6575-YARN-5734.001.patch, 
> YARN-6575-YARN-5734.002.patch, YARN-6575-YARN-5734.003.patch
>
>
> Right now mutating configs assumes they are only queue configs. Support 
> should be added to mutate global scheduler configs.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6575) Support global configuration mutation in MutableConfProvider

2017-06-05 Thread Xuan Gong (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16037843#comment-16037843
 ] 

Xuan Gong commented on YARN-6575:
-

Committed. Thanks for the fix, Jonathan.

> Support global configuration mutation in MutableConfProvider
> 
>
> Key: YARN-6575
> URL: https://issues.apache.org/jira/browse/YARN-6575
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
> Attachments: YARN-6575-YARN-5734.001.patch, 
> YARN-6575-YARN-5734.002.patch, YARN-6575-YARN-5734.003.patch
>
>
> Right now mutating configs assumes they are only queue configs. Support 
> should be added to mutate global scheduler configs.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6575) Support global configuration mutation in MutableConfProvider

2017-06-05 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16037807#comment-16037807
 ] 

Hadoop QA commented on YARN-6575:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
24s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 3 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m 
 6s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
44s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
32s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
52s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
31s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
34s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 39m 
22s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch 
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
19s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 71m  4s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6575 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12871306/YARN-6575-YARN-5734.003.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 1f659e15a307 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | YARN-5734 / ac788fd |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/16120/testReport/ |
| modules | C: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 U: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/16120/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Support global configuration mutation in MutableConfProvider
> 
>
> Key: YARN-6575
> URL: https://issues.apache.org/jira/browse/YARN-6575
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
> Attachments: YARN-6575-YARN-5734.001.patch, 
> YARN-6575-YARN-5734.002.patch, YARN-6575-YARN-5734.003.patch
>
>
> Right now mutating configs assumes they are only queue configs. Support 
> should be added to mutate global scheduler configs.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (YARN-6575) Support global configuration mutation in MutableConfProvider

2017-06-05 Thread Xuan Gong (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16037660#comment-16037660
 ] 

Xuan Gong commented on YARN-6575:
-

The patch looks good. Will commit shortly

> Support global configuration mutation in MutableConfProvider
> 
>
> Key: YARN-6575
> URL: https://issues.apache.org/jira/browse/YARN-6575
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
> Attachments: YARN-6575-YARN-5734.001.patch, 
> YARN-6575-YARN-5734.002.patch, YARN-6575-YARN-5734.003.patch
>
>
> Right now mutating configs assumes they are only queue configs. Support 
> should be added to mutate global scheduler configs.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6575) Support global configuration mutation in MutableConfProvider

2017-05-30 Thread Jonathan Hung (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030229#comment-16030229
 ] 

Jonathan Hung commented on YARN-6575:
-

Attached 002 patch addressing comments.

> Support global configuration mutation in MutableConfProvider
> 
>
> Key: YARN-6575
> URL: https://issues.apache.org/jira/browse/YARN-6575
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
> Attachments: YARN-6575-YARN-5734.001.patch, 
> YARN-6575-YARN-5734.002.patch
>
>
> Right now mutating configs assumes they are only queue configs. Support 
> should be added to mutate global scheduler configs.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6575) Support global configuration mutation in MutableConfProvider

2017-05-24 Thread Wangda Tan (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023411#comment-16023411
 ] 

Wangda Tan commented on YARN-6575:
--

bq. How about add-queue, remove-queue, update-queue? Since each xml object will 
be for a single queue.
Sounds good.

bq. Do you mean in this feature, or a separate feature? 
A separate feature.

bq. Not sure if there are any other configs without the yarn.scheduler.capacity 
prefix, as far as I could tell looking at CapacitySchedulerConfiguration. (will 
need to double check this)
AFAIK, preemption configs are the only configs without prefix. Let's make sure 
user specified global configs comes with full config keys.

> Support global configuration mutation in MutableConfProvider
> 
>
> Key: YARN-6575
> URL: https://issues.apache.org/jira/browse/YARN-6575
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
> Attachments: YARN-6575-YARN-5734.001.patch
>
>
> Right now mutating configs assumes they are only queue configs. Support 
> should be added to mutate global scheduler configs.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6575) Support global configuration mutation in MutableConfProvider

2017-05-23 Thread Jonathan Hung (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022189#comment-16022189
 ] 

Jonathan Hung commented on YARN-6575:
-

Thanks for the comments [~leftnoteasy],
bq. could we rename queues changes (in xml) to added-queues, removed-queues, 
updated-queues
How about add-queue, remove-queue, update-queue? Since each xml object will be 
for a single queue.
bq. We should add support of changing preemption configs via refreshQueues 
shortly.
Do you mean in this feature, or a separate feature? I'm not sure 
preemption-related configs are in the scope of this feature, since the 
preemption configs are set on monitor initialization so can only be changed on 
RM restart, while this feature is for scheduler restart. We'd have to add extra 
support apart from calling CS.reinitialize to change preemption configs at 
runtime.

Not sure if there are any other configs without the yarn.scheduler.capacity 
prefix, as far as I could tell looking at CapacitySchedulerConfiguration. (will 
need to double check this)

> Support global configuration mutation in MutableConfProvider
> 
>
> Key: YARN-6575
> URL: https://issues.apache.org/jira/browse/YARN-6575
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
> Attachments: YARN-6575-YARN-5734.001.patch
>
>
> Right now mutating configs assumes they are only queue configs. Support 
> should be added to mutate global scheduler configs.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6575) Support global configuration mutation in MutableConfProvider

2017-05-23 Thread Wangda Tan (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022154#comment-16022154
 ] 

Wangda Tan commented on YARN-6575:
--

Thanks [~jhung], 

Comments:

1) For SchedConfUpdateInfo, could we rename queues changes (in xml) to 
added-queues, removed-queues, updated-queues. And "global" to "global-updates".

2) Probably we should not add CapacitySchedulerConfiguration.PREFIX to given 
keys. For example, preemption-related configs are not start with 
CapacitySchedulerConfiguration.PREFIX. We should add support of changing 
preemption configs via refreshQueues shortly.

> Support global configuration mutation in MutableConfProvider
> 
>
> Key: YARN-6575
> URL: https://issues.apache.org/jira/browse/YARN-6575
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
> Attachments: YARN-6575-YARN-5734.001.patch
>
>
> Right now mutating configs assumes they are only queue configs. Support 
> should be added to mutate global scheduler configs.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6575) Support global configuration mutation in MutableConfProvider

2017-05-23 Thread Wangda Tan (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022155#comment-16022155
 ] 

Wangda Tan commented on YARN-6575:
--

[~xgong] could u take a look at the patch as well?

> Support global configuration mutation in MutableConfProvider
> 
>
> Key: YARN-6575
> URL: https://issues.apache.org/jira/browse/YARN-6575
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
> Attachments: YARN-6575-YARN-5734.001.patch
>
>
> Right now mutating configs assumes they are only queue configs. Support 
> should be added to mutate global scheduler configs.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6575) Support global configuration mutation in MutableConfProvider

2017-05-22 Thread Jonathan Hung (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020232#comment-16020232
 ] 

Jonathan Hung commented on YARN-6575:
-

Attached patch containing changes discussed with [~leftnoteasy] and [~xgong]:
# Rename REST endpoint / associated objects to be queue independent
# Add support for passing map of global conf changes to REST endpoint
# Add global configuration mutation support in CS configuration provider and 
queue-admin based ACL policy

> Support global configuration mutation in MutableConfProvider
> 
>
> Key: YARN-6575
> URL: https://issues.apache.org/jira/browse/YARN-6575
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
> Attachments: YARN-6575-YARN-5734.001.patch
>
>
> Right now mutating configs assumes they are only queue configs. Support 
> should be added to mutate global scheduler configs.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org