subject:"\[jira\] \[Commented\] \(YARN\-6840\) Implement zookeeper based store for scheduler configuration updates"

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

2017-10-09 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197788#comment-16197788
 ] 

Hudson commented on YARN-6840:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13057 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/13057/])
YARN-6840. Implement zookeeper based store for scheduler configuration (jhung: 
rev ff39c0de206a4fce1f0e8a416357a7a8261f8634)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/MutableConfScheduler.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/InMemoryConfigurationStore.java
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestZKConfigurationStore.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/ZKConfigurationStore.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/LeveldbConfigurationStore.java
* (delete) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestYarnConfigurationStore.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/ConfigurationStoreBaseTest.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/MutableCSConfigurationProvider.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestInMemoryConfigurationStore.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateVersionIncompatibleException.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/MutableConfigurationProvider.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestMutableCSConfigurationProvider.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/YarnConfigurationStore.java


> Implement zookeeper based store for scheduler configuration updates
> ---
>
> Key: YARN-6840
> URL: https://issues.apache.org/jira/browse/YARN-6840
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Jonathan Hung
> Fix For: YARN-5734
>
> Attachments: YARN-6840-YARN-5734.001.patch, 
> YARN-6840-YARN-5734.002.patch, YARN-6840-YARN-5734.003.patch, 
> YARN-6840-YARN-5734.004.patch, YARN-6840-YARN-5734.005.patch, 
> YARN-6840-YARN-5734.006.patch,

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

2017-09-18 Thread Jonathan Hung (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170354#comment-16170354
 ] 

Jonathan Hung commented on YARN-6840:
-

Thanks all!

> Implement zookeeper based store for scheduler configuration updates
> ---
>
> Key: YARN-6840
> URL: https://issues.apache.org/jira/browse/YARN-6840
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Jonathan Hung
> Fix For: YARN-5734
>
> Attachments: YARN-6840-YARN-5734.001.patch, 
> YARN-6840-YARN-5734.002.patch, YARN-6840-YARN-5734.003.patch, 
> YARN-6840-YARN-5734.004.patch, YARN-6840-YARN-5734.005.patch, 
> YARN-6840-YARN-5734.006.patch, YARN-6840-YARN-5734.007.patch
>
>
> Right now there is only in-memory and leveldb based configuration store 
> supported. Need one which supports RM HA.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

2017-09-18 Thread Wangda Tan (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170223#comment-16170223
 ] 

Wangda Tan commented on YARN-6840:
--

+1, will commit today. Thanks [~jhung]

> Implement zookeeper based store for scheduler configuration updates
> ---
>
> Key: YARN-6840
> URL: https://issues.apache.org/jira/browse/YARN-6840
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Jonathan Hung
> Attachments: YARN-6840-YARN-5734.001.patch, 
> YARN-6840-YARN-5734.002.patch, YARN-6840-YARN-5734.003.patch, 
> YARN-6840-YARN-5734.004.patch, YARN-6840-YARN-5734.005.patch, 
> YARN-6840-YARN-5734.006.patch, YARN-6840-YARN-5734.007.patch
>
>
> Right now there is only in-memory and leveldb based configuration store 
> supported. Need one which supports RM HA.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

2017-09-15 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168550#comment-16168550
 ] 

Hadoop QA commented on YARN-6840:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 5 new or modified test 
files. {color} |
|| || || || {color:brown} YARN-5734 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  2m 
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 
28s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  9m 
46s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 2s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m  
2s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
45s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
54s{color} | {color:green} YARN-5734 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m 
36s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
1m 26s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch 
generated 5 new + 364 unchanged - 3 fixed = 369 total (was 367) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  5m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m  
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
44s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
56s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 48m 57s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
30s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}122m  8s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
|   | hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6840 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12887408/YARN-6840-YARN-5734.007.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  xml  |
| uname | Linux ddbd7d16e119 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | YARN-5734 / 98c4a52 |
| Default Java | 1.8.0_144 |
|

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

2017-09-15 Thread Jonathan Hung (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168380#comment-16168380
 ] 

Jonathan Hung commented on YARN-6840:
-

Thanks [~leftnoteasy], attached 007 patch to fix javadoc/unit test.

> Implement zookeeper based store for scheduler configuration updates
> ---
>
> Key: YARN-6840
> URL: https://issues.apache.org/jira/browse/YARN-6840
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Jonathan Hung
> Attachments: YARN-6840-YARN-5734.001.patch, 
> YARN-6840-YARN-5734.002.patch, YARN-6840-YARN-5734.003.patch, 
> YARN-6840-YARN-5734.004.patch, YARN-6840-YARN-5734.005.patch, 
> YARN-6840-YARN-5734.006.patch, YARN-6840-YARN-5734.007.patch
>
>
> Right now there is only in-memory and leveldb based configuration store 
> supported. Need one which supports RM HA.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

2017-09-15 Thread Wangda Tan (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168085#comment-16168085
 ] 

Wangda Tan commented on YARN-6840:
--

[~jhung], 

Thanks for updating the patch, latest patch LGTM, I can commit it once jenkins 
gets clean report. 

Could you please check TestRMWebServicesConfigurationMutation failure and 
javadocs warning?

> Implement zookeeper based store for scheduler configuration updates
> ---
>
> Key: YARN-6840
> URL: https://issues.apache.org/jira/browse/YARN-6840
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Jonathan Hung
> Attachments: YARN-6840-YARN-5734.001.patch, 
> YARN-6840-YARN-5734.002.patch, YARN-6840-YARN-5734.003.patch, 
> YARN-6840-YARN-5734.004.patch, YARN-6840-YARN-5734.005.patch, 
> YARN-6840-YARN-5734.006.patch
>
>
> Right now there is only in-memory and leveldb based configuration store 
> supported. Need one which supports RM HA.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

2017-09-14 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167275#comment-16167275
 ] 

Hadoop QA commented on YARN-6840:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m 
54s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 5 new or modified test 
files. {color} |
|| || || || {color:brown} YARN-5734 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  2m 
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 
44s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  9m 
27s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 1s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
57s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
38s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
37s{color} | {color:green} YARN-5734 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  5m 
46s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
1m  0s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch 
generated 8 new + 364 unchanged - 3 fixed = 372 total (was 367) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
27s{color} | {color:red} 
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager
 generated 1 new + 347 unchanged - 0 fixed = 348 total (was 347) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
33s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
28s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 44m 54s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
28s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}126m 38s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
|   | 
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesConfigurationMutation
 |
|   | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6840 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12887235/YARN-6840-YARN-5734.006.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  xml  |
| uname | Linux 554b15275261 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 
12:48:20

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

2017-09-14 Thread Jonathan Hung (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167196#comment-16167196
 ] 

Jonathan Hung commented on YARN-6840:
-

Thanks [~leftnoteasy], 
bq. Then I suggest to add this as a Javadoc to the base class's method, this 
should be respected by all future implementations.
Added
bq. Avoid call CapacityScheduler methods from provider.
Done. Also exposed the ConfigurationMutationACLPolicy in 
MutableConfigurationProvider, since the policy calls cs.getQueue to check queue 
ACLs. (ACL checking is also done in RMWebServices just like the 
logging/refreshing/confirming.)
bq. Discard pending mutation during recovery. (Make sure that 
AdminService#refreshAll won't include the pending mutation too).
Done. Also got rid of the transaction id logic, since there is only one pending 
mutation at any time. Also added some stuff in 
TestZKConfigurationStore#testFailoverReadsFromUpdatedStore to ensure pending 
mutation is not read on failover.
bq. MutableScheduler#updateConfiguration can be removed. Instead we need an API 
to get confProvider.
Done, added aclPolicy/logging/confirming API to MutableConfigurationProvider

Also since the pendingMutation/YarnConfigurationStore APIs changed, it's 
possible some of the LeveldbConfigurationStore logic broke, but I will address 
this in YARN-7046.

> Implement zookeeper based store for scheduler configuration updates
> ---
>
> Key: YARN-6840
> URL: https://issues.apache.org/jira/browse/YARN-6840
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Jonathan Hung
> Attachments: YARN-6840-YARN-5734.001.patch, 
> YARN-6840-YARN-5734.002.patch, YARN-6840-YARN-5734.003.patch, 
> YARN-6840-YARN-5734.004.patch, YARN-6840-YARN-5734.005.patch, 
> YARN-6840-YARN-5734.006.patch
>
>
> Right now there is only in-memory and leveldb based configuration store 
> supported. Need one which supports RM HA.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

2017-09-14 Thread Wangda Tan (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166946#comment-16166946
 ] 

Wangda Tan commented on YARN-6840:
--

Synced with [~jhung] offline, in additional to above suggestions, other changes:
1) Avoid call CapacityScheduler methods from provider.
2) Discard pending mutation during recovery. (Make sure that 
AdminService#refreshAll won't include the pending mutation too).
3) MutableScheduler#updateConfiguration can be removed. Instead we need an API 
to get confProvider.



> Implement zookeeper based store for scheduler configuration updates
> ---
>
> Key: YARN-6840
> URL: https://issues.apache.org/jira/browse/YARN-6840
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Jonathan Hung
> Attachments: YARN-6840-YARN-5734.001.patch, 
> YARN-6840-YARN-5734.002.patch, YARN-6840-YARN-5734.003.patch, 
> YARN-6840-YARN-5734.004.patch, YARN-6840-YARN-5734.005.patch
>
>
> Right now there is only in-memory and leveldb based configuration store 
> supported. Need one which supports RM HA.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

2017-09-14 Thread Wangda Tan (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166839#comment-16166839
 ] 

Wangda Tan commented on YARN-6840:
--

Thanks [~jhung] for updating the patch,

bq. For leveldb and zk, it will ignore it and use the scheduler configuration 
persisted in the store.
Then I suggest to add this as a Javadoc to the base class's method, this should 
be respected by all future implementations. Otherwise behavior will be changed 
when different store is configured. 

bq. Not sure about this, then we are doing the reservation system 
reinitialization inside scheduler, so every time scheduler#reinitialize is 
called, the reservation system is also initialized, not sure if this is the 
desired behavior. Also we would need to duplicate the reservation system 
reinitialization for all schedulers, or make ResourceScheduler an abstract 
class and add it there. ... 
I just checked the code, probably I should use a different way to describe the 
problem: 
There're two different code path to refresh scheduler config:
Path #1 (When mutation disabled) : Client -> AdminService -> 
Scheduler/ReservationSystem#reinitialize
Path #2 (When mutation enabled) : Client -> RMWebService -> Scheduler -> 
ConfProvider (do log persistent) -> AdminService -> 
Scheduler/ReservationSystem#reinitialize -> ConfProvider (confirm or discard 
mutation).

Please note that in the different code path, ordering of scheduler and 
AdminService is inverted, this is confusing and could possibly cause deadlock, 
etc.

Here's my proposal:
1) Change MutableConfigurationProvider#mutateConfiguration to 
log-scheduler-config-mutation, it will do following things:
a. Merge mutations to existing configs.
b. Call confStore.logMutation to persistent it.

2) Add two new method to MutableConfigurationProvider
a. Confirm last mutation - confirm last logged mutation. (Just call 
YarnConfigurationStore#confirmMutation(valid = true))
b. Discard last mutation - discard last logged mutation. (Just call 
YarnConfigurationStore#confirmMutation(valid = false))
And is it possible to remove id field in the confirmMutation method? Should we 
allow at most one pending mutation?

One we have above, the call path#2 becomes:
(1) Client -> RMWebService#updateSchedulerConfiguration -> 
MutableConfigurationProvider#log-scheduler-config-mutation 
(2) ... RMWebService#updateSchedulerConfiguration -> AdminService#refreshQueues 
-> Scheduler/ReservationSystem#reinitialize.

If reinitialize succeeded:
(3) ... RMWebService#updateSchedulerConfiguration -> 
MutableConfigurationProvider#confirmLastChange

If reinitialize failed:
(4) RMWebService#updateSchedulerConfiguration -> 
MutableConfigurationProvider#discardLastChange


> Implement zookeeper based store for scheduler configuration updates
> ---
>
> Key: YARN-6840
> URL: https://issues.apache.org/jira/browse/YARN-6840
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Jonathan Hung
> Attachments: YARN-6840-YARN-5734.001.patch, 
> YARN-6840-YARN-5734.002.patch, YARN-6840-YARN-5734.003.patch, 
> YARN-6840-YARN-5734.004.patch, YARN-6840-YARN-5734.005.patch
>
>
> Right now there is only in-memory and leveldb based configuration store 
> supported. Need one which supports RM HA.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

2017-09-13 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16165770#comment-16165770
 ] 

Hadoop QA commented on YARN-6840:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 19m 
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 5 new or modified test 
files. {color} |
|| || || || {color:brown} YARN-5734 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  4m  
0s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 
47s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
57s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 1s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
58s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
25s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
40s{color} | {color:green} YARN-5734 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  5m 
28s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
1m  0s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch 
generated 5 new + 365 unchanged - 3 fixed = 370 total (was 368) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
34s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
38s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 46m  7s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
30s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}129m 59s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
|   | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6840 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12887019/YARN-6840-YARN-5734.005.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  xml  |
| uname | Linux ce70e7d4dca3 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | YARN-5734 / 98c4a52 |
| Default Java | 1.8.0_144 |
|

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

2017-09-13 Thread Jonathan Hung (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16165659#comment-16165659
 ] 

Jonathan Hung commented on YARN-6840:
-

Fix some checkstyle issues (005)

> Implement zookeeper based store for scheduler configuration updates
> ---
>
> Key: YARN-6840
> URL: https://issues.apache.org/jira/browse/YARN-6840
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Jonathan Hung
> Attachments: YARN-6840-YARN-5734.001.patch, 
> YARN-6840-YARN-5734.002.patch, YARN-6840-YARN-5734.003.patch, 
> YARN-6840-YARN-5734.004.patch, YARN-6840-YARN-5734.005.patch
>
>
> Right now there is only in-memory and leveldb based configuration store 
> supported. Need one which supports RM HA.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

2017-09-13 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16165645#comment-16165645
 ] 

Hadoop QA commented on YARN-6840:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 20m  
1s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 5 new or modified test 
files. {color} |
|| || || || {color:brown} YARN-5734 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  2m 
44s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 
56s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m  
9s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
57s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
51s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
45s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
31s{color} | {color:green} YARN-5734 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  6m 
27s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 56s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch 
generated 10 new + 364 unchanged - 3 fixed = 374 total (was 367) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
33s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
35s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
50s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 46m 33s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
24s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}133m 30s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6840 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12886995/YARN-6840-YARN-5734.004.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  xml  |
| uname | Linux dcfc2a7e719f 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | YARN-5734 / 98c4a52 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle |

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

2017-09-13 Thread Jonathan Hung (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16165563#comment-16165563
 ] 

Jonathan Hung commented on YARN-6840:
-

Thanks [~leftnoteasy] for the review. Attached 004 addressing most of these 
comments:

bq. Move following code: To refreshAll, and parameter isActiveTransition and 
private void refreshQueues can be removed.
Done
bq. Is it a better idea to move public void refreshQueues() logic to 
scheduler#reinitialize(...). It's not a good idea to invoke 
AdminService#refreshQueues from MutableCSConfigurationProvider. 
Not sure about this, then we are doing the reservation system reinitialization 
inside scheduler, so every time scheduler#reinitialize is called, the 
reservation system is also initialized, not sure if this is the desired 
behavior. Also we would need to duplicate the reservation system 
reinitialization for all schedulers, or make ResourceScheduler an abstract 
class and add it there.
Unless you meant duplicating the reservation system reinitialization logic 
inside MutableCSConfigurationProvider#mutateConfiguration? I think this makes 
more sense, but then we have duplicate code between this and AdminService.
bq. In MutableCSConfigurationProvider: It's better to remove:...
Done
bq. In MutableConfScheduler: Similar to above, it's better to remove:...
Done
bq. refreshConfiguration -> reloadConfigurationFromStore
Done
bq. createAndStartZKManager can be merged to rm#getZKManager()
Renamed to getAndStartZKManager
bq. Getter API of ResourceManager should be exposed by RMContext. We should 
never use ref to ResourceManager directly.
Done for ZKConfigurationStore, I left the references in ZKRMStateStore since 
this class has other direct references to rm object. We can handle this in a 
separate ticket if you'd like.
bq. setResourceManager can be removed and you can pass RMContext ref to 
initialize.
Done
bq. What happens if Configuration schedConf passed to a already initialized 
store?
For leveldb and zk, it will ignore it and use the scheduler configuration 
persisted in the store.
bq. Could you add Javadocs to following methods:
Done

> Implement zookeeper based store for scheduler configuration updates
> ---
>
> Key: YARN-6840
> URL: https://issues.apache.org/jira/browse/YARN-6840
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Jonathan Hung
> Attachments: YARN-6840-YARN-5734.001.patch, 
> YARN-6840-YARN-5734.002.patch, YARN-6840-YARN-5734.003.patch, 
> YARN-6840-YARN-5734.004.patch
>
>
> Right now there is only in-memory and leveldb based configuration store 
> supported. Need one which supports RM HA.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

2017-09-12 Thread Wangda Tan (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163392#comment-16163392
 ] 

Wangda Tan commented on YARN-6840:
--

Thanks [~jhung] for working on this patch, just reviewed overall workflows of 
the patch:

Thanks Jonathan, 

Apologize for the late review, some questions/comments:

In AdminService: 
- Move following code:
{code}
if (isActiveTransition && isSchedulerMutable()) {
  try {
((MutableConfScheduler) rm.getRMContext().getScheduler())
.refreshConfiguration();
  } catch (Exception e) {
throw new IOException("Failed to refresh configuration:", e);
  }
}
{code}
To refreshAll, and parameter {{isActiveTransition}} and {{private void 
refreshQueues}} can be removed.
- Is it a better idea to move {{public void refreshQueues()}} logic to 
scheduler#reinitialize(...). It's not a good idea to invoke 
AdminService#refreshQueues from MutableCSConfigurationProvider. (Please make 
sure that don't acquire any lock of scheduler while invoking 
ReservationSystem.reinitialize()).

In MutableCSConfigurationProvider: 
- It's better to remove:
{code}
  @VisibleForTesting
  YarnConfigurationStore getConfStore();
{code}
>From the base class. Unit test can call implementation's methods.

In MutableConfScheduler:
- Similar to above, it's better to remove:
{code}
  @VisibleForTesting
  MutableConfigurationProvider getMutableConfProvider();
{code}
>From the base class.
- refreshConfiguration -> reloadConfigurationFromStore

In ResourceManager: 
- {{createAndStartZKManager}} can be merged to rm#getZKManager(), maybe it's 
better to rename it to rm#getAndInitializeZKManager(). Even if there's no race 
condition issue since this is only called in RM initialization. It's better to 
add synchronize to the method.
- Getter API of ResourceManager should be exposed by RMContext. We should never 
use ref to ResourceManager directly.

In YarnConfigurationStore:
- {{setResourceManager}} can be removed and you can pass RMContext ref to 
{{initialize}}.
- What happens if {{Configuration schedConf}} passed to a already initialized 
store?
- Could you add Javadocs to following methods:
{code}
  protected abstract Version getConfStoreVersion() throws Exception;

  protected abstract void storeVersion() throws Exception;

  protected abstract Version getCurrentVersion();
{code}

I will leave ZK-related implementation reviews  to [~xgong].

> Implement zookeeper based store for scheduler configuration updates
> ---
>
> Key: YARN-6840
> URL: https://issues.apache.org/jira/browse/YARN-6840
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Jonathan Hung
> Attachments: YARN-6840-YARN-5734.001.patch, 
> YARN-6840-YARN-5734.002.patch, YARN-6840-YARN-5734.003.patch
>
>
> Right now there is only in-memory and leveldb based configuration store 
> supported. Need one which supports RM HA.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

2017-09-07 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158089#comment-16158089
 ] 

Hadoop QA commented on YARN-6840:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 5 new or modified test 
files. {color} |
|| || || || {color:brown} YARN-5734 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  2m 
51s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 
 8s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 
24s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 4s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m  
2s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
46s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
45s{color} | {color:green} YARN-5734 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  6m 
17s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
1m  4s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch 
generated 4 new + 328 unchanged - 2 fixed = 332 total (was 330) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m  
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m  
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
37s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
40s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 45m 13s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
29s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}112m 45s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6840 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12885976/YARN-6840-YARN-5734.003.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  xml  |
| uname | Linux 04cc49068525 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | YARN-5734 / 98c4a52 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle |

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

2017-09-07 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16157960#comment-16157960
 ] 

Hadoop QA commented on YARN-6840:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  3m 
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 5 new or modified test 
files. {color} |
|| || || || {color:brown} YARN-5734 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
54s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 
 0s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 
10s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 1s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
56s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
44s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
45s{color} | {color:green} YARN-5734 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
13s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch 
failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  2m 
13s{color} | {color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  2m 13s{color} 
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 42s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch 
generated 1 new + 208 unchanged - 122 fixed = 209 total (was 330) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red}  0m 
16s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch 
failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
15s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch 
failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
12s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch 
failed. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
28s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
27s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  0m 15s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
16s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 58m 48s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6840 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12885958/YARN-6840-YARN-5734.003.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  xml  |
| uname | Linux 814b0e56ca82 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | YARN-5734 / 98c4a52 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| mvninstall |

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

2017-09-07 Thread Jonathan Hung (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16157871#comment-16157871
 ] 

Jonathan Hung commented on YARN-6840:
-

Thanks [~xgong] - uploaded 003 which changes this.

> Implement zookeeper based store for scheduler configuration updates
> ---
>
> Key: YARN-6840
> URL: https://issues.apache.org/jira/browse/YARN-6840
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Jonathan Hung
> Attachments: YARN-6840-YARN-5734.001.patch, 
> YARN-6840-YARN-5734.002.patch, YARN-6840-YARN-5734.003.patch
>
>
> Right now there is only in-memory and leveldb based configuration store 
> supported. Need one which supports RM HA.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

2017-09-07 Thread Xuan Gong (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16157299#comment-16157299
 ] 

Xuan Gong commented on YARN-6840:
-

Thanks for the fix. 
Overall looks fine.

In logMutation(LogMutation logMutation), should we use zkManager.safeSetData 
instead of zkManager.setData to store the log ?


> Implement zookeeper based store for scheduler configuration updates
> ---
>
> Key: YARN-6840
> URL: https://issues.apache.org/jira/browse/YARN-6840
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Jonathan Hung
> Attachments: YARN-6840-YARN-5734.001.patch, 
> YARN-6840-YARN-5734.002.patch
>
>
> Right now there is only in-memory and leveldb based configuration store 
> supported. Need one which supports RM HA.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

2017-08-25 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16142445#comment-16142445
 ] 

Hadoop QA commented on YARN-6840:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 5 new or modified test 
files. {color} |
|| || || || {color:brown} YARN-5734 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  2m 
49s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 
42s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 
44s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 5s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m  
6s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
53s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
42s{color} | {color:green} YARN-5734 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  6m 
17s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 58s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch 
generated 4 new + 328 unchanged - 2 fixed = 332 total (was 330) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m  
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
43s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
36s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
43s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 46m 10s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
34s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}115m 42s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6840 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12883810/YARN-6840-YARN-5734.002.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  xml  |
| uname | Linux 235c8cf7ac9c 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | YARN-5734 / 98c4a52 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle |

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

2017-08-25 Thread Jonathan Hung (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16142230#comment-16142230
 ] 

Jonathan Hung commented on YARN-6840:
-

002 fixes checkstyle, findbugs, javadoc, and related unit tests

> Implement zookeeper based store for scheduler configuration updates
> ---
>
> Key: YARN-6840
> URL: https://issues.apache.org/jira/browse/YARN-6840
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Jonathan Hung
> Attachments: YARN-6840-YARN-5734.001.patch, 
> YARN-6840-YARN-5734.002.patch
>
>
> Right now there is only in-memory and leveldb based configuration store 
> supported. Need one which supports RM HA.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

2017-08-25 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16142133#comment-16142133
 ] 

Hadoop QA commented on YARN-6840:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 5 new or modified test 
files. {color} |
|| || || || {color:brown} YARN-5734 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  2m 
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 
18s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  9m 
27s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
54s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
17s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
11s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
56s{color} | {color:green} YARN-5734 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  5m 
41s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 53s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch 
generated 36 new + 328 unchanged - 3 fixed = 364 total (was 331) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m 
14s{color} | {color:red} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
27s{color} | {color:red} 
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager
 generated 1 new + 347 unchanged - 0 fixed = 348 total (was 347) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  0m 28s{color} 
| {color:red} hadoop-yarn-api in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 45m  6s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
27s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 99m 36s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | 
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 |
|  |  Found reliance on default encoding in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.conf.ZKConfigurationStore.confirmMutation(long,
 boolean):in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.conf.ZKConfigurationStore.confirmMutation(long,
 boolean): String.getBytes()  At ZKConfigurationStore.java:[line 190] |
| Failed junit tests | hadoop.yarn.conf.TestYarnConfigurationFields |
|   | 
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6840 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12883778/YARN-6840-YARN-5734.001.patch
 |
| Optional Tests |  asflicense

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

2017-08-25 Thread Jonathan Hung (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16141994#comment-16141994
 ] 

Jonathan Hung commented on YARN-6840:
-

Hi [~leftnoteasy] and [~xgong], posted a zookeeper store implementation 001 
version.

> Implement zookeeper based store for scheduler configuration updates
> ---
>
> Key: YARN-6840
> URL: https://issues.apache.org/jira/browse/YARN-6840
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Jonathan Hung
> Attachments: YARN-6840-YARN-5734.001.patch
>
>
> Right now there is only in-memory and leveldb based configuration store 
> supported. Need one which supports RM HA.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

2017-08-17 Thread Jonathan Hung (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131313#comment-16131313
 ] 

Jonathan Hung commented on YARN-6840:
-

Great, thanks [~wangda]/[~subru], will base this ticket on HADOOP-14741.

> Implement zookeeper based store for scheduler configuration updates
> ---
>
> Key: YARN-6840
> URL: https://issues.apache.org/jira/browse/YARN-6840
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Jonathan Hung
>
> Right now there is only in-memory and leveldb based configuration store 
> supported. Need one which supports RM HA.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

2017-08-11 Thread Subru Krishnan (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124388#comment-16124388
 ] 

Subru Krishnan commented on YARN-6840:
--

[~jonathan_huang]/[~leftnoteasy], I committed HADOOP-14741 as it's blocking 
YARN-6900. But do take a look and let us know if you have any feedback, we can 
always fix in a follow-up jira. 

FYI the rebased version of YARN-6900 post HADOOP-14741 commit has only 
Federation specific ZK code, so already looking good :).

> Implement zookeeper based store for scheduler configuration updates
> ---
>
> Key: YARN-6840
> URL: https://issues.apache.org/jira/browse/YARN-6840
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Jonathan Hung
>
> Right now there is only in-memory and leveldb based configuration store 
> supported. Need one which supports RM HA.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

2017-08-11 Thread Wangda Tan (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124041#comment-16124041
 ] 

Wangda Tan commented on YARN-6840:
--

[~jhung], HADOOP-14741 is close to be committed, could you check if you can use 
the common framework added by HADOOP-14741?

> Implement zookeeper based store for scheduler configuration updates
> ---
>
> Key: YARN-6840
> URL: https://issues.apache.org/jira/browse/YARN-6840
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Jonathan Hung
>
> Right now there is only in-memory and leveldb based configuration store 
> supported. Need one which supports RM HA.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

2017-08-02 Thread Kai Zheng (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112061#comment-16112061
 ] 

Kai Zheng commented on YARN-6840:
-

Sharing some codes in Hadoop common sounds good. Or further, not sure if it's 
good to have a generic Hadoop StatesStore (ZK maybe as the default back end) 
across both HDFS and YARN. As I don't think there is very heavy usage of the 
store (so can't share) so I guess the same store could be used to simplify the 
deployment. We might evolve towards that.

> Implement zookeeper based store for scheduler configuration updates
> ---
>
> Key: YARN-6840
> URL: https://issues.apache.org/jira/browse/YARN-6840
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Jonathan Hung
>
> Right now there is only in-memory and leveldb based configuration store 
> supported. Need one which supports RM HA.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

2017-08-02 Thread Subru Krishnan (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112017#comment-16112017
 ] 

Subru Krishnan commented on YARN-6840:
--

[~zhz]/[~leftnoteasy], I am reviewing YARN-6900 and I observed parts of the 
curator code that are dealing with ZK interactions are being repeated so I have 
requested [~elgoiri] to refactor them to a common util class 
[here|https://issues.apache.org/jira/browse/YARN-6900?focusedCommentId=16111986].

> Implement zookeeper based store for scheduler configuration updates
> ---
>
> Key: YARN-6840
> URL: https://issues.apache.org/jira/browse/YARN-6840
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Jonathan Hung
>
> Right now there is only in-memory and leveldb based configuration store 
> supported. Need one which supports RM HA.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

2017-08-02 Thread Wangda Tan (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111748#comment-16111748
 ] 

Wangda Tan commented on YARN-6840:
--

[~elgoiri], thanks for your comments, actually I'm looking for some common 
functionalities on top of Curator. If there's no much commonality between these 
features, I'm fine to implement this feature by using Curator separately.

> Implement zookeeper based store for scheduler configuration updates
> ---
>
> Key: YARN-6840
> URL: https://issues.apache.org/jira/browse/YARN-6840
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Jonathan Hung
>
> Right now there is only in-memory and leveldb based configuration store 
> supported. Need one which supports RM HA.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

2017-08-02 Thread Inigo Goiri (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111717#comment-16111717
 ] 

Inigo Goiri commented on YARN-6840:
---

[~wangda], we are already using curator for HDFS-10631 and YARN-6900.
Curator already wraps some of the common functionality.
There might be some functions that can be made into a utility but not sure if 
there is too much value.

> Implement zookeeper based store for scheduler configuration updates
> ---
>
> Key: YARN-6840
> URL: https://issues.apache.org/jira/browse/YARN-6840
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Jonathan Hung
>
> Right now there is only in-memory and leveldb based configuration store 
> supported. Need one which supports RM HA.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

2017-08-02 Thread Wangda Tan (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111468#comment-16111468
 ] 

Wangda Tan commented on YARN-6840:
--

[~zhz],

I'm not sure if we can leverage them, it will be good if we can have same basic 
ZK implementation for all related tasks. However, pulling common code out may 
potentially destabilize existing features. I don't have strong opinion on this, 
if existing feature owner is supportive for this, I will be good :).

+ [~elgoiri]/[~subru]. Do you think we can pull common ZK implementation out 
and make it in this feature as well? 

> Implement zookeeper based store for scheduler configuration updates
> ---
>
> Key: YARN-6840
> URL: https://issues.apache.org/jira/browse/YARN-6840
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Jonathan Hung
>
> Right now there is only in-memory and leveldb based configuration store 
> supported. Need one which supports RM HA.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

[jira] [Commented] (YARN-6840) Implement zookeeper based store for scheduler configuration updates

31 matches

Site Navigation

Mail list logo

Footer information