[jira] [Commented] (YARN-9699) [Phase 1] Migration tool that help to generate CS config based on FS config

2019-10-14 Thread Szilard Nemeth (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16950780#comment-16950780
 ] 

Szilard Nemeth commented on YARN-9699:
--

Will commit this EOD (CEST) if no ojections.

> [Phase 1] Migration tool that help to generate CS config based on FS config
> ---
>
> Key: YARN-9699
> URL: https://issues.apache.org/jira/browse/YARN-9699
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wanqiang Ji
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: FS_to_CS_migration_POC.patch, YARN-9699-003.patch, 
> YARN-9699-004.patch, YARN-9699-005.patch, YARN-9699-006.patch, 
> YARN-9699-007.patch, YARN-9699-008.patch, YARN-9699-009.patch, 
> YARN-9699-010.patch, YARN-9699-011.patch, YARN-9699-012.patch, 
> YARN-9699-013.patch, YARN-9699-014.patch, YARN-9699-015.patch, 
> YARN-9699-016.patch, YARN-9699-017.patch, YARN-9699.001.patch, 
> YARN-9699.002.patch
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9699) [Phase 1] Migration tool that help to generate CS config based on FS config

2019-10-11 Thread Hadoop QA (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949447#comment-16949447
 ] 

Hadoop QA commented on YARN-9699:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 28m 
29s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 16 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
49s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
14m 58s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
14s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  6m 
39s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
1m 19s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch 
generated 4 new + 137 unchanged - 7 fixed = 141 total (was 144) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red}  0m 
31s{color} | {color:red} hadoop-yarn-site in the patch failed. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:red}-1{color} | {color:red} xml {color} | {color:red}  0m 10s{color} | 
{color:red} The patch has 2 ill-formed XML file(s). {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m  9s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 81m 
13s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch 
passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
26s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
45s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}182m  7s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| XML | Parsing Error(s): |
|   |

[jira] [Commented] (YARN-9699) [Phase 1] Migration tool that help to generate CS config based on FS config

2019-10-11 Thread Szilard Nemeth (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949311#comment-16949311
 ] 

Szilard Nemeth commented on YARN-9699:
--

Added a final patch (017) that fixes checkstyle/whitespace issues and adds a 
warning text, discussed above.

> [Phase 1] Migration tool that help to generate CS config based on FS config
> ---
>
> Key: YARN-9699
> URL: https://issues.apache.org/jira/browse/YARN-9699
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wanqiang Ji
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: FS_to_CS_migration_POC.patch, YARN-9699-003.patch, 
> YARN-9699-004.patch, YARN-9699-005.patch, YARN-9699-006.patch, 
> YARN-9699-007.patch, YARN-9699-008.patch, YARN-9699-009.patch, 
> YARN-9699-010.patch, YARN-9699-011.patch, YARN-9699-012.patch, 
> YARN-9699-013.patch, YARN-9699-014.patch, YARN-9699-015.patch, 
> YARN-9699-016.patch, YARN-9699-017.patch, YARN-9699.001.patch, 
> YARN-9699.002.patch
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9699) [Phase 1] Migration tool that help to generate CS config based on FS config

2019-10-11 Thread Gergely Pollak (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949277#comment-16949277
 ] 

Gergely Pollak commented on YARN-9699:
--

Hi [~snemeth] and [~pbacsko], thank you for the patch, LGTM+1 (non-binding)

> [Phase 1] Migration tool that help to generate CS config based on FS config
> ---
>
> Key: YARN-9699
> URL: https://issues.apache.org/jira/browse/YARN-9699
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wanqiang Ji
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: FS_to_CS_migration_POC.patch, YARN-9699-003.patch, 
> YARN-9699-004.patch, YARN-9699-005.patch, YARN-9699-006.patch, 
> YARN-9699-007.patch, YARN-9699-008.patch, YARN-9699-009.patch, 
> YARN-9699-010.patch, YARN-9699-011.patch, YARN-9699-012.patch, 
> YARN-9699-013.patch, YARN-9699-014.patch, YARN-9699-015.patch, 
> YARN-9699-016.patch, YARN-9699.001.patch, YARN-9699.002.patch
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9699) [Phase 1] Migration tool that help to generate CS config based on FS config

2019-10-10 Thread Hadoop QA (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948941#comment-16948941
 ] 

Hadoop QA commented on YARN-9699:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 25m 
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 16 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m 
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
15m 52s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
9s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m 
37s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
1m 24s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch 
generated 23 new + 137 unchanged - 7 fixed = 160 total (was 144) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
23s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 8 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:red}-1{color} | {color:red} xml {color} | {color:red}  0m 12s{color} | 
{color:red} The patch has 2 ill-formed XML file(s). {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
13m 40s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 95m 
46s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch 
passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
33s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  1m 
 0s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}201m  5s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| XML | Parsing Error(s): |
|   |

[jira] [Commented] (YARN-9699) [Phase 1] Migration tool that help to generate CS config based on FS config

2019-10-10 Thread Hadoop QA (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948926#comment-16948926
 ] 

Hadoop QA commented on YARN-9699:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
27s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 16 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  5m 
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m 
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
15m 28s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
59s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m 
10s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
1m 17s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch 
generated 23 new + 136 unchanged - 7 fixed = 159 total (was 143) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 8 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:red}-1{color} | {color:red} xml {color} | {color:red}  0m 10s{color} | 
{color:red} The patch has 2 ill-formed XML file(s). {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
13m 26s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 80m 
54s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch 
passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
21s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
39s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}163m 42s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| XML | Parsing Error(s): |
|   |

[jira] [Commented] (YARN-9699) [Phase 1] Migration tool that help to generate CS config based on FS config

2019-10-10 Thread Szilard Nemeth (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948855#comment-16948855
 ] 

Szilard Nemeth commented on YARN-9699:
--

Hi [~pbacsko]!

Just talked to [~leftnoteasy] offline and he told we need to put a disclaimer 
text to the script so users can immediately see that this is an alpha feature 
and they need to use it with care. Can you add it to output of the script and 
to the documentation (javadoc, etc)?

 

Thanks!

> [Phase 1] Migration tool that help to generate CS config based on FS config
> ---
>
> Key: YARN-9699
> URL: https://issues.apache.org/jira/browse/YARN-9699
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wanqiang Ji
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: FS_to_CS_migration_POC.patch, YARN-9699-003.patch, 
> YARN-9699-004.patch, YARN-9699-005.patch, YARN-9699-006.patch, 
> YARN-9699-007.patch, YARN-9699-008.patch, YARN-9699-009.patch, 
> YARN-9699-010.patch, YARN-9699-011.patch, YARN-9699-012.patch, 
> YARN-9699-013.patch, YARN-9699-014.patch, YARN-9699-015.patch, 
> YARN-9699-016.patch, YARN-9699.001.patch, YARN-9699.002.patch
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9699) [Phase 1] Migration tool that help to generate CS config based on FS config

2019-10-10 Thread Szilard Nemeth (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948795#comment-16948795
 ] 

Szilard Nemeth commented on YARN-9699:
--

Hi [~pbacsko] !


Did a final round of review.
Apart from fixing the checkstyle issues, I have these comments which are fixed 
with the patch I'm attaching now.

1. In FSConfigToCSConfigArgumentHandler#parseAndConvert: 
IllegalArgumentException have not caught therefore it had been resulted in an 
uncatched exception, up until ResourceManager.main() would caught it, but in 
this case, the full stacktrace would had been printed to the console.
Instead, we should log the exception and show a minimum amount of information 
on the console (stderr).
More on this in the same method, we should also catch MissingArgumentException 
and swallow its stacktrace, similarly to other exceptions caught in this method.
As FSConfigToCSConfigArgumentHandler#parseAndConvert does not throw exceptions 
anymore (as it catches PreconditionException, UnsupportedPropertyException and 
ConversionException), the code in ResourceManager#main could not reach the 
System.exit(-1) call.

The following piece of code could easily test the missing option vs. handling 
the exception properly case:

 
{code:java}
@Test
 public void testResourceManagerConvertFSConfigurationMissingOption()
 throws Exception {
 setupFSConfigConversionFiles();
final String mainSwitch = "-convert-fs-configuration";
 FSConfigToCSConfigConverter mockConverter =
 mock(FSConfigToCSConfigConverter.class);
ResourceManager.initFSArgumentHandler(mockConverter);
 ResourceManager.main(new String[] {mainSwitch});
 }
{code}
 

However, I found no easy way to test if System.exit() was called with a 
specific value, see: 
https://stackoverflow.com/questions/309396/java-how-to-test-methods-that-call-system-exit
Moreover, if System.exit is called, the JUnit testcase will fail so I'd refrain 
to add a new testcase for this, which I don't like at all but this is what it 
is.

2. Nit: FSYarnSiteConverter: Typo in javadoc of this class. "Converts a Fair 
Schedule". Last word should be "scheduler".

3. Nit: QueuePlacementConverter#convertPlacementPolicy: rulehandler parameter 
should be camelcase.

4. QueuePlacementConverter: "u:%user:%user" is occuring twice, it should be 
extracted to a constant. 
I'd also extract %user, %primary_group and %secondary_group to constants.

5. QueuePlacementConverter: if-else chain is hard to read. Please extract at 
least the nested rule handling part to a separate method.

6. Nit: FSConfigToCSConfigConverter: There are a bunch of occasions of strings: 
"yarn-site.xml" / "capacity-scheduler.xml" and we have constants for them. We 
should use the constants wherever possible.

7. Nit: FSConfigToCSConfigConverter#prepareOutputFiles: Inconsistent naming: 
yarnSiteXmlOutputFile / schedulerXmlOutput. This can confuse readers.

8. Nit: Name of method FSConfigToCSConfigConverter#emitDefaultMaxAMshare: Share 
should be with capital S.

9. FSConfigToCSConfigConverter#convert receives the cluster resource as 
parameter but this method only sets it to a field and does nothing else with 
it: The caller code should simply set the field and not pass the resource as a 
parameter.

10. Nit: In FSConfigToCSConfigConverter#convertCapacitySchedulerXml: 
properties.entrySet().forEach could be replaced with properties.forEach.

11. Nit: In FSConfigToCSConfigConverter#emitACLs: entrySet().stream().forEach 
can be replaced with entrySet().forEach

12. Nit: In FSConfigToCSConfigRuleHandler#loadRulesFromFile: rules variable 
could be inlined.

13. Nit: FSQueueConverter#MAXRUNNINGAPPS_UNSET should be renamed to 
MAX_RUNNING_APPS_UNSET.

14. In FSQueueConverter#convertQueueHierarchy: Please move comments above 
method calls to their javadoc instead.

15. Nit: In FSQueueConverter#emitChildQueues: Call to toString() is unnecessary 
in statement: 
capacitySchedulerConfig.set(PREFIX + queueName + ".queues",
 childQueues.toString());

16. Nit: FSQueueConverter#emitChildCapacity: entrySet().stream().forEach can be 
replaced with entrySet().forEach

 

I fixed the following issues above and uploaded a (hopefully) final patch. 
Waiting for jenkins results...
In the meantime, [~sunilg] : Do you have some free cycles to review this patch?
If not, I will commit the latest patch EOD tomorrow.

 

Thanks!

> [Phase 1] Migration tool that help to generate CS config based on FS config
> ---
>
> Key: YARN-9699
> URL: https://issues.apache.org/jira/browse/YARN-9699
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wanqiang Ji
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: FS_to_CS_migration_POC.patch, YARN-9699-003.patch, 
> YARN-9699-004.patch, YARN-9699-005.patch,

[jira] [Commented] (YARN-9699) [Phase 1] Migration tool that help to generate CS config based on FS config

2019-10-10 Thread Hadoop QA (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948724#comment-16948724
 ] 

Hadoop QA commented on YARN-9699:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
49s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 16 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
49s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m  
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
15m 25s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
1s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m 
20s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
1m 18s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch 
generated 11 new + 137 unchanged - 7 fixed = 148 total (was 144) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} xml {color} | {color:red}  0m 11s{color} | 
{color:red} The patch has 2 ill-formed XML file(s). {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
13m 51s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 85m 
39s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch 
passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
20s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
44s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}163m 49s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| XML | Parsing Error(s): |
|   | 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml
 |
|   |