[jira] [Commented] (YARN-10687) Add option to disable/enable free disk space checking and percentage checking for full and not-full disks

2021-03-12 Thread Qi Zhu (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17300390#comment-17300390
 ] 

Qi Zhu commented on YARN-10687:
---

Thanks [~Jim_Brennan] for last check.

> Add option to disable/enable free disk space checking and percentage checking 
> for full and not-full disks
> -
>
> Key: YARN-10687
> URL: https://issues.apache.org/jira/browse/YARN-10687
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 3.2.2, 3.4.0
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10687.001.patch, YARN-10687.002.patch, 
> YARN-10687.003.patch, YARN-10687.004.patch
>
>
> Now the two option:
> max-disk-utilization-per-disk-percentage
>  min-free-space-per-disk-mb
> for full/not full disk check are all enabled always, i think it's more 
> reasonable to enable or disable this, and default will be all enabled.
>  
> In our clusters, when the disk is so huge we want to use  
> min-free-space-per-disk-mb.
> In our clusters,  when the disk is so small we want to use 
> max-disk-utilization-per-disk-percentage.
>  
> We should make this more reasonable and not confused.
>  
> cc [~pbacsko]  [~Jim_Brennan]  [~ebadger]  [~gandras] 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10687) Add option to disable/enable free disk space checking and percentage checking for full and not-full disks

2021-03-12 Thread Jim Brennan (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17300369#comment-17300369
 ] 

Jim Brennan commented on YARN-10687:


Thanks for the updates [~zhuqi]!  I am +1 on patch 004.   I will commit this 
later today.

> Add option to disable/enable free disk space checking and percentage checking 
> for full and not-full disks
> -
>
> Key: YARN-10687
> URL: https://issues.apache.org/jira/browse/YARN-10687
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 3.2.2, 3.4.0
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10687.001.patch, YARN-10687.002.patch, 
> YARN-10687.003.patch, YARN-10687.004.patch
>
>
> Now the two option:
> max-disk-utilization-per-disk-percentage
>  min-free-space-per-disk-mb
> for full/not full disk check are all enabled always, i think it's more 
> reasonable to enable or disable this, and default will be all enabled.
>  
> In our clusters, when the disk is so huge we want to use  
> min-free-space-per-disk-mb.
> In our clusters,  when the disk is so small we want to use 
> max-disk-utilization-per-disk-percentage.
>  
> We should make this more reasonable and not confused.
>  
> cc [~pbacsko]  [~Jim_Brennan]  [~ebadger]  [~gandras] 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10687) Add option to disable/enable free disk space checking and percentage checking for full and not-full disks

2021-03-12 Thread Hadoop QA (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17300240#comment-17300240
 ] 

Hadoop QA commented on YARN-10687:
--

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime ||  Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
41s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green}  0m  
0s{color} | {color:green}{color} | {color:green} No case conflicting files 
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green}{color} | {color:green} The patch does not contain any 
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green}  0m  0s{color} 
| {color:green}test4tests{color} | {color:green} The patch appears to include 1 
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
43s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for 
branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 
23s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  9m  
6s{color} | {color:green}{color} | {color:green} trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
50s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
46s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
54s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
18m  2s{color} | {color:green}{color} | {color:green} branch has no errors when 
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
30s{color} | {color:green}{color} | {color:green} trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
41s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 28m 
49s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are 
enabled, using SpotBugs. {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green}  5m 
38s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
26s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for 
patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
54s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
28s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  8m 
28s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
54s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m 
54s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
45s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
36s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace 
issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green}{color} | {color:green} The patch has no ill-formed 
XML file. {color} |
| {color:green}+1{color} | {color:gree

[jira] [Commented] (YARN-10687) Add option to disable/enable free disk space checking and percentage checking for full and not-full disks

2021-03-12 Thread Qi Zhu (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17300155#comment-17300155
 ] 

Qi Zhu commented on YARN-10687:
---

Fixed the remaining checkstyle in latest patch.

> Add option to disable/enable free disk space checking and percentage checking 
> for full and not-full disks
> -
>
> Key: YARN-10687
> URL: https://issues.apache.org/jira/browse/YARN-10687
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 3.2.2, 3.4.0
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10687.001.patch, YARN-10687.002.patch, 
> YARN-10687.003.patch, YARN-10687.004.patch
>
>
> Now the two option:
> max-disk-utilization-per-disk-percentage
>  min-free-space-per-disk-mb
> for full/not full disk check are all enabled always, i think it's more 
> reasonable to enable or disable this, and default will be all enabled.
>  
> In our clusters, when the disk is so huge we want to use  
> min-free-space-per-disk-mb.
> In our clusters,  when the disk is so small we want to use 
> max-disk-utilization-per-disk-percentage.
>  
> We should make this more reasonable and not confused.
>  
> cc [~pbacsko]  [~Jim_Brennan]  [~ebadger]  [~gandras] 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10687) Add option to disable/enable free disk space checking and percentage checking for full and not-full disks

2021-03-11 Thread Hadoop QA (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17300076#comment-17300076
 ] 

Hadoop QA commented on YARN-10687:
--

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime ||  Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
41s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green}  0m  
0s{color} | {color:green}{color} | {color:green} No case conflicting files 
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green}{color} | {color:green} The patch does not contain any 
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green}  0m  0s{color} 
| {color:green}test4tests{color} | {color:green} The patch appears to include 1 
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  2m  
2s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for 
branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m 
10s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m  
6s{color} | {color:green}{color} | {color:green} trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
47s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
45s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
54s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
20m  1s{color} | {color:green}{color} | {color:green} branch has no errors when 
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
28s{color} | {color:green}{color} | {color:green} trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
38s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 31m  
4s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are 
enabled, using SpotBugs. {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green}  6m  
1s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
28s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for 
patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
 2s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  9m 
54s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  9m 
54s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
55s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  8m 
55s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
1m 50s{color} | 
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/787/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt{color}
 | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch generated 2 new + 
238 unchanged - 0 fixed = 240 total (was 238) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
35s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace 
issues. {

[jira] [Commented] (YARN-10687) Add option to disable/enable free disk space checking and percentage checking for full and not-full disks

2021-03-11 Thread Qi Zhu (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17300019#comment-17300019
 ] 

Qi Zhu commented on YARN-10687:
---

Thanks a lot [~Jim_Brennan] for patient review.

I have fixed it in latest patch, and added the unit test.:D

> Add option to disable/enable free disk space checking and percentage checking 
> for full and not-full disks
> -
>
> Key: YARN-10687
> URL: https://issues.apache.org/jira/browse/YARN-10687
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 3.2.2, 3.4.0
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10687.001.patch, YARN-10687.002.patch, 
> YARN-10687.003.patch
>
>
> Now the two option:
> max-disk-utilization-per-disk-percentage
>  min-free-space-per-disk-mb
> for full/not full disk check are all enabled always, i think it's more 
> reasonable to enable or disable this, and default will be all enabled.
>  
> In our clusters, when the disk is so huge we want to use  
> min-free-space-per-disk-mb.
> In our clusters,  when the disk is so small we want to use 
> max-disk-utilization-per-disk-percentage.
>  
> We should make this more reasonable and not confused.
>  
> cc [~pbacsko]  [~Jim_Brennan]  [~ebadger]  [~gandras] 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10687) Add option to disable/enable free disk space checking and percentage checking for full and not-full disks

2021-03-11 Thread Jim Brennan (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17299827#comment-17299827
 ] 

Jim Brennan commented on YARN-10687:


Thanks for updating [~zhuqi]!  Can you please add a unit test to 
TestDirectoryCollection?  Also, did you see [~gandras]'s suggestion about 
changing the property names?  e.g., {{disk-utilization-percentage.enabled}}?

> Add option to disable/enable free disk space checking and percentage checking 
> for full and not-full disks
> -
>
> Key: YARN-10687
> URL: https://issues.apache.org/jira/browse/YARN-10687
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 3.2.2, 3.4.0
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10687.001.patch, YARN-10687.002.patch
>
>
> Now the two option:
> max-disk-utilization-per-disk-percentage
>  min-free-space-per-disk-mb
> for full/not full disk check are all enabled always, i think it's more 
> reasonable to enable or disable this, and default will be all enabled.
>  
> In our clusters, when the disk is so huge we want to use  
> min-free-space-per-disk-mb.
> In our clusters,  when the disk is so small we want to use 
> max-disk-utilization-per-disk-percentage.
>  
> We should make this more reasonable and not confused.
>  
> cc [~pbacsko]  [~Jim_Brennan]  [~ebadger]  [~gandras] 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10687) Add option to disable/enable free disk space checking and percentage checking for full and not-full disks

2021-03-11 Thread Hadoop QA (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17299786#comment-17299786
 ] 

Hadoop QA commented on YARN-10687:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime ||  Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  1m 
16s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green}  0m  
0s{color} | {color:green}{color} | {color:green} No case conflicting files 
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green}{color} | {color:green} The patch does not contain any 
@author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red}{color} | {color:red} The patch doesn't appear to 
include any new or modified tests. Please justify why no new tests are needed 
for this patch. Also please list what manual steps were performed to verify 
this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
48s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for 
branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m 
38s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 
20s{color} | {color:green}{color} | {color:green} trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
18s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
41s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
29s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
19m 43s{color} | {color:green}{color} | {color:green} branch has no errors when 
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m  
5s{color} | {color:green}{color} | {color:green} trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
14s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 29m 
30s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are 
enabled, using SpotBugs. {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green}  5m 
30s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
20s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for 
patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
50s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  9m 
10s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  9m 
10s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
12s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  8m 
12s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
38s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
21s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace 
issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} 

[jira] [Commented] (YARN-10687) Add option to disable/enable free disk space checking and percentage checking for full and not-full disks

2021-03-11 Thread Qi Zhu (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17299649#comment-17299649
 ] 

Qi Zhu commented on YARN-10687:
---

[~Jim_Brennan] [~gandras]

Updated the fix in latest patch.

Thanks.

> Add option to disable/enable free disk space checking and percentage checking 
> for full and not-full disks
> -
>
> Key: YARN-10687
> URL: https://issues.apache.org/jira/browse/YARN-10687
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 3.2.2, 3.4.0
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10687.001.patch, YARN-10687.002.patch
>
>
> Now the two option:
> max-disk-utilization-per-disk-percentage
>  min-free-space-per-disk-mb
> for full/not full disk check are all enabled always, i think it's more 
> reasonable to enable or disable this, and default will be all enabled.
>  
> In our clusters, when the disk is so huge we want to use  
> min-free-space-per-disk-mb.
> In our clusters,  when the disk is so small we want to use 
> max-disk-utilization-per-disk-percentage.
>  
> We should make this more reasonable and not confused.
>  
> cc [~pbacsko]  [~Jim_Brennan]  [~ebadger]  [~gandras] 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10687) Add option to disable/enable free disk space checking and percentage checking for full and not-full disks

2021-03-11 Thread Qi Zhu (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17299616#comment-17299616
 ] 

Qi Zhu commented on YARN-10687:
---

Thanks a lot [~Jim_Brennan] for review and confirm.

I will update your valid suggestions:

"we should also update the descriptions for the threshold properties to 
indicate that they only apply when the corresponding {{enabled}} property is 
true."

in next patch.

> Add option to disable/enable free disk space checking and percentage checking 
> for full and not-full disks
> -
>
> Key: YARN-10687
> URL: https://issues.apache.org/jira/browse/YARN-10687
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 3.2.2, 3.4.0
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10687.001.patch
>
>
> Now the two option:
> max-disk-utilization-per-disk-percentage
>  min-free-space-per-disk-mb
> for full/not full disk check are all enabled always, i think it's more 
> reasonable to enable or disable this, and default will be all enabled.
>  
> In our clusters, when the disk is so huge we want to use  
> min-free-space-per-disk-mb.
> In our clusters,  when the disk is so small we want to use 
> max-disk-utilization-per-disk-percentage.
>  
> We should make this more reasonable and not confused.
>  
> cc [~pbacsko]  [~Jim_Brennan]  [~ebadger]  [~gandras] 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10687) Add option to disable/enable free disk space checking and percentage checking for full and not-full disks

2021-03-11 Thread Jim Brennan (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17299613#comment-17299613
 ] 

Jim Brennan commented on YARN-10687:


Thanks [~zhuqi]!  These are not strictly needed, because the values can be set 
to effectively enable/disable each threshold.  But I agree that having these 
makes the configuration options easier to understand.  In addition the the name 
changes recommended by [~gandras], we should also update the descriptions for 
the threshold properties to indicate that they only apply when the 
corresponding {{enabled}} property is true.


> Add option to disable/enable free disk space checking and percentage checking 
> for full and not-full disks
> -
>
> Key: YARN-10687
> URL: https://issues.apache.org/jira/browse/YARN-10687
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 3.2.2, 3.4.0
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10687.001.patch
>
>
> Now the two option:
> max-disk-utilization-per-disk-percentage
>  min-free-space-per-disk-mb
> for full/not full disk check are all enabled always, i think it's more 
> reasonable to enable or disable this, and default will be all enabled.
>  
> In our clusters, when the disk is so huge we want to use  
> min-free-space-per-disk-mb.
> In our clusters,  when the disk is so small we want to use 
> max-disk-utilization-per-disk-percentage.
>  
> We should make this more reasonable and not confused.
>  
> cc [~pbacsko]  [~Jim_Brennan]  [~ebadger]  [~gandras] 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10687) Add option to disable/enable free disk space checking and percentage checking for full and not-full disks

2021-03-11 Thread Hadoop QA (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17299447#comment-17299447
 ] 

Hadoop QA commented on YARN-10687:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime ||  Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 34m 
37s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green}  0m  
0s{color} | {color:green}{color} | {color:green} No case conflicting files 
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green}{color} | {color:green} The patch does not contain any 
@author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red}{color} | {color:red} The patch doesn't appear to 
include any new or modified tests. Please justify why no new tests are needed 
for this patch. Also please list what manual steps were performed to verify 
this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
50s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for 
branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m 
23s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  9m 
48s{color} | {color:green}{color} | {color:green} trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
18s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
40s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
31s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
19m 43s{color} | {color:green}{color} | {color:green} branch has no errors when 
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m  
4s{color} | {color:green}{color} | {color:green} trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
16s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 29m 
25s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are 
enabled, using SpotBugs. {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green}  5m 
26s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
20s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for 
patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
51s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  9m 
21s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  9m 
21s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
18s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  8m 
18s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
37s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
20s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace 
issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} 

[jira] [Commented] (YARN-10687) Add option to disable/enable free disk space checking and percentage checking for full and not-full disks

2021-03-10 Thread Qi Zhu (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17299387#comment-17299387
 ] 

Qi Zhu commented on YARN-10687:
---

Thanks a lot [~gandras] for review, very valid suggestion to me.

I will update this in next patch.:D

> Add option to disable/enable free disk space checking and percentage checking 
> for full and not-full disks
> -
>
> Key: YARN-10687
> URL: https://issues.apache.org/jira/browse/YARN-10687
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 3.2.2, 3.4.0
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10687.001.patch
>
>
> Now the two option:
> max-disk-utilization-per-disk-percentage
>  min-free-space-per-disk-mb
> for full/not full disk check are all enabled always, i think it's more 
> reasonable to enable or disable this, and default will be all enabled.
>  
> In our clusters, when the disk is so huge we want to use  
> min-free-space-per-disk-mb.
> In our clusters,  when the disk is so small we want to use 
> max-disk-utilization-per-disk-percentage.
>  
> We should make this more reasonable and not confused.
>  
> cc [~pbacsko]  [~Jim_Brennan]  [~ebadger]  [~gandras] 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10687) Add option to disable/enable free disk space checking and percentage checking for full and not-full disks

2021-03-10 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17299386#comment-17299386
 ] 

Andras Gyori commented on YARN-10687:
-

Thank you [~zhuqi] for the patch. I am by no means expert on this area, but it 
does not seem uniform to me either. I would, however choose different names for 
these newly introduced properties, something more descriptive like:
 * diskUtilizationThresholdEnabled
 ** Enables the disk utilisation percentage threshold for disk health checker
 * diskFreeSpaceThresholdEnabled
 ** Enables the minimum disk free space threshold for disk health checker

Currently, the disk-free-space-enabled is quite misleading (it is the threshold 
we are enabling). Also, I think the disk-utilization-percentage.enabled 
convention is more frequently used in the configuration, than the 
disk-utilization-percentage-enable.

> Add option to disable/enable free disk space checking and percentage checking 
> for full and not-full disks
> -
>
> Key: YARN-10687
> URL: https://issues.apache.org/jira/browse/YARN-10687
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 3.2.2, 3.4.0
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10687.001.patch
>
>
> Now the two option:
> max-disk-utilization-per-disk-percentage
>  min-free-space-per-disk-mb
> for full/not full disk check are all enabled always, i think it's more 
> reasonable to enable or disable this, and default will be all enabled.
>  
> In our clusters, when the disk is so huge we want to use  
> min-free-space-per-disk-mb.
> In our clusters,  when the disk is so small we want to use 
> max-disk-utilization-per-disk-percentage.
>  
> We should make this more reasonable and not confused.
>  
> cc [~pbacsko]  [~Jim_Brennan]  [~ebadger]  [~gandras] 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10687) Add option to disable/enable free disk space checking and percentage checking for full and not-full disks

2021-03-10 Thread Qi Zhu (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17299336#comment-17299336
 ] 

Qi Zhu commented on YARN-10687:
---

 [~pbacsko]  [~Jim_Brennan]  [~ebadger]  [~gandras] 

If you could help review this?

Thanks.:D

> Add option to disable/enable free disk space checking and percentage checking 
> for full and not-full disks
> -
>
> Key: YARN-10687
> URL: https://issues.apache.org/jira/browse/YARN-10687
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 3.2.2, 3.4.0
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10687.001.patch
>
>
> Now the two option:
> max-disk-utilization-per-disk-percentage
>  min-free-space-per-disk-mb
> for full/not full disk check are all enabled always, i think it's more 
> reasonable to enable or disable this, and default will be all enabled.
>  
> In our clusters, when the disk is so huge we want to use  
> min-free-space-per-disk-mb.
> In our clusters,  when the disk is so small we want to use 
> max-disk-utilization-per-disk-percentage.
>  
> We should make this more reasonable and not confused.
>  
> cc [~pbacsko]  [~Jim_Brennan]  [~ebadger]  [~gandras] 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org