subject:"\[jira\] \[Commented\] \(HDFS\-3570\) Balancer shouldn't rely on \"DFS Space Used %\" as that ignores non\-DFS used space"

[jira] [Commented] (HDFS-3570) Balancer shouldn't rely on "DFS Space Used %" as that ignores non-DFS used space

2024-01-03 Thread Shilun Fan (Jira)



[ 
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17802564#comment-17802564
 ] 

Shilun Fan commented on HDFS-3570:
--

Bulk update: moved all 3.4.0 non-blocker issues, please move back if it is a 
blocker. Retarget 3.5.0.

> Balancer shouldn't rely on "DFS Space Used %" as that ignores non-DFS used 
> space
> 
>
> Key: HDFS-3570
> URL: https://issues.apache.org/jira/browse/HDFS-3570
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: balancer  mover
>Affects Versions: 2.0.0-alpha
>Reporter: Harsh J
>Assignee: Ashutosh Gupta
>Priority: Minor
>  Labels: pull-request-available
> Attachments: HDFS-3570.003.patch, HDFS-3570.2.patch, 
> HDFS-3570.aash.1.patch
>
>
> Report from a user here: 
> https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pIhNyDVxdVY/b7ENZmEvBjIJ,
>  post archived at http://pastebin.com/eVFkk0A0
> This user had a specific DN that had a large non-DFS usage among 
> dfs.data.dirs, and very little DFS usage (which is computed against total 
> possible capacity). 
> Balancer apparently only looks at the usage, and ignores to consider that 
> non-DFS usage may also be high on a DN/cluster. Hence, it thinks that if a 
> DFS Usage report from DN is 8% only, its got a lot of free space to write 
> more blocks, when that isn't true as shown by the case of this user. It went 
> on scheduling writes to the DN to balance it out, but the DN simply can't 
> accept any more blocks as a result of its disks' state.
> I think it would be better if we _computed_ the actual utilization based on 
> {{(100-(actual remaining space))/(capacity)}}, as opposed to the current 
> {{(dfs used)/(capacity)}}. Thoughts?
> This isn't very critical, however, cause it is very rare to see DN space 
> being used for non DN data, but it does expose a valid bug.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-3570) Balancer shouldn't rely on "DFS Space Used %" as that ignores non-DFS used space

2023-05-10 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721337#comment-17721337
 ] 

ASF GitHub Bot commented on HDFS-3570:
--

ashutoshcipher commented on PR #5044:
URL: https://github.com/apache/hadoop/pull/5044#issuecomment-1542156086

   @ZanderXu @slfan1989 any suggestions ?




> Balancer shouldn't rely on "DFS Space Used %" as that ignores non-DFS used 
> space
> 
>
> Key: HDFS-3570
> URL: https://issues.apache.org/jira/browse/HDFS-3570
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: balancer  mover
>Affects Versions: 2.0.0-alpha
>Reporter: Harsh J
>Assignee: Ashutosh Gupta
>Priority: Minor
>  Labels: pull-request-available
> Attachments: HDFS-3570.003.patch, HDFS-3570.2.patch, 
> HDFS-3570.aash.1.patch
>
>
> Report from a user here: 
> https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pIhNyDVxdVY/b7ENZmEvBjIJ,
>  post archived at http://pastebin.com/eVFkk0A0
> This user had a specific DN that had a large non-DFS usage among 
> dfs.data.dirs, and very little DFS usage (which is computed against total 
> possible capacity). 
> Balancer apparently only looks at the usage, and ignores to consider that 
> non-DFS usage may also be high on a DN/cluster. Hence, it thinks that if a 
> DFS Usage report from DN is 8% only, its got a lot of free space to write 
> more blocks, when that isn't true as shown by the case of this user. It went 
> on scheduling writes to the DN to balance it out, but the DN simply can't 
> accept any more blocks as a result of its disks' state.
> I think it would be better if we _computed_ the actual utilization based on 
> {{(100-(actual remaining space))/(capacity)}}, as opposed to the current 
> {{(dfs used)/(capacity)}}. Thoughts?
> This isn't very critical, however, cause it is very rare to see DN space 
> being used for non DN data, but it does expose a valid bug.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-3570) Balancer shouldn't rely on "DFS Space Used %" as that ignores non-DFS used space

2023-02-06 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17684888#comment-17684888
 ] 

ASF GitHub Bot commented on HDFS-3570:
--

ashutoshcipher commented on PR #5044:
URL: https://github.com/apache/hadoop/pull/5044#issuecomment-1419620667

   It seems to be a bit tricky for writing UT for this. Any suggestions on how 
to take this PR forward or on UT.
   
   @slfan1989 - Any suggestions from your side.
   
   Thanks




> Balancer shouldn't rely on "DFS Space Used %" as that ignores non-DFS used 
> space
> 
>
> Key: HDFS-3570
> URL: https://issues.apache.org/jira/browse/HDFS-3570
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: balancer  mover
>Affects Versions: 2.0.0-alpha
>Reporter: Harsh J
>Assignee: Ashutosh Gupta
>Priority: Minor
>  Labels: pull-request-available
> Attachments: HDFS-3570.003.patch, HDFS-3570.2.patch, 
> HDFS-3570.aash.1.patch
>
>
> Report from a user here: 
> https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pIhNyDVxdVY/b7ENZmEvBjIJ,
>  post archived at http://pastebin.com/eVFkk0A0
> This user had a specific DN that had a large non-DFS usage among 
> dfs.data.dirs, and very little DFS usage (which is computed against total 
> possible capacity). 
> Balancer apparently only looks at the usage, and ignores to consider that 
> non-DFS usage may also be high on a DN/cluster. Hence, it thinks that if a 
> DFS Usage report from DN is 8% only, its got a lot of free space to write 
> more blocks, when that isn't true as shown by the case of this user. It went 
> on scheduling writes to the DN to balance it out, but the DN simply can't 
> accept any more blocks as a result of its disks' state.
> I think it would be better if we _computed_ the actual utilization based on 
> {{(100-(actual remaining space))/(capacity)}}, as opposed to the current 
> {{(dfs used)/(capacity)}}. Thoughts?
> This isn't very critical, however, cause it is very rare to see DN space 
> being used for non DN data, but it does expose a valid bug.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-3570) Balancer shouldn't rely on "DFS Space Used %" as that ignores non-DFS used space

2022-11-01 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17627136#comment-17627136
 ] 

ASF GitHub Bot commented on HDFS-3570:
--

ashutoshcipher commented on PR #5044:
URL: https://github.com/apache/hadoop/pull/5044#issuecomment-1298432054

   Thanks @ZanderXu, I will work on writing UT.




> Balancer shouldn't rely on "DFS Space Used %" as that ignores non-DFS used 
> space
> 
>
> Key: HDFS-3570
> URL: https://issues.apache.org/jira/browse/HDFS-3570
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: balancer  mover
>Affects Versions: 2.0.0-alpha
>Reporter: Harsh J
>Assignee: Ashutosh Gupta
>Priority: Minor
>  Labels: pull-request-available
> Attachments: HDFS-3570.003.patch, HDFS-3570.2.patch, 
> HDFS-3570.aash.1.patch
>
>
> Report from a user here: 
> https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pIhNyDVxdVY/b7ENZmEvBjIJ,
>  post archived at http://pastebin.com/eVFkk0A0
> This user had a specific DN that had a large non-DFS usage among 
> dfs.data.dirs, and very little DFS usage (which is computed against total 
> possible capacity). 
> Balancer apparently only looks at the usage, and ignores to consider that 
> non-DFS usage may also be high on a DN/cluster. Hence, it thinks that if a 
> DFS Usage report from DN is 8% only, its got a lot of free space to write 
> more blocks, when that isn't true as shown by the case of this user. It went 
> on scheduling writes to the DN to balance it out, but the DN simply can't 
> accept any more blocks as a result of its disks' state.
> I think it would be better if we _computed_ the actual utilization based on 
> {{(100-(actual remaining space))/(capacity)}}, as opposed to the current 
> {{(dfs used)/(capacity)}}. Thoughts?
> This isn't very critical, however, cause it is very rare to see DN space 
> being used for non DN data, but it does expose a valid bug.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-3570) Balancer shouldn't rely on "DFS Space Used %" as that ignores non-DFS used space

2022-10-27 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17625005#comment-17625005
 ] 

ASF GitHub Bot commented on HDFS-3570:
--

ashutoshcipher commented on PR #5044:
URL: https://github.com/apache/hadoop/pull/5044#issuecomment-1293292950

   Thank you so much.




> Balancer shouldn't rely on "DFS Space Used %" as that ignores non-DFS used 
> space
> 
>
> Key: HDFS-3570
> URL: https://issues.apache.org/jira/browse/HDFS-3570
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: balancer  mover
>Affects Versions: 2.0.0-alpha
>Reporter: Harsh J
>Assignee: Ashutosh Gupta
>Priority: Minor
>  Labels: pull-request-available
> Attachments: HDFS-3570.003.patch, HDFS-3570.2.patch, 
> HDFS-3570.aash.1.patch
>
>
> Report from a user here: 
> https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pIhNyDVxdVY/b7ENZmEvBjIJ,
>  post archived at http://pastebin.com/eVFkk0A0
> This user had a specific DN that had a large non-DFS usage among 
> dfs.data.dirs, and very little DFS usage (which is computed against total 
> possible capacity). 
> Balancer apparently only looks at the usage, and ignores to consider that 
> non-DFS usage may also be high on a DN/cluster. Hence, it thinks that if a 
> DFS Usage report from DN is 8% only, its got a lot of free space to write 
> more blocks, when that isn't true as shown by the case of this user. It went 
> on scheduling writes to the DN to balance it out, but the DN simply can't 
> accept any more blocks as a result of its disks' state.
> I think it would be better if we _computed_ the actual utilization based on 
> {{(100-(actual remaining space))/(capacity)}}, as opposed to the current 
> {{(dfs used)/(capacity)}}. Thoughts?
> This isn't very critical, however, cause it is very rare to see DN space 
> being used for non DN data, but it does expose a valid bug.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-3570) Balancer shouldn't rely on "DFS Space Used %" as that ignores non-DFS used space

2022-10-27 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17625004#comment-17625004
 ] 

ASF GitHub Bot commented on HDFS-3570:
--

ZanderXu commented on PR #5044:
URL: https://github.com/apache/hadoop/pull/5044#issuecomment-1293291897

   @ashutoshcipher Noted, I will review it this weekends.




> Balancer shouldn't rely on "DFS Space Used %" as that ignores non-DFS used 
> space
> 
>
> Key: HDFS-3570
> URL: https://issues.apache.org/jira/browse/HDFS-3570
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: balancer  mover
>Affects Versions: 2.0.0-alpha
>Reporter: Harsh J
>Assignee: Ashutosh Gupta
>Priority: Minor
>  Labels: pull-request-available
> Attachments: HDFS-3570.003.patch, HDFS-3570.2.patch, 
> HDFS-3570.aash.1.patch
>
>
> Report from a user here: 
> https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pIhNyDVxdVY/b7ENZmEvBjIJ,
>  post archived at http://pastebin.com/eVFkk0A0
> This user had a specific DN that had a large non-DFS usage among 
> dfs.data.dirs, and very little DFS usage (which is computed against total 
> possible capacity). 
> Balancer apparently only looks at the usage, and ignores to consider that 
> non-DFS usage may also be high on a DN/cluster. Hence, it thinks that if a 
> DFS Usage report from DN is 8% only, its got a lot of free space to write 
> more blocks, when that isn't true as shown by the case of this user. It went 
> on scheduling writes to the DN to balance it out, but the DN simply can't 
> accept any more blocks as a result of its disks' state.
> I think it would be better if we _computed_ the actual utilization based on 
> {{(100-(actual remaining space))/(capacity)}}, as opposed to the current 
> {{(dfs used)/(capacity)}}. Thoughts?
> This isn't very critical, however, cause it is very rare to see DN space 
> being used for non DN data, but it does expose a valid bug.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-3570) Balancer shouldn't rely on "DFS Space Used %" as that ignores non-DFS used space

2022-10-25 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17624087#comment-17624087
 ] 

ASF GitHub Bot commented on HDFS-3570:
--

ashutoshcipher commented on PR #5044:
URL: https://github.com/apache/hadoop/pull/5044#issuecomment-1291278089

   @aajisaka @ZanderXu,  Please help in reviewing this PR in your free time. 
Thanks.




> Balancer shouldn't rely on "DFS Space Used %" as that ignores non-DFS used 
> space
> 
>
> Key: HDFS-3570
> URL: https://issues.apache.org/jira/browse/HDFS-3570
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: balancer  mover
>Affects Versions: 2.0.0-alpha
>Reporter: Harsh J
>Assignee: Ashutosh Gupta
>Priority: Minor
>  Labels: pull-request-available
> Attachments: HDFS-3570.003.patch, HDFS-3570.2.patch, 
> HDFS-3570.aash.1.patch
>
>
> Report from a user here: 
> https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pIhNyDVxdVY/b7ENZmEvBjIJ,
>  post archived at http://pastebin.com/eVFkk0A0
> This user had a specific DN that had a large non-DFS usage among 
> dfs.data.dirs, and very little DFS usage (which is computed against total 
> possible capacity). 
> Balancer apparently only looks at the usage, and ignores to consider that 
> non-DFS usage may also be high on a DN/cluster. Hence, it thinks that if a 
> DFS Usage report from DN is 8% only, its got a lot of free space to write 
> more blocks, when that isn't true as shown by the case of this user. It went 
> on scheduling writes to the DN to balance it out, but the DN simply can't 
> accept any more blocks as a result of its disks' state.
> I think it would be better if we _computed_ the actual utilization based on 
> {{(100-(actual remaining space))/(capacity)}}, as opposed to the current 
> {{(dfs used)/(capacity)}}. Thoughts?
> This isn't very critical, however, cause it is very rare to see DN space 
> being used for non DN data, but it does expose a valid bug.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-3570) Balancer shouldn't rely on "DFS Space Used %" as that ignores non-DFS used space

2022-10-20 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17621054#comment-17621054
 ] 

ASF GitHub Bot commented on HDFS-3570:
--

ashutoshcipher commented on PR #5044:
URL: https://github.com/apache/hadoop/pull/5044#issuecomment-1285423482

   @aajisaka @ZanderXu - Please help in reviewing . Thanks.




> Balancer shouldn't rely on "DFS Space Used %" as that ignores non-DFS used 
> space
> 
>
> Key: HDFS-3570
> URL: https://issues.apache.org/jira/browse/HDFS-3570
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: balancer  mover
>Affects Versions: 2.0.0-alpha
>Reporter: Harsh J
>Assignee: Ashutosh Gupta
>Priority: Minor
>  Labels: pull-request-available
> Attachments: HDFS-3570.003.patch, HDFS-3570.2.patch, 
> HDFS-3570.aash.1.patch
>
>
> Report from a user here: 
> https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pIhNyDVxdVY/b7ENZmEvBjIJ,
>  post archived at http://pastebin.com/eVFkk0A0
> This user had a specific DN that had a large non-DFS usage among 
> dfs.data.dirs, and very little DFS usage (which is computed against total 
> possible capacity). 
> Balancer apparently only looks at the usage, and ignores to consider that 
> non-DFS usage may also be high on a DN/cluster. Hence, it thinks that if a 
> DFS Usage report from DN is 8% only, its got a lot of free space to write 
> more blocks, when that isn't true as shown by the case of this user. It went 
> on scheduling writes to the DN to balance it out, but the DN simply can't 
> accept any more blocks as a result of its disks' state.
> I think it would be better if we _computed_ the actual utilization based on 
> {{(100-(actual remaining space))/(capacity)}}, as opposed to the current 
> {{(dfs used)/(capacity)}}. Thoughts?
> This isn't very critical, however, cause it is very rare to see DN space 
> being used for non DN data, but it does expose a valid bug.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-3570) Balancer shouldn't rely on "DFS Space Used %" as that ignores non-DFS used space

2022-10-19 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17620751#comment-17620751
 ] 

ASF GitHub Bot commented on HDFS-3570:
--

hadoop-yetus commented on PR #5044:
URL: https://github.com/apache/hadoop/pull/5044#issuecomment-1284967160

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 56s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  detsecrets  |   0m  0s |  |  detect-secrets was not available.  
|
   | +0 :ok: |  markdownlint  |   0m  0s |  |  markdownlint was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | -1 :x: |  test4tests  |   0m  0s |  |  The patch doesn't appear to include 
any new or modified tests. Please justify why no new tests are needed for this 
patch. Also please list what manual steps were performed to verify this patch.  
|
    _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  42m 15s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   1m 44s |  |  trunk passed with JDK 
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  compile  |   1m 30s |  |  trunk passed with JDK 
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  checkstyle  |   1m 26s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   1m 42s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   1m 26s |  |  trunk passed with JDK 
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  javadoc  |   1m 39s |  |  trunk passed with JDK 
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |   3m 40s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  23m  2s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   1m 24s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m 23s |  |  the patch passed with JDK 
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  javac  |   1m 23s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m 19s |  |  the patch passed with JDK 
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  javac  |   1m 19s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks 
issues.  |
   | +1 :green_heart: |  checkstyle  |   0m 58s |  |  the patch passed  |
   | +1 :green_heart: |  mvnsite  |   1m 25s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   0m 56s |  |  the patch passed with JDK 
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  javadoc  |   1m 30s |  |  the patch passed with JDK 
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |   3m 16s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  22m 28s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | +1 :green_heart: |  unit  | 243m 56s |  |  hadoop-hdfs in the patch 
passed.  |
   | +1 :green_heart: |  asflicense  |   1m  7s |  |  The patch does not 
generate ASF License warnings.  |
   |  |   | 357m 41s |  |  |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5044/2/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/5044 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint 
|
   | uname | Linux 35a3ff40da0a 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 43e802df586a2e1d8e8a429d64b9163b20d927f9 |
   | Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
   | Multi-JDK versions | 
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 
/usr/lib/jvm/java-8-openjdk-amd64:Private 
Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
   |  Test Results | 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5044/2/testReport/ |
   | Max. process+thread count | 3023 (vs. ulimit of 5500) |
   | modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
   | Console output | 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5044/2/console |
   |

[jira] [Commented] (HDFS-3570) Balancer shouldn't rely on "DFS Space Used %" as that ignores non-DFS used space

2022-10-19 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17620568#comment-17620568
 ] 

ASF GitHub Bot commented on HDFS-3570:
--

hadoop-yetus commented on PR #5044:
URL: https://github.com/apache/hadoop/pull/5044#issuecomment-1284516102

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 56s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  detsecrets  |   0m  0s |  |  detect-secrets was not available.  
|
   | +0 :ok: |  markdownlint  |   0m  0s |  |  markdownlint was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | -1 :x: |  test4tests  |   0m  0s |  |  The patch doesn't appear to include 
any new or modified tests. Please justify why no new tests are needed for this 
patch. Also please list what manual steps were performed to verify this patch.  
|
    _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  42m  8s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   1m 38s |  |  trunk passed with JDK 
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  compile  |   1m 31s |  |  trunk passed with JDK 
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  checkstyle  |   1m 17s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   1m 40s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   1m 16s |  |  trunk passed with JDK 
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  javadoc  |   1m 38s |  |  trunk passed with JDK 
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |   3m 44s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  25m 59s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   1m 25s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m 28s |  |  the patch passed with JDK 
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  javac  |   1m 28s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m 19s |  |  the patch passed with JDK 
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  javac  |   1m 19s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks 
issues.  |
   | -0 :warning: |  checkstyle  |   1m  0s | 
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5044/1/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
 |  hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 21 unchanged - 
0 fixed = 22 total (was 21)  |
   | +1 :green_heart: |  mvnsite  |   1m 26s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   0m 56s |  |  the patch passed with JDK 
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  javadoc  |   1m 25s |  |  the patch passed with JDK 
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |   3m 33s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  25m 44s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | -1 :x: |  unit  | 353m 14s | 
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5044/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
 |  hadoop-hdfs in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 56s |  |  The patch does not 
generate ASF License warnings.  |
   |  |   | 471m 51s |  |  |
   
   
   | Reason | Tests |
   |---:|:--|
   | Failed junit tests | hadoop.hdfs.server.namenode.ha.TestObserverNode |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5044/1/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/5044 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint 
|
   | uname | Linux f6778e909231 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / e0e1a60554aa05ff878fc9685e6cb4b3ec01f618 |
   | Default Java | Private

[jira] [Commented] (HDFS-3570) Balancer shouldn't rely on "DFS Space Used %" as that ignores non-DFS used space

2022-10-19 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17620309#comment-17620309
 ] 

ASF GitHub Bot commented on HDFS-3570:
--

ashutoshcipher opened a new pull request, #5044:
URL: https://github.com/apache/hadoop/pull/5044

   ### Description of PR
   
   **Balancer shouldn't rely on "DFS Space Used %" as that ignores non-DFS used 
space**
   
   
   Report from a user here: 
https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pIhNyDVxdVY/b7ENZmEvBjIJ
 (Not available now) , post archived at http://pastebin.com/eVFkk0A0
   
   This user had a specific DN that had a large non-DFS usage among 
dfs.data.dirs, and very little DFS usage (which is computed against total 
possible capacity).
   
   Balancer apparently only looks at the usage, and ignores to consider that 
non-DFS usage may also be high on a DN/cluster. Hence, it thinks that if a DFS 
Usage report from DN is 8% only, its got a lot of free space to write more 
blocks, when that isn't true as shown by the case of this user. It went on 
scheduling writes to the DN to balance it out, but the DN simply can't accept 
any more blocks as a result of its disks' state.
   
   It would be better if we computed the actual utilization based on 
(100-(actual remaining space))/(capacity), as opposed to the current (dfs 
used)/(capacity). Thoughts?
   
   This isn't very critical, however, cause it is very rare to see DN space 
being used for non DN data, but it does expose a valid bug.
   
   
   
   ### How was this patch tested?
   
   UT
   
   
   ### For code changes:
   
   - [X] Does the title or this PR starts with the corresponding JIRA issue id 
(e.g. 'HADOOP-17799. Your PR title ...')?
   - [ ] Object storage: have the integration tests been executed and the 
endpoint declared according to the connector-specific documentation?
   - [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, 
`NOTICE-binary` files?
   
   




> Balancer shouldn't rely on "DFS Space Used %" as that ignores non-DFS used 
> space
> 
>
> Key: HDFS-3570
> URL: https://issues.apache.org/jira/browse/HDFS-3570
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: balancer  mover
>Affects Versions: 2.0.0-alpha
>Reporter: Harsh J
>Assignee: Ashutosh Gupta
>Priority: Minor
> Attachments: HDFS-3570.003.patch, HDFS-3570.2.patch, 
> HDFS-3570.aash.1.patch
>
>
> Report from a user here: 
> https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pIhNyDVxdVY/b7ENZmEvBjIJ,
>  post archived at http://pastebin.com/eVFkk0A0
> This user had a specific DN that had a large non-DFS usage among 
> dfs.data.dirs, and very little DFS usage (which is computed against total 
> possible capacity). 
> Balancer apparently only looks at the usage, and ignores to consider that 
> non-DFS usage may also be high on a DN/cluster. Hence, it thinks that if a 
> DFS Usage report from DN is 8% only, its got a lot of free space to write 
> more blocks, when that isn't true as shown by the case of this user. It went 
> on scheduling writes to the DN to balance it out, but the DN simply can't 
> accept any more blocks as a result of its disks' state.
> I think it would be better if we _computed_ the actual utilization based on 
> {{(100-(actual remaining space))/(capacity)}}, as opposed to the current 
> {{(dfs used)/(capacity)}}. Thoughts?
> This isn't very critical, however, cause it is very rare to see DN space 
> being used for non DN data, but it does expose a valid bug.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-3570) Balancer shouldn't rely on "DFS Space Used %" as that ignores non-DFS used space

2022-10-19 Thread Ashutosh Gupta (Jira)



[ 
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17620302#comment-17620302
 ] 

Ashutosh Gupta commented on HDFS-3570:
--

I have gone through the discussion. Taking it for fix.

> Balancer shouldn't rely on "DFS Space Used %" as that ignores non-DFS used 
> space
> 
>
> Key: HDFS-3570
> URL: https://issues.apache.org/jira/browse/HDFS-3570
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: balancer  mover
>Affects Versions: 2.0.0-alpha
>Reporter: Harsh J
>Assignee: Ashutosh Gupta
>Priority: Minor
> Attachments: HDFS-3570.003.patch, HDFS-3570.2.patch, 
> HDFS-3570.aash.1.patch
>
>
> Report from a user here: 
> https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pIhNyDVxdVY/b7ENZmEvBjIJ,
>  post archived at http://pastebin.com/eVFkk0A0
> This user had a specific DN that had a large non-DFS usage among 
> dfs.data.dirs, and very little DFS usage (which is computed against total 
> possible capacity). 
> Balancer apparently only looks at the usage, and ignores to consider that 
> non-DFS usage may also be high on a DN/cluster. Hence, it thinks that if a 
> DFS Usage report from DN is 8% only, its got a lot of free space to write 
> more blocks, when that isn't true as shown by the case of this user. It went 
> on scheduling writes to the DN to balance it out, but the DN simply can't 
> accept any more blocks as a result of its disks' state.
> I think it would be better if we _computed_ the actual utilization based on 
> {{(100-(actual remaining space))/(capacity)}}, as opposed to the current 
> {{(dfs used)/(capacity)}}. Thoughts?
> This isn't very critical, however, cause it is very rare to see DN space 
> being used for non DN data, but it does expose a valid bug.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-3570) Balancer shouldn't rely on "DFS Space Used %" as that ignores non-DFS used space

2020-04-11 Thread Hadoop QA (Jira)



[ 
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17081540#comment-17081540
 ] 

Hadoop QA commented on HDFS-3570:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
48s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m 
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
16m 40s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
42s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
58s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 38s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 1 new + 42 unchanged - 0 fixed = 43 total (was 42) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
14m 22s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
39s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 32m  5s{color} 
| {color:red} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
26s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 98m 50s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.tools.TestStoragePolicyCommands |
|   | hadoop.hdfs.TestByteBufferPread |
|   | hadoop.hdfs.tools.TestECAdmin |
|   | hadoop.hdfs.tools.TestViewFSStoragePolicyCommands |
|   | hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer |
|   | hadoop.hdfs.TestFileAppend4 |
|   | hadoop.hdfs.TestErasureCodingExerciseAPIs |
|   | hadoop.hdfs.tools.TestDFSAdmin |
|   | 
hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerWithStripedBlocks |
|   | hadoop.cli.TestHDFSCLI |
|   | hadoop.hdfs.TestDFSStripedOutputStream |
|   | hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer |
|   | hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForAcl |
|   | hadoop.hdfs.tools.TestDebugAdmin |
|   | hadoop.hdfs.tools.TestDFSAdminWithHA |
|   | hadoop.hdfs.TestErasureCodingPolicies |
|   | hadoop.hdfs.TestBlockStoragePolicy |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.8 Server=19.03.8 Image:yetus/hadoop:e6455cc864d |
| JIRA Issue | HDFS-3570 |
| JIRA Patch URL |

[jira] [Commented] (HDFS-3570) Balancer shouldn't rely on "DFS Space Used %" as that ignores non-DFS used space

2018-11-23 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16697262#comment-16697262
 ] 

Hadoop QA commented on HDFS-3570:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 
 5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 24s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
43s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
49s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 40s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 1 new + 41 unchanged - 0 fixed = 42 total (was 41) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 15s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
40s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 74m 37s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
26s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}129m  8s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport |
|   | hadoop.hdfs.server.namenode.sps.TestBlockStorageMovementAttemptedItems |
|   | hadoop.hdfs.web.TestWebHdfsTimeouts |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-3570 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12746476/HDFS-3570.003.patch |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  |
| uname | Linux b130a0ef0b49 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 753f421 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HDFS-Build/25616/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| unit |

[jira] [Commented] (HDFS-3570) Balancer shouldn't rely on "DFS Space Used %" as that ignores non-DFS used space

2017-10-11 Thread Subru Krishnan (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201147#comment-16201147
 ] 

Subru Krishnan commented on HDFS-3570:
--

Pushing it out from 2.9.0 due to lack of recent activity. Feel free to revert 
if required.

> Balancer shouldn't rely on "DFS Space Used %" as that ignores non-DFS used 
> space
> 
>
> Key: HDFS-3570
> URL: https://issues.apache.org/jira/browse/HDFS-3570
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: balancer & mover
>Affects Versions: 2.0.0-alpha
>Reporter: Harsh J
>Assignee: Akira Ajisaka
>Priority: Minor
> Attachments: HDFS-3570.003.patch, HDFS-3570.2.patch, 
> HDFS-3570.aash.1.patch
>
>
> Report from a user here: 
> https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pIhNyDVxdVY/b7ENZmEvBjIJ,
>  post archived at http://pastebin.com/eVFkk0A0
> This user had a specific DN that had a large non-DFS usage among 
> dfs.data.dirs, and very little DFS usage (which is computed against total 
> possible capacity). 
> Balancer apparently only looks at the usage, and ignores to consider that 
> non-DFS usage may also be high on a DN/cluster. Hence, it thinks that if a 
> DFS Usage report from DN is 8% only, its got a lot of free space to write 
> more blocks, when that isn't true as shown by the case of this user. It went 
> on scheduling writes to the DN to balance it out, but the DN simply can't 
> accept any more blocks as a result of its disks' state.
> I think it would be better if we _computed_ the actual utilization based on 
> {{(100-(actual remaining space))/(capacity)}}, as opposed to the current 
> {{(dfs used)/(capacity)}}. Thoughts?
> This isn't very critical, however, cause it is very rare to see DN space 
> being used for non DN data, but it does expose a valid bug.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-3570) Balancer shouldn't rely on "DFS Space Used %" as that ignores non-DFS used space

2017-09-29 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16186755#comment-16186755
 ] 

Hadoop QA commented on HDFS-3570:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
25s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 29s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
47s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
2s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 41s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 1 new + 45 unchanged - 0 fixed = 46 total (was 45) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 18s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}122m 21s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
30s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}174m  0s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestReconstructStripedFile |
|   | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
|   | hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-3570 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12746476/HDFS-3570.003.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 56607df6288a 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 373d0a5 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HDFS-Build/21449/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| unit |

[jira] [Commented] (HDFS-3570) Balancer shouldn't rely on "DFS Space Used %" as that ignores non-DFS used space

2017-01-06 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15806198#comment-15806198
 ] 

Hadoop QA commented on HDFS-3570:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
53s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 30s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 1 new + 45 unchanged - 0 fixed = 46 total (was 45) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
38s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 85m 51s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
34s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}112m 40s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
|   | hadoop.hdfs.TestEncryptionZones |
| Timed out junit tests | 
org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-3570 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12746476/HDFS-3570.003.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux b9d4a9e6071d 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 2977bc6 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HDFS-Build/18060/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/18060/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/18060/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/18060/console |
| Powered by | Apache Yetus

[jira] [Commented] (HDFS-3570) Balancer shouldn't rely on DFS Space Used % as that ignores non-DFS used space

2015-07-28 Thread Allen Wittenauer (JIRA)

[
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14644720#comment-14644720
]

Allen Wittenauer commented on HDFS-3570:

bq. Setting the parameter for non-dfs used space is an ideal way to avoid the
problem

Not really. The negative math model just flat out doesn't work in practice.
It makes assumptions that whatever else is on the file system has a way to
contain how much space is used which is pretty much impossible. It's one
of the reasons why I've been advocated a dedicated partition per disk for HDFS
for years now. Those that do seem to have a lot less problems with HDFS at the
cost of some initial setup pain.

Balancer shouldn't rely on DFS Space Used % as that ignores non-DFS used
space

Key: HDFS-3570
URL: https://issues.apache.org/jira/browse/HDFS-3570
Project: Hadoop HDFS
Issue Type: Bug
Components: balancer mover
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Assignee: Akira AJISAKA
Priority: Minor
Attachments: HDFS-3570.003.patch, HDFS-3570.2.patch,
HDFS-3570.aash.1.patch

Report from a user here:
https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pIhNyDVxdVY/b7ENZmEvBjIJ,
post archived at http://pastebin.com/eVFkk0A0
This user had a specific DN that had a large non-DFS usage among
dfs.data.dirs, and very little DFS usage (which is computed against total
possible capacity).
Balancer apparently only looks at the usage, and ignores to consider that
non-DFS usage may also be high on a DN/cluster. Hence, it thinks that if a
DFS Usage report from DN is 8% only, its got a lot of free space to write
more blocks, when that isn't true as shown by the case of this user. It went
on scheduling writes to the DN to balance it out, but the DN simply can't
accept any more blocks as a result of its disks' state.
I think it would be better if we _computed_ the actual utilization based on
{{(100-(actual remaining space))/(capacity)}}, as opposed to the current
{{(dfs used)/(capacity)}}. Thoughts?
This isn't very critical, however, cause it is very rare to see DN space
being used for non DN data, but it does expose a valid bug.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-3570) Balancer shouldn't rely on DFS Space Used % as that ignores non-DFS used space

2015-07-28 Thread Allen Wittenauer (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14644870#comment-14644870
 ] 

Allen Wittenauer commented on HDFS-3570:


bq. We could get space used by calling df rather than du

... which, as a reminder, would return incorrect numbers on a lot of pooled 
storage systems (ZFS, btrfs, etc, etc).

 Balancer shouldn't rely on DFS Space Used % as that ignores non-DFS used 
 space
 

 Key: HDFS-3570
 URL: https://issues.apache.org/jira/browse/HDFS-3570
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: balancer  mover
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Assignee: Akira AJISAKA
Priority: Minor
 Attachments: HDFS-3570.003.patch, HDFS-3570.2.patch, 
 HDFS-3570.aash.1.patch


 Report from a user here: 
 https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pIhNyDVxdVY/b7ENZmEvBjIJ,
  post archived at http://pastebin.com/eVFkk0A0
 This user had a specific DN that had a large non-DFS usage among 
 dfs.data.dirs, and very little DFS usage (which is computed against total 
 possible capacity). 
 Balancer apparently only looks at the usage, and ignores to consider that 
 non-DFS usage may also be high on a DN/cluster. Hence, it thinks that if a 
 DFS Usage report from DN is 8% only, its got a lot of free space to write 
 more blocks, when that isn't true as shown by the case of this user. It went 
 on scheduling writes to the DN to balance it out, but the DN simply can't 
 accept any more blocks as a result of its disks' state.
 I think it would be better if we _computed_ the actual utilization based on 
 {{(100-(actual remaining space))/(capacity)}}, as opposed to the current 
 {{(dfs used)/(capacity)}}. Thoughts?
 This isn't very critical, however, cause it is very rare to see DN space 
 being used for non DN data, but it does expose a valid bug.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-3570) Balancer shouldn't rely on DFS Space Used % as that ignores non-DFS used space

2015-07-28 Thread Colin Patrick McCabe (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14644814#comment-14644814
 ] 

Colin Patrick McCabe commented on HDFS-3570:


I agree that it would be nice to have an optimized code path assuming a 
dedicated partition for HDFS.  We could get space used by calling df rather 
than du, which would be much more efficient.  However, in the past, we've 
avoided doing this because MR almost always spills to the same disks that HDFS 
is using, so we would have to have 2 partitions on every disk.  I'm not sure if 
there is a good way around this problem...

 Balancer shouldn't rely on DFS Space Used % as that ignores non-DFS used 
 space
 

 Key: HDFS-3570
 URL: https://issues.apache.org/jira/browse/HDFS-3570
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: balancer  mover
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Assignee: Akira AJISAKA
Priority: Minor
 Attachments: HDFS-3570.003.patch, HDFS-3570.2.patch, 
 HDFS-3570.aash.1.patch


 Report from a user here: 
 https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pIhNyDVxdVY/b7ENZmEvBjIJ,
  post archived at http://pastebin.com/eVFkk0A0
 This user had a specific DN that had a large non-DFS usage among 
 dfs.data.dirs, and very little DFS usage (which is computed against total 
 possible capacity). 
 Balancer apparently only looks at the usage, and ignores to consider that 
 non-DFS usage may also be high on a DN/cluster. Hence, it thinks that if a 
 DFS Usage report from DN is 8% only, its got a lot of free space to write 
 more blocks, when that isn't true as shown by the case of this user. It went 
 on scheduling writes to the DN to balance it out, but the DN simply can't 
 accept any more blocks as a result of its disks' state.
 I think it would be better if we _computed_ the actual utilization based on 
 {{(100-(actual remaining space))/(capacity)}}, as opposed to the current 
 {{(dfs used)/(capacity)}}. Thoughts?
 This isn't very critical, however, cause it is very rare to see DN space 
 being used for non DN data, but it does expose a valid bug.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-3570) Balancer shouldn't rely on DFS Space Used % as that ignores non-DFS used space

2015-07-28 Thread Tsz Wo Nicholas Sze (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14644935#comment-14644935
 ] 

Tsz Wo Nicholas Sze commented on HDFS-3570:
---

 ... It went on scheduling writes to the DN to balance it out, but the DN 
 simply can't accept any more blocks as a result of its disks' state.

This is similar to HDFS-8278.  I suggest that Balancer also checks if the 
remaining space is larger then a threshold before adding the datanode to 
underUtilized or belowAvgUtilized.

 Balancer shouldn't rely on DFS Space Used % as that ignores non-DFS used 
 space
 

 Key: HDFS-3570
 URL: https://issues.apache.org/jira/browse/HDFS-3570
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: balancer  mover
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Assignee: Akira AJISAKA
Priority: Minor
 Attachments: HDFS-3570.003.patch, HDFS-3570.2.patch, 
 HDFS-3570.aash.1.patch


 Report from a user here: 
 https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pIhNyDVxdVY/b7ENZmEvBjIJ,
  post archived at http://pastebin.com/eVFkk0A0
 This user had a specific DN that had a large non-DFS usage among 
 dfs.data.dirs, and very little DFS usage (which is computed against total 
 possible capacity). 
 Balancer apparently only looks at the usage, and ignores to consider that 
 non-DFS usage may also be high on a DN/cluster. Hence, it thinks that if a 
 DFS Usage report from DN is 8% only, its got a lot of free space to write 
 more blocks, when that isn't true as shown by the case of this user. It went 
 on scheduling writes to the DN to balance it out, but the DN simply can't 
 accept any more blocks as a result of its disks' state.
 I think it would be better if we _computed_ the actual utilization based on 
 {{(100-(actual remaining space))/(capacity)}}, as opposed to the current 
 {{(dfs used)/(capacity)}}. Thoughts?
 This isn't very critical, however, cause it is very rare to see DN space 
 being used for non DN data, but it does expose a valid bug.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-3570) Balancer shouldn't rely on DFS Space Used % as that ignores non-DFS used space

2015-07-27 Thread Tsz Wo Nicholas Sze (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14643695#comment-14643695
 ] 

Tsz Wo Nicholas Sze commented on HDFS-3570:
---

Have you set dfs.datanode.du.reserved for the non-dfs used space?

 Balancer shouldn't rely on DFS Space Used % as that ignores non-DFS used 
 space
 

 Key: HDFS-3570
 URL: https://issues.apache.org/jira/browse/HDFS-3570
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: balancer  mover
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Assignee: Akira AJISAKA
Priority: Minor
 Attachments: HDFS-3570.003.patch, HDFS-3570.2.patch, 
 HDFS-3570.aash.1.patch


 Report from a user here: 
 https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pIhNyDVxdVY/b7ENZmEvBjIJ,
  post archived at http://pastebin.com/eVFkk0A0
 This user had a specific DN that had a large non-DFS usage among 
 dfs.data.dirs, and very little DFS usage (which is computed against total 
 possible capacity). 
 Balancer apparently only looks at the usage, and ignores to consider that 
 non-DFS usage may also be high on a DN/cluster. Hence, it thinks that if a 
 DFS Usage report from DN is 8% only, its got a lot of free space to write 
 more blocks, when that isn't true as shown by the case of this user. It went 
 on scheduling writes to the DN to balance it out, but the DN simply can't 
 accept any more blocks as a result of its disks' state.
 I think it would be better if we _computed_ the actual utilization based on 
 {{(100-(actual remaining space))/(capacity)}}, as opposed to the current 
 {{(dfs used)/(capacity)}}. Thoughts?
 This isn't very critical, however, cause it is very rare to see DN space 
 being used for non DN data, but it does expose a valid bug.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-3570) Balancer shouldn't rely on DFS Space Used % as that ignores non-DFS used space

2015-07-27 Thread Akira AJISAKA (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14643825#comment-14643825
 ] 

Akira AJISAKA commented on HDFS-3570:
-

bq. Have you set dfs.datanode.du.reserved for the non-dfs used space?
I don't set the parameter. Setting the parameter for non-dfs used space is an 
ideal way to avoid the problem, however, I'd like to deal with such a situation 
that someone unintentionally puts big files to a DataNode and then another one 
runs balancer.

 Balancer shouldn't rely on DFS Space Used % as that ignores non-DFS used 
 space
 

 Key: HDFS-3570
 URL: https://issues.apache.org/jira/browse/HDFS-3570
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: balancer  mover
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Assignee: Akira AJISAKA
Priority: Minor
 Attachments: HDFS-3570.003.patch, HDFS-3570.2.patch, 
 HDFS-3570.aash.1.patch


 Report from a user here: 
 https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pIhNyDVxdVY/b7ENZmEvBjIJ,
  post archived at http://pastebin.com/eVFkk0A0
 This user had a specific DN that had a large non-DFS usage among 
 dfs.data.dirs, and very little DFS usage (which is computed against total 
 possible capacity). 
 Balancer apparently only looks at the usage, and ignores to consider that 
 non-DFS usage may also be high on a DN/cluster. Hence, it thinks that if a 
 DFS Usage report from DN is 8% only, its got a lot of free space to write 
 more blocks, when that isn't true as shown by the case of this user. It went 
 on scheduling writes to the DN to balance it out, but the DN simply can't 
 accept any more blocks as a result of its disks' state.
 I think it would be better if we _computed_ the actual utilization based on 
 {{(100-(actual remaining space))/(capacity)}}, as opposed to the current 
 {{(dfs used)/(capacity)}}. Thoughts?
 This isn't very critical, however, cause it is very rare to see DN space 
 being used for non DN data, but it does expose a valid bug.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-3570) Balancer shouldn't rely on DFS Space Used % as that ignores non-DFS used space

2015-07-22 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14636597#comment-14636597
 ] 

Hadoop QA commented on HDFS-3570:
-

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch |  18m 25s | Findbugs (version ) appears to 
be broken on trunk. |
| {color:green}+1{color} | @author |   0m  0s | The patch does not contain any 
@author tags. |
| {color:red}-1{color} | tests included |   0m  0s | The patch doesn't appear 
to include any new or modified tests.  Please justify why no new tests are 
needed for this patch. Also please list what manual steps were performed to 
verify this patch. |
| {color:green}+1{color} | javac |   7m 39s | There were no new javac warning 
messages. |
| {color:green}+1{color} | javadoc |   9m 40s | There were no new javadoc 
warning messages. |
| {color:green}+1{color} | release audit |   0m 22s | The applied patch does 
not increase the total number of release audit warnings. |
| {color:green}+1{color} | site |   3m  1s | Site still builds. |
| {color:green}+1{color} | checkstyle |   0m 51s | There were no new checkstyle 
issues. |
| {color:green}+1{color} | whitespace |   0m  0s | The patch has no lines that 
end in whitespace. |
| {color:green}+1{color} | install |   1m 23s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse |   0m 32s | The patch built with 
eclipse:eclipse. |
| {color:green}+1{color} | findbugs |   2m 32s | The patch does not introduce 
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native |   3m  2s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 161m 27s | Tests failed in hadoop-hdfs. |
| | | 208m 59s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.TestDistributedFileSystem |
|   | hadoop.hdfs.server.namenode.ha.TestStandbyIsHot |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | 
http://issues.apache.org/jira/secure/attachment/12746476/HDFS-3570.003.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle site |
| git revision | trunk / 94c6a4a |
| hadoop-hdfs test log | 
https://builds.apache.org/job/PreCommit-HDFS-Build/11782/artifact/patchprocess/testrun_hadoop-hdfs.txt
 |
| Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/11782/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP 
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/11782/console |


This message was automatically generated.

 Balancer shouldn't rely on DFS Space Used % as that ignores non-DFS used 
 space
 

 Key: HDFS-3570
 URL: https://issues.apache.org/jira/browse/HDFS-3570
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: balancer  mover
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Assignee: Akira AJISAKA
Priority: Minor
 Attachments: HDFS-3570.003.patch, HDFS-3570.2.patch, 
 HDFS-3570.aash.1.patch


 Report from a user here: 
 https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pIhNyDVxdVY/b7ENZmEvBjIJ,
  post archived at http://pastebin.com/eVFkk0A0
 This user had a specific DN that had a large non-DFS usage among 
 dfs.data.dirs, and very little DFS usage (which is computed against total 
 possible capacity). 
 Balancer apparently only looks at the usage, and ignores to consider that 
 non-DFS usage may also be high on a DN/cluster. Hence, it thinks that if a 
 DFS Usage report from DN is 8% only, its got a lot of free space to write 
 more blocks, when that isn't true as shown by the case of this user. It went 
 on scheduling writes to the DN to balance it out, but the DN simply can't 
 accept any more blocks as a result of its disks' state.
 I think it would be better if we _computed_ the actual utilization based on 
 {{(100-(actual remaining space))/(capacity)}}, as opposed to the current 
 {{(dfs used)/(capacity)}}. Thoughts?
 This isn't very critical, however, cause it is very rare to see DN space 
 being used for non DN data, but it does expose a valid bug.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-3570) Balancer shouldn't rely on DFS Space Used % as that ignores non-DFS used space

2015-07-21 Thread Akira AJISAKA (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14636317#comment-14636317
 ] 

Akira AJISAKA commented on HDFS-3570:
-

Thanks [~anu] for the comment.

bq. I see we are still referring to capacity in the old sense.
Agree.

bq. Would that mean now we have two different views of what is the available 
capacity of a DN; That is, one that is viewed from the balancer vs. something 
like {{DFSClient.getDatanodeReport}}.
Balancer uses {{DFSClient.getDatanodeStorageReport}}, so these views look the 
same to me. We need to take into account non-DFS usage.

 Balancer shouldn't rely on DFS Space Used % as that ignores non-DFS used 
 space
 

 Key: HDFS-3570
 URL: https://issues.apache.org/jira/browse/HDFS-3570
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: balancer  mover
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Assignee: Akira AJISAKA
Priority: Minor
 Attachments: HDFS-3570.2.patch, HDFS-3570.aash.1.patch


 Report from a user here: 
 https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pIhNyDVxdVY/b7ENZmEvBjIJ,
  post archived at http://pastebin.com/eVFkk0A0
 This user had a specific DN that had a large non-DFS usage among 
 dfs.data.dirs, and very little DFS usage (which is computed against total 
 possible capacity). 
 Balancer apparently only looks at the usage, and ignores to consider that 
 non-DFS usage may also be high on a DN/cluster. Hence, it thinks that if a 
 DFS Usage report from DN is 8% only, its got a lot of free space to write 
 more blocks, when that isn't true as shown by the case of this user. It went 
 on scheduling writes to the DN to balance it out, but the DN simply can't 
 accept any more blocks as a result of its disks' state.
 I think it would be better if we _computed_ the actual utilization based on 
 {{(100-(actual remaining space))/(capacity)}}, as opposed to the current 
 {{(dfs used)/(capacity)}}. Thoughts?
 This isn't very critical, however, cause it is very rare to see DN space 
 being used for non DN data, but it does expose a valid bug.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-3570) Balancer shouldn't rely on DFS Space Used % as that ignores non-DFS used space

2015-07-21 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14635450#comment-14635450
 ] 

Hadoop QA commented on HDFS-3570:
-

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch |   0m  0s | The patch command could not apply 
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | 
http://issues.apache.org/jira/secure/attachment/12627311/HDFS-3570.2.patch |
| Optional Tests | site javadoc javac unit findbugs checkstyle |
| git revision | trunk / 3b7ffc4 |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/11771/console |


This message was automatically generated.

 Balancer shouldn't rely on DFS Space Used % as that ignores non-DFS used 
 space
 

 Key: HDFS-3570
 URL: https://issues.apache.org/jira/browse/HDFS-3570
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: balancer  mover
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Assignee: Akira AJISAKA
Priority: Minor
 Attachments: HDFS-3570.2.patch, HDFS-3570.aash.1.patch


 Report from a user here: 
 https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pIhNyDVxdVY/b7ENZmEvBjIJ,
  post archived at http://pastebin.com/eVFkk0A0
 This user had a specific DN that had a large non-DFS usage among 
 dfs.data.dirs, and very little DFS usage (which is computed against total 
 possible capacity). 
 Balancer apparently only looks at the usage, and ignores to consider that 
 non-DFS usage may also be high on a DN/cluster. Hence, it thinks that if a 
 DFS Usage report from DN is 8% only, its got a lot of free space to write 
 more blocks, when that isn't true as shown by the case of this user. It went 
 on scheduling writes to the DN to balance it out, but the DN simply can't 
 accept any more blocks as a result of its disks' state.
 I think it would be better if we _computed_ the actual utilization based on 
 {{(100-(actual remaining space))/(capacity)}}, as opposed to the current 
 {{(dfs used)/(capacity)}}. Thoughts?
 This isn't very critical, however, cause it is very rare to see DN space 
 being used for non DN data, but it does expose a valid bug.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-3570) Balancer shouldn't rely on DFS Space Used % as that ignores non-DFS used space

2015-07-21 Thread Anu Engineer (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14635788#comment-14635788
 ] 

Anu Engineer commented on HDFS-3570:


Hi [~ajisakaa] Thanks for the patch. I think it is a good change and defines 
the notion of remaining space is a better way. However I had a minor question. 
In {{DatanodeInfo.getBlockPoolUsedPercent}} and 
{{Datanodeinfo.getRemainingPercent}} I see we are still referring to 
{{capacity}} in the old sense. Would that mean now we have two different views 
of what is the available capacity of a DN; That is, one that is viewed from the 
balancer vs. something like {{DFSClient.getDatanodeReport}}.


 Balancer shouldn't rely on DFS Space Used % as that ignores non-DFS used 
 space
 

 Key: HDFS-3570
 URL: https://issues.apache.org/jira/browse/HDFS-3570
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: balancer  mover
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Assignee: Akira AJISAKA
Priority: Minor
 Attachments: HDFS-3570.2.patch, HDFS-3570.aash.1.patch


 Report from a user here: 
 https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pIhNyDVxdVY/b7ENZmEvBjIJ,
  post archived at http://pastebin.com/eVFkk0A0
 This user had a specific DN that had a large non-DFS usage among 
 dfs.data.dirs, and very little DFS usage (which is computed against total 
 possible capacity). 
 Balancer apparently only looks at the usage, and ignores to consider that 
 non-DFS usage may also be high on a DN/cluster. Hence, it thinks that if a 
 DFS Usage report from DN is 8% only, its got a lot of free space to write 
 more blocks, when that isn't true as shown by the case of this user. It went 
 on scheduling writes to the DN to balance it out, but the DN simply can't 
 accept any more blocks as a result of its disks' state.
 I think it would be better if we _computed_ the actual utilization based on 
 {{(100-(actual remaining space))/(capacity)}}, as opposed to the current 
 {{(dfs used)/(capacity)}}. Thoughts?
 This isn't very critical, however, cause it is very rare to see DN space 
 being used for non DN data, but it does expose a valid bug.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-3570) Balancer shouldn't rely on DFS Space Used % as that ignores non-DFS used space

2015-05-01 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14524760#comment-14524760
 ] 

Hadoop QA commented on HDFS-3570:
-

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch |   0m  0s | The patch command could not apply 
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | 
http://issues.apache.org/jira/secure/attachment/12627311/HDFS-3570.2.patch |
| Optional Tests | site javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/10591/console |


This message was automatically generated.

 Balancer shouldn't rely on DFS Space Used % as that ignores non-DFS used 
 space
 

 Key: HDFS-3570
 URL: https://issues.apache.org/jira/browse/HDFS-3570
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: balancer  mover
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Assignee: Akira AJISAKA
Priority: Minor
 Attachments: HDFS-3570.2.patch, HDFS-3570.aash.1.patch


 Report from a user here: 
 https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pIhNyDVxdVY/b7ENZmEvBjIJ,
  post archived at http://pastebin.com/eVFkk0A0
 This user had a specific DN that had a large non-DFS usage among 
 dfs.data.dirs, and very little DFS usage (which is computed against total 
 possible capacity). 
 Balancer apparently only looks at the usage, and ignores to consider that 
 non-DFS usage may also be high on a DN/cluster. Hence, it thinks that if a 
 DFS Usage report from DN is 8% only, its got a lot of free space to write 
 more blocks, when that isn't true as shown by the case of this user. It went 
 on scheduling writes to the DN to balance it out, but the DN simply can't 
 accept any more blocks as a result of its disks' state.
 I think it would be better if we _computed_ the actual utilization based on 
 {{(100-(actual remaining space))/(capacity)}}, as opposed to the current 
 {{(dfs used)/(capacity)}}. Thoughts?
 This isn't very critical, however, cause it is very rare to see DN space 
 being used for non DN data, but it does expose a valid bug.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-3570) Balancer shouldn't rely on DFS Space Used % as that ignores non-DFS used space

2015-05-01 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14524827#comment-14524827
 ] 

Hadoop QA commented on HDFS-3570:
-

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch |   0m  0s | The patch command could not apply 
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | 
http://issues.apache.org/jira/secure/attachment/12627311/HDFS-3570.2.patch |
| Optional Tests | site javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/10620/console |


This message was automatically generated.

 Balancer shouldn't rely on DFS Space Used % as that ignores non-DFS used 
 space
 

 Key: HDFS-3570
 URL: https://issues.apache.org/jira/browse/HDFS-3570
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: balancer  mover
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Assignee: Akira AJISAKA
Priority: Minor
 Attachments: HDFS-3570.2.patch, HDFS-3570.aash.1.patch


 Report from a user here: 
 https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pIhNyDVxdVY/b7ENZmEvBjIJ,
  post archived at http://pastebin.com/eVFkk0A0
 This user had a specific DN that had a large non-DFS usage among 
 dfs.data.dirs, and very little DFS usage (which is computed against total 
 possible capacity). 
 Balancer apparently only looks at the usage, and ignores to consider that 
 non-DFS usage may also be high on a DN/cluster. Hence, it thinks that if a 
 DFS Usage report from DN is 8% only, its got a lot of free space to write 
 more blocks, when that isn't true as shown by the case of this user. It went 
 on scheduling writes to the DN to balance it out, but the DN simply can't 
 accept any more blocks as a result of its disks' state.
 I think it would be better if we _computed_ the actual utilization based on 
 {{(100-(actual remaining space))/(capacity)}}, as opposed to the current 
 {{(dfs used)/(capacity)}}. Thoughts?
 This isn't very critical, however, cause it is very rare to see DN space 
 being used for non DN data, but it does expose a valid bug.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-3570) Balancer shouldn't rely on DFS Space Used % as that ignores non-DFS used space

2014-02-18 Thread Akira AJISAKA (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13904914#comment-13904914
 ] 

Akira AJISAKA commented on HDFS-3570:
-

Thank you for verifying, [~ash211]!
[~qwertymaniac], would you please review the patch?

 Balancer shouldn't rely on DFS Space Used % as that ignores non-DFS used 
 space
 

 Key: HDFS-3570
 URL: https://issues.apache.org/jira/browse/HDFS-3570
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: balancer
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Assignee: Akira AJISAKA
Priority: Minor
 Attachments: HDFS-3570.2.patch, HDFS-3570.aash.1.patch


 Report from a user here: 
 https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pIhNyDVxdVY/b7ENZmEvBjIJ,
  post archived at http://pastebin.com/eVFkk0A0
 This user had a specific DN that had a large non-DFS usage among 
 dfs.data.dirs, and very little DFS usage (which is computed against total 
 possible capacity). 
 Balancer apparently only looks at the usage, and ignores to consider that 
 non-DFS usage may also be high on a DN/cluster. Hence, it thinks that if a 
 DFS Usage report from DN is 8% only, its got a lot of free space to write 
 more blocks, when that isn't true as shown by the case of this user. It went 
 on scheduling writes to the DN to balance it out, but the DN simply can't 
 accept any more blocks as a result of its disks' state.
 I think it would be better if we _computed_ the actual utilization based on 
 {{(100-(actual remaining space))/(capacity)}}, as opposed to the current 
 {{(dfs used)/(capacity)}}. Thoughts?
 This isn't very critical, however, cause it is very rare to see DN space 
 being used for non DN data, but it does expose a valid bug.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HDFS-3570) Balancer shouldn't rely on DFS Space Used % as that ignores non-DFS used space

2014-02-07 Thread Andrew Ash (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13895208#comment-13895208
 ] 

Andrew Ash commented on HDFS-3570:
--

Confirmed that this did what I thought it would, and non-DFS used space is 
being taken into account.  Here are my before and after stats when running with 
the default threshold (10%).  The delta between overloaded and underloaded 
isn't exactly at 10% since there's been more activity since the balancer 
finished, but I'm good to go on this.

IP  CapacityUsedNon DFS usedUsed %  Actual Use %
.33 3.220.511.3915.84%  27.87%
.35 3.221.870.2058.07%  61.92%
.37 3.221.790.3655.59%  62.59%
.39 3.221.590.3349.38%  55.02%
.41 3.220.181.915.59%   13.74%  

IP  CapacityUsedNon DFS usedUsed %  Actual Use %
.33 3.220.751.3223.29%  39.47%
.35 3.221.640.1750.93%  53.77%
.37 3.221.550.3348.14%  53.63%
.39 3.221.470.3145.65%  50.52%
.41 3.220.521.9016.15%  39.39%


Ready for merging!

 Balancer shouldn't rely on DFS Space Used % as that ignores non-DFS used 
 space
 

 Key: HDFS-3570
 URL: https://issues.apache.org/jira/browse/HDFS-3570
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: balancer
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Assignee: Akira AJISAKA
Priority: Minor
 Attachments: HDFS-3570.2.patch, HDFS-3570.aash.1.patch


 Report from a user here: 
 https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pIhNyDVxdVY/b7ENZmEvBjIJ,
  post archived at http://pastebin.com/eVFkk0A0
 This user had a specific DN that had a large non-DFS usage among 
 dfs.data.dirs, and very little DFS usage (which is computed against total 
 possible capacity). 
 Balancer apparently only looks at the usage, and ignores to consider that 
 non-DFS usage may also be high on a DN/cluster. Hence, it thinks that if a 
 DFS Usage report from DN is 8% only, its got a lot of free space to write 
 more blocks, when that isn't true as shown by the case of this user. It went 
 on scheduling writes to the DN to balance it out, but the DN simply can't 
 accept any more blocks as a result of its disks' state.
 I think it would be better if we _computed_ the actual utilization based on 
 {{(100-(actual remaining space))/(capacity)}}, as opposed to the current 
 {{(dfs used)/(capacity)}}. Thoughts?
 This isn't very critical, however, cause it is very rare to see DN space 
 being used for non DN data, but it does expose a valid bug.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HDFS-3570) Balancer shouldn't rely on DFS Space Used % as that ignores non-DFS used space

2014-02-06 Thread Andrew Ash (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13894196#comment-13894196
 ] 

Andrew Ash commented on HDFS-3570:
--

I applied this patch to cdh4.4.0 source and swapped out the hadoop-hdfs-*jar in 
my hadoop install's lib/ directory.  Running hadoop balancer shows the new 
option for -policy now and running with that option on my 5-node cluster is 
progressing.  The blocks are being shifted in a way that's the same for both 
the datanode and datanode-actual policies though, so it will be a bit before I 
can determine that the end state is the proper one.

 Balancer shouldn't rely on DFS Space Used % as that ignores non-DFS used 
 space
 

 Key: HDFS-3570
 URL: https://issues.apache.org/jira/browse/HDFS-3570
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: balancer
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Assignee: Akira AJISAKA
Priority: Minor
 Attachments: HDFS-3570.2.patch, HDFS-3570.aash.1.patch


 Report from a user here: 
 https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pIhNyDVxdVY/b7ENZmEvBjIJ,
  post archived at http://pastebin.com/eVFkk0A0
 This user had a specific DN that had a large non-DFS usage among 
 dfs.data.dirs, and very little DFS usage (which is computed against total 
 possible capacity). 
 Balancer apparently only looks at the usage, and ignores to consider that 
 non-DFS usage may also be high on a DN/cluster. Hence, it thinks that if a 
 DFS Usage report from DN is 8% only, its got a lot of free space to write 
 more blocks, when that isn't true as shown by the case of this user. It went 
 on scheduling writes to the DN to balance it out, but the DN simply can't 
 accept any more blocks as a result of its disks' state.
 I think it would be better if we _computed_ the actual utilization based on 
 {{(100-(actual remaining space))/(capacity)}}, as opposed to the current 
 {{(dfs used)/(capacity)}}. Thoughts?
 This isn't very critical, however, cause it is very rare to see DN space 
 being used for non DN data, but it does expose a valid bug.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HDFS-3570) Balancer shouldn't rely on DFS Space Used % as that ignores non-DFS used space

2014-02-05 Thread Andrew Ash (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13892807#comment-13892807
 ] 

Andrew Ash commented on HDFS-3570:
--

+1  I'm observing this issue on cdh4.4.0

My setup is that there are 4 nodes, 3 of which are dedicated solely to HDFS and 
the 4th is partially HDFS and partially other data.  When running the balancer 
it moves data from the lesser-used 3 nodes onto the higher-used 4th node 
because its Used(%) is lower, and eventually fills up the 4th node's disk while 
the remaining 3 nodes are underutilized.  I'd rather the balancing be done 
based on DFS Used %, which I'd define as:

DFS Used %  =  UsedTB / (ConfiguredCapacityTB - NonDFSUsedTB)

This hasn't been addressed in another ticket somewhere has it?

 Balancer shouldn't rely on DFS Space Used % as that ignores non-DFS used 
 space
 

 Key: HDFS-3570
 URL: https://issues.apache.org/jira/browse/HDFS-3570
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: balancer
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Priority: Minor

 Report from a user here: 
 https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pIhNyDVxdVY/b7ENZmEvBjIJ,
  post archived at http://pastebin.com/eVFkk0A0
 This user had a specific DN that had a large non-DFS usage among 
 dfs.data.dirs, and very little DFS usage (which is computed against total 
 possible capacity). 
 Balancer apparently only looks at the usage, and ignores to consider that 
 non-DFS usage may also be high on a DN/cluster. Hence, it thinks that if a 
 DFS Usage report from DN is 8% only, its got a lot of free space to write 
 more blocks, when that isn't true as shown by the case of this user. It went 
 on scheduling writes to the DN to balance it out, but the DN simply can't 
 accept any more blocks as a result of its disks' state.
 I think it would be better if we _computed_ the actual utilization based on 
 {{(100-(actual remaining space))/(capacity)}}, as opposed to the current 
 {{(dfs used)/(capacity)}}. Thoughts?
 This isn't very critical, however, cause it is very rare to see DN space 
 being used for non DN data, but it does expose a valid bug.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HDFS-3570) Balancer shouldn't rely on DFS Space Used % as that ignores non-DFS used space

2014-02-05 Thread Akira AJISAKA (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13893034#comment-13893034
 ] 

Akira AJISAKA commented on HDFS-3570:
-

I looked around the JIRA but cannot find another ticket addressing this.

bq. DFS Used % = UsedTB / (ConfiguredCapacityTB - NonDFSUsedTB)

+1 for the definition. I'll create a patch.

 Balancer shouldn't rely on DFS Space Used % as that ignores non-DFS used 
 space
 

 Key: HDFS-3570
 URL: https://issues.apache.org/jira/browse/HDFS-3570
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: balancer
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Priority: Minor

 Report from a user here: 
 https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pIhNyDVxdVY/b7ENZmEvBjIJ,
  post archived at http://pastebin.com/eVFkk0A0
 This user had a specific DN that had a large non-DFS usage among 
 dfs.data.dirs, and very little DFS usage (which is computed against total 
 possible capacity). 
 Balancer apparently only looks at the usage, and ignores to consider that 
 non-DFS usage may also be high on a DN/cluster. Hence, it thinks that if a 
 DFS Usage report from DN is 8% only, its got a lot of free space to write 
 more blocks, when that isn't true as shown by the case of this user. It went 
 on scheduling writes to the DN to balance it out, but the DN simply can't 
 accept any more blocks as a result of its disks' state.
 I think it would be better if we _computed_ the actual utilization based on 
 {{(100-(actual remaining space))/(capacity)}}, as opposed to the current 
 {{(dfs used)/(capacity)}}. Thoughts?
 This isn't very critical, however, cause it is very rare to see DN space 
 being used for non DN data, but it does expose a valid bug.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HDFS-3570) Balancer shouldn't rely on DFS Space Used % as that ignores non-DFS used space

2014-02-05 Thread Andrew Ash (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13893037#comment-13893037
 ] 

Andrew Ash commented on HDFS-3570:
--

I've almost got one ready, so give me a couple hours and I'll get it in?





 Balancer shouldn't rely on DFS Space Used % as that ignores non-DFS used 
 space
 

 Key: HDFS-3570
 URL: https://issues.apache.org/jira/browse/HDFS-3570
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: balancer
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Assignee: Akira AJISAKA
Priority: Minor

 Report from a user here: 
 https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pIhNyDVxdVY/b7ENZmEvBjIJ,
  post archived at http://pastebin.com/eVFkk0A0
 This user had a specific DN that had a large non-DFS usage among 
 dfs.data.dirs, and very little DFS usage (which is computed against total 
 possible capacity). 
 Balancer apparently only looks at the usage, and ignores to consider that 
 non-DFS usage may also be high on a DN/cluster. Hence, it thinks that if a 
 DFS Usage report from DN is 8% only, its got a lot of free space to write 
 more blocks, when that isn't true as shown by the case of this user. It went 
 on scheduling writes to the DN to balance it out, but the DN simply can't 
 accept any more blocks as a result of its disks' state.
 I think it would be better if we _computed_ the actual utilization based on 
 {{(100-(actual remaining space))/(capacity)}}, as opposed to the current 
 {{(dfs used)/(capacity)}}. Thoughts?
 This isn't very critical, however, cause it is very rare to see DN space 
 being used for non DN data, but it does expose a valid bug.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HDFS-3570) Balancer shouldn't rely on DFS Space Used % as that ignores non-DFS used space

2014-02-05 Thread Andrew Ash (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13893039#comment-13893039
 ] 

Andrew Ash commented on HDFS-3570:
--

Haven't actually tested or even compiled this, but it's what I had in mind.  
[~ajisakaa] is that similar to what you were thinking?

 Balancer shouldn't rely on DFS Space Used % as that ignores non-DFS used 
 space
 

 Key: HDFS-3570
 URL: https://issues.apache.org/jira/browse/HDFS-3570
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: balancer
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Assignee: Akira AJISAKA
Priority: Minor
 Attachments: HDFS-3570.aash.1.patch


 Report from a user here: 
 https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pIhNyDVxdVY/b7ENZmEvBjIJ,
  post archived at http://pastebin.com/eVFkk0A0
 This user had a specific DN that had a large non-DFS usage among 
 dfs.data.dirs, and very little DFS usage (which is computed against total 
 possible capacity). 
 Balancer apparently only looks at the usage, and ignores to consider that 
 non-DFS usage may also be high on a DN/cluster. Hence, it thinks that if a 
 DFS Usage report from DN is 8% only, its got a lot of free space to write 
 more blocks, when that isn't true as shown by the case of this user. It went 
 on scheduling writes to the DN to balance it out, but the DN simply can't 
 accept any more blocks as a result of its disks' state.
 I think it would be better if we _computed_ the actual utilization based on 
 {{(100-(actual remaining space))/(capacity)}}, as opposed to the current 
 {{(dfs used)/(capacity)}}. Thoughts?
 This isn't very critical, however, cause it is very rare to see DN space 
 being used for non DN data, but it does expose a valid bug.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HDFS-3570) Balancer shouldn't rely on DFS Space Used % as that ignores non-DFS used space

2014-02-05 Thread Akira AJISAKA (JIRA)

[
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13893052#comment-13893052
]

Akira AJISAKA commented on HDFS-3570:
-

Thanks for the patch! It's very similar to my thinking.
I think a test for the new parameter is needed.

Balancer shouldn't rely on DFS Space Used % as that ignores non-DFS used
space

Key: HDFS-3570
URL: https://issues.apache.org/jira/browse/HDFS-3570
Project: Hadoop HDFS
Issue Type: Bug
Components: balancer
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Priority: Minor
Attachments: HDFS-3570.aash.1.patch

--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HDFS-3570) Balancer shouldn't rely on DFS Space Used % as that ignores non-DFS used space

2014-02-05 Thread Andrew Ash (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13893058#comment-13893058
 ] 

Andrew Ash commented on HDFS-3570:
--

A test is certainly warranted, but I hadn't gotten that far yet since I
wanted to save you from doing duplicate work.  I'm not an expert on this
codebase, so if you'd like to take it from here I'd truly appreciate it.





 Balancer shouldn't rely on DFS Space Used % as that ignores non-DFS used 
 space
 

 Key: HDFS-3570
 URL: https://issues.apache.org/jira/browse/HDFS-3570
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: balancer
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Priority: Minor
 Attachments: HDFS-3570.aash.1.patch


 Report from a user here: 
 https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pIhNyDVxdVY/b7ENZmEvBjIJ,
  post archived at http://pastebin.com/eVFkk0A0
 This user had a specific DN that had a large non-DFS usage among 
 dfs.data.dirs, and very little DFS usage (which is computed against total 
 possible capacity). 
 Balancer apparently only looks at the usage, and ignores to consider that 
 non-DFS usage may also be high on a DN/cluster. Hence, it thinks that if a 
 DFS Usage report from DN is 8% only, its got a lot of free space to write 
 more blocks, when that isn't true as shown by the case of this user. It went 
 on scheduling writes to the DN to balance it out, but the DN simply can't 
 accept any more blocks as a result of its disks' state.
 I think it would be better if we _computed_ the actual utilization based on 
 {{(100-(actual remaining space))/(capacity)}}, as opposed to the current 
 {{(dfs used)/(capacity)}}. Thoughts?
 This isn't very critical, however, cause it is very rare to see DN space 
 being used for non DN data, but it does expose a valid bug.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HDFS-3570) Balancer shouldn't rely on DFS Space Used % as that ignores non-DFS used space

2014-02-05 Thread Akira AJISAKA (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13893062#comment-13893062
 ] 

Akira AJISAKA commented on HDFS-3570:
-

I'll add a test. Thanks, [~ash211]!

 Balancer shouldn't rely on DFS Space Used % as that ignores non-DFS used 
 space
 

 Key: HDFS-3570
 URL: https://issues.apache.org/jira/browse/HDFS-3570
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: balancer
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Priority: Minor
 Attachments: HDFS-3570.aash.1.patch


 Report from a user here: 
 https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pIhNyDVxdVY/b7ENZmEvBjIJ,
  post archived at http://pastebin.com/eVFkk0A0
 This user had a specific DN that had a large non-DFS usage among 
 dfs.data.dirs, and very little DFS usage (which is computed against total 
 possible capacity). 
 Balancer apparently only looks at the usage, and ignores to consider that 
 non-DFS usage may also be high on a DN/cluster. Hence, it thinks that if a 
 DFS Usage report from DN is 8% only, its got a lot of free space to write 
 more blocks, when that isn't true as shown by the case of this user. It went 
 on scheduling writes to the DN to balance it out, but the DN simply can't 
 accept any more blocks as a result of its disks' state.
 I think it would be better if we _computed_ the actual utilization based on 
 {{(100-(actual remaining space))/(capacity)}}, as opposed to the current 
 {{(dfs used)/(capacity)}}. Thoughts?
 This isn't very critical, however, cause it is very rare to see DN space 
 being used for non DN data, but it does expose a valid bug.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

39 matches

Mail list logo