Harsh J created HDFS-3570:
-----------------------------
Summary: Balancer shouldn't rely on "DFS Space Used %" as that
ignores non-DFS used space
Key: HDFS-3570
URL: https://issues.apache.org/jira/browse/HDFS-3570
Project: Hadoop HDFS
Issue Type: Bug
Components: balancer
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Priority: Minor
Report from a user here:
https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pIhNyDVxdVY/b7ENZmEvBjIJ,
post archived at http://pastebin.com/eVFkk0A0
This user had a specific DN that had a large non-DFS usage among dfs.data.dirs,
and very little DFS usage (which is computed against total possible capacity).
Balancer apparently only looks at the usage, and ignores to consider that
non-DFS usage may also be high on a DN/cluster. Hence, it thinks that if a DFS
Usage report from DN is 8% only, its got a lot of free space to write more
blocks, when that isn't true as shown by the case of this user. It went on
scheduling writes to the DN to balance it out, but the DN simply can't accept
any more blocks as a result of its disks' state.
I think it would be better if we _computed_ the actual utilization based on
{{(100-(actual remaining space))/(capacity)}}, as opposed to the current {{(dfs
used)/(capacity)}}. Thoughts?
This isn't very critical, however, cause it is very rare to see DN space being
used for non DN data, but it does expose a valid bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
