[
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17620309#comment-17620309
]
ASF GitHub Bot commented on HDFS-3570:
--------------------------------------
ashutoshcipher opened a new pull request, #5044:
URL: https://github.com/apache/hadoop/pull/5044
### Description of PR
**Balancer shouldn't rely on "DFS Space Used %" as that ignores non-DFS used
space**
Report from a user here:
https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pIhNyDVxdVY/b7ENZmEvBjIJ
(Not available now) , post archived at http://pastebin.com/eVFkk0A0
This user had a specific DN that had a large non-DFS usage among
dfs.data.dirs, and very little DFS usage (which is computed against total
possible capacity).
Balancer apparently only looks at the usage, and ignores to consider that
non-DFS usage may also be high on a DN/cluster. Hence, it thinks that if a DFS
Usage report from DN is 8% only, its got a lot of free space to write more
blocks, when that isn't true as shown by the case of this user. It went on
scheduling writes to the DN to balance it out, but the DN simply can't accept
any more blocks as a result of its disks' state.
It would be better if we computed the actual utilization based on
(100-(actual remaining space))/(capacity), as opposed to the current (dfs
used)/(capacity). Thoughts?
This isn't very critical, however, cause it is very rare to see DN space
being used for non DN data, but it does expose a valid bug.
### How was this patch tested?
UT
### For code changes:
- [X] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
> Balancer shouldn't rely on "DFS Space Used %" as that ignores non-DFS used
> space
> --------------------------------------------------------------------------------
>
> Key: HDFS-3570
> URL: https://issues.apache.org/jira/browse/HDFS-3570
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
> Affects Versions: 2.0.0-alpha
> Reporter: Harsh J
> Assignee: Ashutosh Gupta
> Priority: Minor
> Attachments: HDFS-3570.003.patch, HDFS-3570.2.patch,
> HDFS-3570.aash.1.patch
>
>
> Report from a user here:
> https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pIhNyDVxdVY/b7ENZmEvBjIJ,
> post archived at http://pastebin.com/eVFkk0A0
> This user had a specific DN that had a large non-DFS usage among
> dfs.data.dirs, and very little DFS usage (which is computed against total
> possible capacity).
> Balancer apparently only looks at the usage, and ignores to consider that
> non-DFS usage may also be high on a DN/cluster. Hence, it thinks that if a
> DFS Usage report from DN is 8% only, its got a lot of free space to write
> more blocks, when that isn't true as shown by the case of this user. It went
> on scheduling writes to the DN to balance it out, but the DN simply can't
> accept any more blocks as a result of its disks' state.
> I think it would be better if we _computed_ the actual utilization based on
> {{(100-(actual remaining space))/(capacity)}}, as opposed to the current
> {{(dfs used)/(capacity)}}. Thoughts?
> This isn't very critical, however, cause it is very rare to see DN space
> being used for non DN data, but it does expose a valid bug.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
