[
https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15057077#comment-15057077
]
Arpit Agarwal edited comment on HDFS-9038 at 12/15/15 12:46 AM:
----------------------------------------------------------------
[~brahma], thanks for your patience as we work through the math. It may be
useful to describe your proposed derivation as a Jira comment before you post
another patch.
[~cnauroth] I think you are right in conclusion but there is a misstep in the
equations in [this
comment|https://issues.apache.org/jira/browse/HDFS-9038?focusedCommentId=15051894&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15051894].
{{reserved}} _should_ cancel out and it should not factor in the final
computation.
The problem with the v005 patch we missed earlier is that getCapacity()
subtracts reserved space. We should use the raw capacity.
One way to fix it is:
{code}
public long getNonDfsUsed() throws IOException {
long totalFreeSpace = currentDir.getFreeSpace();
long nonDfsUsed = getCapacity() + reserved - getDfsUsed() - totalFreeSpace;
return (nonDfsUsed >= 0) ? nonDfsUsed : 0;
}
{code}
Then the derivation becomes:
{code}
1: non-DFS used = getCapacity() + reserved - getDfsUsed() - totalFreeSpace
2: = usage.getCapacity() - reserved + reserved - getDfsUsed() -
totalFreeSpace
3: = usage.getCapacity() - getDfsUsed() - totalFreeSpace
4: = File#getTotalSpace - getDfsUsed() - File#getFreeSpace
{code}
Hope that makes sense.
was (Author: arpitagarwal):
[~brahma], thanks for your patience as we work through the math. It may be
useful to describe your proposed derivation as a Jira comment before you post
another patch.
[~cnauroth] I think you are right in conclusion but there is a misstep in the
equations in [this
comment|https://issues.apache.org/jira/browse/HDFS-9038?focusedCommentId=15051894&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15051894].
{{reserved}} _should_ cancel out and it should not factor in the final
computation.
The problem with the v005 patch we missed earlier is that getCapacity()
subtracts reserved space. We should use the raw capacity.
One way to fix it is:
{code}
public long getNonDfsUsed() throws IOException {
long totalFreeSpace = currentDir.getUsableSpace();
long nonDfsUsed = getCapacity() + reserved - getDfsUsed() - totalFreeSpace;
return (nonDfsUsed >= 0) ? nonDfsUsed : 0;
}
{code}
Then the derivation becomes:
{code}
1: non-DFS used = getCapacity() + reserved - getDfsUsed() - totalFreeSpace
2: = usage.getCapacity() - reserved + reserved - getDfsUsed() -
totalFreeSpace
3: = usage.getCapacity() - getDfsUsed() - totalFreeSpace
4: = File#getTotalSpace - getDfsUsed() - File#getFreeSpace
{code}
Hope that makes sense.
> DFS reserved space is erroneously counted towards non-DFS used.
> ---------------------------------------------------------------
>
> Key: HDFS-9038
> URL: https://issues.apache.org/jira/browse/HDFS-9038
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
> Affects Versions: 2.7.1
> Reporter: Chris Nauroth
> Assignee: Brahma Reddy Battula
> Attachments: HDFS-9038-002.patch, HDFS-9038-003.patch,
> HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038-006.patch,
> HDFS-9038-007.patch, HDFS-9038.patch
>
>
> HDFS-5215 changed the DataNode volume available space calculation to consider
> the reserved space held by the {{dfs.datanode.du.reserved}} configuration
> property. As a side effect, reserved space is now counted towards non-DFS
> used. I don't believe it was intentional to change the definition of non-DFS
> used. This issue proposes restoring the prior behavior: do not count
> reserved space towards non-DFS used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
