[
https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15052039#comment-15052039
]
Vinayakumar B commented on HDFS-9038:
-------------------------------------
Thanks [~cnauroth] for the explanation.
Lets go through one simple example.
Lets consider following
1. Disk Capacity - 10GB -- {{File#getTotalSpace}}
2. DFS reserved - 1GB -- {{dfs.datanode.du.reserved}}
3. DFS Usage - 2GB -- {{dfsUsed}}
4. Usage by other files - 3GB -- Actual {{non-dfsusage}}.
Assuming, System reserved ( which is ~5%) as 0 for this example.
total disk used is 5GB = 2GB by dfs files + 3GB by non-dfs files.
{{File.getUsableSpace()}} returns 5GB.
Lets substitute this values in derived expressions.
*Before HDFS-5215*
{noformat}
nonDfsUsage=File#getTotalSpace - dfs.datanode.du.reserved - dfsUsed -
File#getUsableSpace
nonDfsUsage = 10G - 1G - 2G - 5G
nonDfsUsage = 2G.
{noformat}
{color:red}{{nonDfsUsage=2GB}} is not same as actual non-dfs usage, 3GB.{color}
*After HDFS-5215* and as per patch.
{noformat}
nonDfsUsage=(File#getTotalSpace - dfs.datanode.du.reserved) - dfsUsed -
(File#getUsableSpace - dfs.datanode.du.reserved)
nonDfSUsage= (10G - 1G ) - 2G - (5G - 1G)
nonDfSUsage = 9G - 2G - 4G
nonDfsUsage = 3G
{noformat}
{color:green}{{nonDfsUsage = 3G}} is same as expected actual usage. i.e.
3G{color}
By seeing the above example,
I believe, post HDFS-5215, calculation turns out to be the correct one.
{{dfs.datanode,du.reserved}} is expected to be the freespace, which should not
be used by DFS. So its reasonable to subtract this in getAvailable(), but not
while finding the non-dfs usage.
Do you agree [~cnauroth]?
> DFS reserved space is erroneously counted towards non-DFS used.
> ---------------------------------------------------------------
>
> Key: HDFS-9038
> URL: https://issues.apache.org/jira/browse/HDFS-9038
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
> Affects Versions: 2.7.1
> Reporter: Chris Nauroth
> Assignee: Brahma Reddy Battula
> Attachments: HDFS-9038-002.patch, HDFS-9038-003.patch,
> HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038-006.patch, HDFS-9038.patch
>
>
> HDFS-5215 changed the DataNode volume available space calculation to consider
> the reserved space held by the {{dfs.datanode.du.reserved}} configuration
> property. As a side effect, reserved space is now counted towards non-DFS
> used. I don't believe it was intentional to change the definition of non-DFS
> used. This issue proposes restoring the prior behavior: do not count
> reserved space towards non-DFS used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
