[
https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15058764#comment-15058764
]
Chris Nauroth commented on HDFS-9038:
-------------------------------------
Thanks everyone for sticking with this. This has turned out to be much
trickier than I anticipated when I filed the issue. I'd like to summarize
current status.
Arpit and I are in agreement about my analysis of how the calculation changed
after HDFS-5215. However, we are not yet in agreement about which calculation
is truly correct. I believe the pre-HDFS-5215 calculation (subtracting
{{dfs.datanode.du.reserved}}) is correct, because it allowed me to monitor for
unexpected non-zero non-DFS usage and react. Since this was an established
operations workflow (at least for me), I argue that we have a responsibility to
restore that behavior. Arpit believes that it's correct to cancel out
{{dfs.datanode.du.reserved}}, because then non-DFS used would report space used
for non-HDFS purposes more accurately. Essentially, it's a question of whether
this metric means "Raw Non-DFS Used" or "Unplanned Non-DFS Used".
We also discovered an interesting side issue about {{File#getUsableSpace}} vs.
{{File#getFreeSpace}}. Pre-HDFS-5215, it could be considered a bug that we did
not account for system reserved space. Interestingly, it seems in our testing
that ext holds back 5% by default, but xfs does not.
I pushed pretty hard for restoring the pre-HDFS-5215 behavior in my earlier
comments, but I'm just one voice. I suggest that we leave this issue open for
a while for others to comment. I could be swayed if others think I'm
approaching this incorrectly. Meanwhile, [~brahmareddy], would you please hold
off on posting more patches? Let's wait for the discussion to settle a little
more first. Thanks for your patience.
> DFS reserved space is erroneously counted towards non-DFS used.
> ---------------------------------------------------------------
>
> Key: HDFS-9038
> URL: https://issues.apache.org/jira/browse/HDFS-9038
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
> Affects Versions: 2.7.1
> Reporter: Chris Nauroth
> Assignee: Brahma Reddy Battula
> Attachments: HDFS-9038-002.patch, HDFS-9038-003.patch,
> HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038-006.patch,
> HDFS-9038-007.patch, HDFS-9038.patch
>
>
> HDFS-5215 changed the DataNode volume available space calculation to consider
> the reserved space held by the {{dfs.datanode.du.reserved}} configuration
> property. As a side effect, reserved space is now counted towards non-DFS
> used. I don't believe it was intentional to change the definition of non-DFS
> used. This issue proposes restoring the prior behavior: do not count
> reserved space towards non-DFS used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
