[
https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15018761#comment-15018761
]
Arpit Agarwal commented on HDFS-9038:
-------------------------------------
Hi [~brahmareddy], thanks for taking this up. The approach looks good.
# The calculation in FsVolumeImpl#getNonDfsUsed looks wrong. Won't it always be
zero? The correct calculation should replace getAvailable() with actual volume
free space. Something like
{{Files.getFileStore(currentDir.toPath()).getUnallocatedSpace()}} perhaps.
{code}
public long getAvailable() throws IOException {
long remaining = getCapacity() - getDfsUsed() - reservedForReplicas.get();
...
public long getNonDfsUsed() throws IOException {
long nonDFSUsed = getCapacity() - getDfsUsed() - getAvailable()
- reservedForReplicas.get();
{code}
# Can you add a targeted unit test for the above calculation? A mock-based test
would be ideal.
# If the DN does not report nonDfsUsed, the NN should initialize it using the
older calculation. Else the NN will report nonDfsUsed as zero for older DNs
e.g. during upgrade. The fix can be done in
{{PBHelperClient#convert(DatanodeInfoProto)}}.
# We can expose nonDfsUsed via StorageTypeStats and DatanodeStatistics. Okay to
do in a separate Jira.
# Also we can expose reservedForReplicas in a separate Jira. Previously this
space was accounted for in nonDfsUsed but now it will not be accounted for
anywhere.
# Unrelated - we should replace {{DatanodeInfo}} constructors with a builder
pattern in a separate Jira.
> Reserved space is erroneously counted towards non-DFS used.
> -----------------------------------------------------------
>
> Key: HDFS-9038
> URL: https://issues.apache.org/jira/browse/HDFS-9038
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
> Affects Versions: 2.7.1
> Reporter: Chris Nauroth
> Assignee: Brahma Reddy Battula
> Attachments: HDFS-9038-002.patch, HDFS-9038-003.patch,
> HDFS-9038-004.patch, HDFS-9038.patch
>
>
> HDFS-5215 changed the DataNode volume available space calculation to consider
> the reserved space held by the {{dfs.datanode.du.reserved}} configuration
> property. As a side effect, reserved space is now counted towards non-DFS
> used. I don't believe it was intentional to change the definition of non-DFS
> used. This issue proposes restoring the prior behavior: do not count
> reserved space towards non-DFS used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
