[
https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15034818#comment-15034818
]
Arpit Agarwal commented on HDFS-9038:
-------------------------------------
Thanks for the updated patch [~brahmareddy].
bq. 3. I think we should cap the returned value to 0 as a matter of defensive
coding against negative values.
Hi [~cnauroth], this is a good idea.
bq. 2. The latest getNonDfsUsed does not include reservedForReplicas. I think
it should, since the reservedForReplicas amount is effectively in use by HDFS.
Brahma changed this at my suggestion. Exposing reserved bytes through
nonDfsUsed was inadvertent side effect of HDFS-6898. I see Brahma filed
HDFS-9481 to expose reserved bytes as a separate counter which should present a
clearer picture to administrators. What do you think?
bq. I think we should stick with File#getUsableSpace here
I looked at the JDK code to find the difference between {{File#getUsableSpace}}
and {{File#getFreeSpace}}.
http://hg.openjdk.java.net/jdk7/jdk7/jdk/file/9b8c96f96a0f/src/solaris/native/java/io/UnixFileSystem_md.c#l412
So it comes down to the difference between {{f_bavail}} and {{f_bfree}}.
{code}
struct statvfs {
unsigned long f_bsize; /* filesystem block size */
unsigned long f_frsize; /* fragment size */
fsblkcnt_t f_blocks; /* size of fs in f_frsize units */
fsblkcnt_t f_bfree; /* # free blocks */
fsblkcnt_t f_bavail; /* # free blocks for unprivileged
users */
{code}
{{f_bavail}} does not include some system-reserved partition space. The
consequence of using {{f_bavail}} is that non-DFS used will include this space
which is 5% of the volume size on ext2fs and successors, because
{{File#getTotalSpace}} counts everything.
http://www.microhowto.info/howto/reduce_the_space_reserved_for_root_on_an_ext2_ext3_or_ext4_filesystem.html
This may be confusing to administrators.
> Reserved space is erroneously counted towards non-DFS used.
> -----------------------------------------------------------
>
> Key: HDFS-9038
> URL: https://issues.apache.org/jira/browse/HDFS-9038
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
> Affects Versions: 2.7.1
> Reporter: Chris Nauroth
> Assignee: Brahma Reddy Battula
> Attachments: HDFS-9038-002.patch, HDFS-9038-003.patch,
> HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038.patch
>
>
> HDFS-5215 changed the DataNode volume available space calculation to consider
> the reserved space held by the {{dfs.datanode.du.reserved}} configuration
> property. As a side effect, reserved space is now counted towards non-DFS
> used. I don't believe it was intentional to change the definition of non-DFS
> used. This issue proposes restoring the prior behavior: do not count
> reserved space towards non-DFS used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
