[
https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15321892#comment-15321892
]
Arpit Agarwal edited comment on HDFS-9038 at 6/9/16 5:02 AM:
-------------------------------------------------------------
I spent a few hours looking at this today. I now think that the current logic
in FsVolumeImpl#getAvailable is too pessimistic and the impact of bug may be
more severe than reported.
{code}
public long getAvailable() throws IOException {
long remaining = getCapacity() - getDfsUsed() - reservedForReplicas.get();
long available = usage.getAvailable() - reserved
- reservedForReplicas.get();
if (remaining > available) {
remaining = available;
}
return (remaining > 0) ? remaining : 0;
}
{code}
Specifically {{available = usage.getAvailable() - reserved ... }} can account
for the non-DFS usage twice. e.g. say 10GB is reserved and YARN is using all of
that 10GB. usage.getAvailable() has already accounted for that 10GB and further
subtracting it makes us underestimate available.
Instead the calculation should only subtract the unused portion of non-DFS
used. What do you guys think?
was (Author: arpitagarwal):
I spent a few hours looking at this today. I now think that the current logic
in FsVolumeImpl#getAvailable is too pessimistic the impact of bug may be more a
more severe than reported.
{code}
public long getAvailable() throws IOException {
long remaining = getCapacity() - getDfsUsed() - reservedForReplicas.get();
long available = usage.getAvailable() - reserved
- reservedForReplicas.get();
if (remaining > available) {
remaining = available;
}
return (remaining > 0) ? remaining : 0;
}
{code}
Specifically {{available = usage.getAvailable() - reserved ... }} can account
for the non-DFS usage twice. e.g. say 10GB is reserved and YARN is using all of
that 10GB. usage.getAvailable() has already accounted for that 10GB and further
subtracting it makes us underestimate available.
Instead the calculation should only subtract the unused portion of non-DFS
used. What do you guys think?
> DFS reserved space is erroneously counted towards non-DFS used.
> ---------------------------------------------------------------
>
> Key: HDFS-9038
> URL: https://issues.apache.org/jira/browse/HDFS-9038
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
> Affects Versions: 2.7.1
> Reporter: Chris Nauroth
> Assignee: Brahma Reddy Battula
> Attachments: GetFree.java, HDFS-9038-002.patch, HDFS-9038-003.patch,
> HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038-006.patch,
> HDFS-9038-007.patch, HDFS-9038.patch
>
>
> HDFS-5215 changed the DataNode volume available space calculation to consider
> the reserved space held by the {{dfs.datanode.du.reserved}} configuration
> property. As a side effect, reserved space is now counted towards non-DFS
> used. I don't believe it was intentional to change the definition of non-DFS
> used. This issue proposes restoring the prior behavior: do not count
> reserved space towards non-DFS used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
