[
https://issues.apache.org/jira/browse/HDFS-11257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16526963#comment-16526963
]
Íñigo Goiri commented on HDFS-11257:
------------------------------------
bq. The NN cannot know/decide if the disk is filling from an abusive job or
from another "legitimate" tenant of the host.
Correct, this is something specific to our scenario where we know we need to
evacuate.
Another option would be to enable this per DN for example.
I'm not sure if we can achieve the behavior from [^HDFS-11257.000.patch] using
SPS.
[~andrew.wang], ideas?
> Evacuate DN when the remaining is negative
> ------------------------------------------
>
> Key: HDFS-11257
> URL: https://issues.apache.org/jira/browse/HDFS-11257
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
> Reporter: Íñigo Goiri
> Assignee: Anbang Hu
> Priority: Major
> Attachments: HDFS-11257.000.patch
>
>
> Datanodes have a maximum amount of disk they can use. This is set using
> {{dfs.datanode.du.reserved}}. For example, if we have a 1TB disk and we set
> the reserved to 100GB, the DN can only use ~900GB. However, if we fill the DN
> and later other processes (e.g., logs or co-located services) start to use
> the disk space, the remaining space will go to a negative and the used
> storage >100%.
> The Rebalancer or decommissioning would cover this situation. However, both
> approaches require administrator intervention while this is a situation that
> violates the settings. Note that decommisioning, would be too extreme as it
> would evacuate all the data.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
