[
https://issues.apache.org/jira/browse/HDFS-7960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-7960:
---------------------------------------
Description:
The full block report should contain a "numStoragesOnDn" field that allows the
NameNode to prune old storages. Currently, the NameNode can't do this pruning
because it doesn't know how many storages are on the DataNode, and the full
block report can be split into multiple RPCs.
While it is true that storages will be removed by HDFS-7596 when they are
empty, this mechanism doesn't work for datanode hotplug. In the case of
datanode hotplug, an entire storage simply goes away because of
reconfiguration. Currently, the NN may never perceive this "zombie storage" as
being empty, unless the NN itself is restarted.
was:NameNode should be able to remove storages that is not represented on
DataNode. For example, hot swapped on that DataNode or DN restarted with less
data dirs. It seems that once a datanode storage is removed from a datanode,
those blocks on the storage will not be reconciled as gone from the Namenode
until the namenode has been restarted.
Priority: Critical (was: Major)
Target Version/s: 2.7.0 (was: 3.0.0)
Summary: The full block report should contain a "numStoragesOnDn"
field that allows the NameNode to prune old storages (was: NameNode should
prune storages that are no longer existed on DataNode)
> The full block report should contain a "numStoragesOnDn" field that allows
> the NameNode to prune old storages
> -------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-7960
> URL: https://issues.apache.org/jira/browse/HDFS-7960
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.6.0
> Reporter: Lei (Eddy) Xu
> Assignee: Colin Patrick McCabe
> Priority: Critical
>
> The full block report should contain a "numStoragesOnDn" field that allows
> the NameNode to prune old storages. Currently, the NameNode can't do this
> pruning because it doesn't know how many storages are on the DataNode, and
> the full block report can be split into multiple RPCs.
> While it is true that storages will be removed by HDFS-7596 when they are
> empty, this mechanism doesn't work for datanode hotplug. In the case of
> datanode hotplug, an entire storage simply goes away because of
> reconfiguration. Currently, the NN may never perceive this "zombie storage"
> as being empty, unless the NN itself is restarted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
