[
https://issues.apache.org/jira/browse/HDFS-7960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-7960:
---------------------------------------
Description: The full block report should prune zombie storages even if
they're not empty. We have seen cases in production where zombie storages have
not been pruned subsequent to HDFS-7575. This could arise any time the
NameNode thinks there is a block in some old storage which is actually not
there. In this case, the block will not show up in the "new" storage (once old
is renamed to new) and the old storage will linger forever as a zombie, even
with the HDFS-7596 fix applied. This also happens with datanode hotplug, when
a drive is removed. In this case, an entire storage (volume) goes away but the
blocks do not show up in another storage on the same datanode. (was: The full
block report should contain a "numStoragesOnDn" field that allows the NameNode
to prune old storages. Currently, the NameNode can't do this pruning because
it doesn't know how many storages are on the DataNode, and the full block
report can be split into multiple RPCs.
While it is true that storages will be removed by HDFS-7596 when they are
empty, this mechanism doesn't work for datanode hotplug. In the case of
datanode hotplug, an entire storage simply goes away because of
reconfiguration. Currently, the NN may never perceive this "zombie storage" as
being empty, unless the NN itself is restarted.)
> The full block report should prune zombie storages even if they're not empty
> ----------------------------------------------------------------------------
>
> Key: HDFS-7960
> URL: https://issues.apache.org/jira/browse/HDFS-7960
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.6.0
> Reporter: Lei (Eddy) Xu
> Assignee: Colin Patrick McCabe
> Priority: Critical
>
> The full block report should prune zombie storages even if they're not empty.
> We have seen cases in production where zombie storages have not been pruned
> subsequent to HDFS-7575. This could arise any time the NameNode thinks there
> is a block in some old storage which is actually not there. In this case,
> the block will not show up in the "new" storage (once old is renamed to new)
> and the old storage will linger forever as a zombie, even with the HDFS-7596
> fix applied. This also happens with datanode hotplug, when a drive is
> removed. In this case, an entire storage (volume) goes away but the blocks
> do not show up in another storage on the same datanode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
