[
https://issues.apache.org/jira/browse/HDFS-4833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13659988#comment-13659988
]
Todd Lipcon commented on HDFS-4833:
-----------------------------------
The issue here is that {{invalidateCorruptReplicas}} is only called from
{{addStoredBlock}} in the latter half of the function after the early return
when {{!isPopulatingReplQueues()}}.
When the NN is about to exit safe mode, it calls {{processMisReplicatedBlock}}
on each block, but that function does not call through to invalidate the
corrupt replicas unless the block is also over-replicated.
I was able to reproduce this as follows:
- Start a cluster with one DN. Ingest 90 blocks worth of data at replication
count 1.
- Add a second DN.
- Shutdown the cluster.
- Copy one of the blocks from the first DN to the second DN, but modify its
meta file to have an earlier genstamp.
- Restart the NN.
- Restart the DN which has the corrupt block
- Restart the DN which has the rest of the blocks.
You will see the corrupt block count stuck at 1 and never dropping.
> Corrupt blocks are not invalidated when first processing repl queues
> --------------------------------------------------------------------
>
> Key: HDFS-4833
> URL: https://issues.apache.org/jira/browse/HDFS-4833
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 2.0.4-alpha
> Reporter: Todd Lipcon
>
> When the NN processes misreplicated blocks in {{processMisReplicatedBlock}}
> (eg during initial startup when first processing repl queues), it does not
> invalidate corrupt replicas unless the block is also over-replicated. This
> can result in replicas stuck in "corrupt" state forever if they were that way
> when the cluster booted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
