[
https://issues.apache.org/jira/browse/HDFS-4799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated HDFS-4799:
------------------------------
Attachment: hdfs-4799.txt
Here's a patch which should fix the issue.
I took the least invasive route I could to fix the problem, by just having
{{invalidateBlock}} return a boolean indicating whether it was actually
invalidated. Only if it was invalidated do we remove from the corruptReplicas
set.
I'm not certain that the call to corruptReplicas.removeFromCorruptReplicasMap
is even necessary here at all, since {{removeStoredBlock}} itself will call
that when removing from the blocksMap. But, I didn't want to make the larger
change here, lest I accidentally introduce a leak in the corruptReplica
tracking structure (something we don't have good automated tests for).
The patch also includes a unit test which reliably fails on the unpatched code
> Corrupt replica can be prematurely removed from corruptReplicas map
> -------------------------------------------------------------------
>
> Key: HDFS-4799
> URL: https://issues.apache.org/jira/browse/HDFS-4799
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
> Affects Versions: 2.0.4-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Attachments: hdfs-4799.txt, hdfs-4799-unittest.txt
>
>
> We saw the following sequence of events in a cluster result in losing the
> most recent genstamp of a block:
> - client is writing to a pipeline of 3
> - the pipeline had nodes fail over some period of time, such that it left 3
> old-genstamp replicas on the original three nodes, having recruited 3 new
> replicas with a later genstamp.
> -- so, we have 6 total replicas in the cluster, three with old genstamps on
> downed nodes, and 3 with the latest genstamp
> - cluster reboots, and the nodes with old genstamps blockReport first. The
> replicas are correctly added to the corrupt replicas map since they have a
> too-old genstamp
> - the nodes with the new genstamp block report. When the latest one block
> reports, chooseExcessReplicates is called and incorrectly decides to remove
> the three good replicas, leaving only the old-genstamp replicas.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
