[
https://issues.apache.org/jira/browse/HADOOP-1444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
dhruba borthakur updated HADOOP-1444:
-------------------------------------
Attachment: duplicateBlockId.patch
This patch does two things:
1. Moves all the pendingCreates logic into PendingCreates.java. This data
structures encapsulates pendingCreates and pendingCreateBlocks from
FSNamesystem.
2. The block allocation routine FSNamesystem.allocate() verifies that the new
blockid that is generated does not already exists in the blocksMap as well as
pendingCreates.
> Block allocation method does not check pendingCreates for duplicate block ids
> -----------------------------------------------------------------------------
>
> Key: HADOOP-1444
> URL: https://issues.apache.org/jira/browse/HADOOP-1444
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: duplicateBlockId.patch
>
>
> The HDFS namenode allocates a new random blockid when requested. It then
> checks the blocksMap to verify if this blockid is already in use. If this
> block is is already in use, it generates another random number and above
> process continues. When a blocksid that does not exist in the blocksMap is
> found, it stores this blocksid in pendingCreateBlocks and returns the
> blocksid to the requesting client.
> The above check for detecting duplicate blockid should check
> pendingCreateBlocks as well.
> A related problem exists when a file is deleted. Deleting a file causes all
> its blocks to be deleted from the blocksMap immediately. These blockids move
> to recentInvalidateSets and are sent out to the corresponding datanodes as
> part of responses to succeeding heartbeats. So, there is a time window when a
> block exists in the datanode but not in the blocksMap. At this time, if the
> blockid-random-number generator generates a blockid that exists in the
> datanode but not on the blocksMap, then the namenode will fail to detect that
> this is a duplicate blockid.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
