[
https://issues.apache.org/jira/browse/HADOOP-2065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12590624#action_12590624
]
lohit vijayarenu commented on HADOOP-2065:
------------------------------------------
Had a discussion with Dhruba and Raghu regarding the organization of blocks and
how best the situation described in this JIRA could be addressed.
To summarize: A block could have all of its replicas corrupted at any point of
time in HDFS. By default, HDFS detects this and deletes corrupted blocks until
the last copy. The last copy is never deleted. The request of this JIRA is to
allow replication of this corrupted single replica so that we have more than
one copy of this block even if it is corrupt. Another way to look at the
problem is that when all replicas are corrupt, lets not delete any of them.
Initially, we thought we should somehow mark a block as corrupt if we identify
its replicas are bad, but without checking (by reading or datanode reporting as
bad block), we do not know if all replicas of the block are corrupt. Consider
the case when HDFS is up and running and one of the datanode reports that the
replica it holds is corrupt. The current behavior is that we delete this block
and request for replication.
One proposed idea is as follows. When we get a notification from Datanode that
the block is corrupt, we mark the block associated with this datanode to be
corrupt and store it in DatanodeDescriptor as a list possibly. We also request
a replication of this block but do not deleted this replica yet. At this point,
if we have another good replica, it would get replicated and eventually we
would get the additional addBlock request. Now, we check if this is an
additonal copy and if this block has corrupt replica, if so, we add this new
block and delete the corrupt replica. If a block has all of its copies as
corrupt, then in some time, we do come to know about this and we havnt
requested them to be deleted any of them. We should thinking about how to
filter this copy (similar to decommission flag) and how best such blocks could
be reported.
Thoughts?
> Replication policy for corrupted block
> ---------------------------------------
>
> Key: HADOOP-2065
> URL: https://issues.apache.org/jira/browse/HADOOP-2065
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Koji Noguchi
> Assignee: lohit vijayarenu
> Fix For: 0.18.0
>
>
> Thanks to HADOOP-1955, even if one of the replica is corrupted, the block
> should get replicated from a good replica relatively fast.
> Created this ticket to continue the discussion from
> http://issues.apache.org/jira/browse/HADOOP-1955#action_12531162.
> bq. 2. Delete corrupted source replica
> bq. 3. If all replicas are corrupt, stop replication.
> For (2), it'll be nice if the namenode can delete the corrupted block if
> there's a good replica on other nodes.
> For (3), I prefer if the namenode can still replicate the block.
> Before 0.14, if the file was corrupted, users were still able to pull the
> data and decide if they want to delete those files. (HADOOP-2063)
> In 0.14 and later, we cannot/don't replicate these blocks so they eventually
> get lost.
> To make the matters worse, if the corrupted file is accessed, all the
> corrupted replicas would be deleted except for one and stay as replication
> factor of 1 forever.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
