[
https://issues.apache.org/jira/browse/HDFS-6186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-6186:
----------------------------
Attachment: HDFS-6186.000.patch
Initial patch to demonstrate the idea. In the patch I record the creation time
for each DatanodeStorageInfo in NN. Then for an unknown block in full block
report, we can check for how long the DatanodeStorageInfo has been created. If
it is created less than 1h ago, we do not put the block into the invalidate
list but a map recording unknown blocks.
> Pause deletion of blocks when the namenode starts up
> ----------------------------------------------------
>
> Key: HDFS-6186
> URL: https://issues.apache.org/jira/browse/HDFS-6186
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
> Reporter: Suresh Srinivas
> Attachments: HDFS-6186.000.patch
>
>
> HDFS namenode can delete blocks very quickly, given the deletion happens as a
> parallel operation spread across many datanodes. One of the frequent
> anxieties I see is that a lot of data can be deleted very quickly, when a
> cluster is brought up, especially when one of the storage directories has
> failed and namenode metadata was copied from another storage. Copying wrong
> metadata would results in some of the newer files (if old metadata was
> copied) being deleted along with their blocks.
> HDFS-5986 now captures the number of pending deletion block on namenode webUI
> and JMX. I propose pausing deletion of blocks for a configured period of time
> (default 1 hour?) after namenode comes out of safemode. This will give enough
> time for the administrator to notice large number of pending deletion blocks
> and take corrective action.
> Thoughts?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
