[
https://issues.apache.org/jira/browse/HADOOP-1704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12525484
]
Raghu Angadi commented on HADOOP-1704:
--------------------------------------
> Raghu's block crc upgrade code throttles the deletion of .crc files.
blockCrc upgrade does not throttle deletes explicitly. It deletes in a single
thread and each deletiion results in editsLog entry.. these two naturally
throttle the rate to around 700-800 deletes a second. Deleting a directory
deletes whole tree with sigle editsLog entry. This is the difference.
The similarity is that that none of these 5M blocks were removed from the
namespace until after the upgrade is complete. In that sense memory overhead is
same as deleting 5M files in one shot.
> Throttling for HDFS Trash purging
> ---------------------------------
>
> Key: HADOOP-1704
> URL: https://issues.apache.org/jira/browse/HADOOP-1704
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
>
> When HDFS Trash is enabled, deletion of a file/directory results in it being
> moved to the "Trash" directory. The "Trash" directory is periodically purged
> by the Namenode. This means that all files/directories that users deleted in
> the last Trash period, gets "really" deleted when the Trash purging occurs.
> This might cause a burst of file/directory deletions.
> The Namenode tracks blocks that belonged to deleted files in a data structure
> named "RecentInvalidateSets". There is a possibility that Trash purging may
> cause this data structure to bloat, causing undesireable behaviour of the
> Namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
