[
https://issues.apache.org/jira/browse/HDFS-173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745524#action_12745524
]
Suresh Srinivas commented on HDFS-173:
--------------------------------------
Currently following steps are taken for recursive deletion of directories:
# remove target inode from the parents. This recursively traverses the subtree
to compute quota used by the subtree to update the parent node quota.
# travers the subtree of target node again and delete inodes and collect blocks
# remove lease for the target node and the subtree
# delete blocks from the block map, corrupt map and schedule block for deletion
at datanode by adding it to {{BlockManager.recentInvalidateSets}}
While adding to {{BlockManager.recentInvalidateSets}}, in step 4 above, an info
level log is printed for every replica. In my testing, deletion time for 1
million files with a block each, came down from 15 seconds to 3 seconds on
commenting this log out.
In the previous patch there was concern that during incremental deletion, the
file inodes from the target node is still referenced in the lease and by the
blocks.
Here is the proposed change:
Step 1 - FSNamesystem lock is held for longer time here
# remove target inode from the parents. At this time, two tree traversals are
from the previous implementation will be combined to one. Computing quota used
by the subtree, deletion of inodes and collecting blocks will be done in a
single step. In this step, while collecting blocks, the inode referenced by the
block will be set to null.
Step 2: lock held for shorted duration for incremental block deletion
# Over multiple iteration, collected blocks will be deleted from block map,
corrupt map and added {{BlockManager.recentInvalidateSets}}
# Logging of addition to invalidate sets will be done outside the lock. Instead
of printing a log for every block replica, a single log will be printed for all
the replicas of a block.
> Recursively deleting a directory with millions of files makes NameNode
> unresponsive for other commands until the deletion completes
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-173
> URL: https://issues.apache.org/jira/browse/HDFS-173
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Attachments: HDFS-173.patch
>
>
> Delete a directory with millions of files. This could take several minutes
> (observed 12 mins for 9 million files). While the operation is in progress
> FSNamesystem lock is held and the requests from clients are not handled until
> deletion completes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
