[
https://issues.apache.org/jira/browse/HDFS-4995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13709012#comment-13709012
]
Kihwal Lee commented on HDFS-4995:
----------------------------------
Here is my first cut of idea.
If the result of these commands don't need to be strictly consistent, we can
probably avoid potentially long lock region.
- Lock directory when obtaining a copy of children list (ReadOnlyList).
- Lock namesystem when computing in INodeFile.
- Handle stale children in the depth first traversing.
Recursion may need to be done at FSDirectory-level. This also makes it easy to
ensure the namespace lock is not acquired while the directory lock is held. If
we do this at the finest granularity, lock contention will increase and at
minimum getContentSummary() will take considerably longer. We may internally
batch (increased lock granularity) to reduce the impact. In this case,
processing of a batch needs to be in the same namespace lock. Locking may look
less tangled if the recursion is in FSNamesystem.
> Make getContentSummary() less expensive
> ---------------------------------------
>
> Key: HDFS-4995
> URL: https://issues.apache.org/jira/browse/HDFS-4995
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
> Affects Versions: 0.23.9, 2.3.0
> Reporter: Kihwal Lee
>
> When users call du or count DFS command, getContentSummary() method is called
> against namenode. If the directory has many directories and files, it could
> hold the namesystem lock for a long time. We've seen it taking over 20
> seconds. Namenode should not allow regular users to cause extended locking.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
