[
https://issues.apache.org/jira/browse/HDFS-15584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17198204#comment-17198204
]
Stephen O'Donnell commented on HDFS-15584:
------------------------------------------
Even though the loop in delete drops the lock and takes it back, we do still
see large deletes slowing down the namenode badly. In other parts of the code,
I see a similar pattern where the thread sleeps a short time before acquiring
the lock again. A short sleep here is probably the best answer.
Before this patch, the thread processes 1000 blocks and then releases the lock.
Do we have an idea of how long that takes? I would think much less than 100ms.
The timer you have added includes the lock wait time as well as the time to
remove 1000 blocks. Is your idea that, if the lock is busy, the delete thread
will take a long time to get the write lock, and if that is the case, it should
sleep to reduce the time it holds the lock. However if the lock is not busy, it
will not sleep at all?
The tricky part of this, is picking a good default for
"dfs.namenode.block.deletion.lock.time.threshold".
Once the delete completes and the blocks are on pendingDeletion - do you still
see problems on the namenode, or does it work OK while Pending Deletion is
processed?
> Improve HDFS large deletion cause namenode lockqueue boom and pending
> deletion boom.
> ------------------------------------------------------------------------------------
>
> Key: HDFS-15584
> URL: https://issues.apache.org/jira/browse/HDFS-15584
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Affects Versions: 3.4.0
> Reporter: zhuqi
> Assignee: zhuqi
> Priority: Major
> Attachments: HDFS-15584.001.patch
>
>
> In our production cluster, the large deletion will boom the namenode lock
> queue, also will lead to the boom of pending deletion in invalidate blocks.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
