[
https://issues.apache.org/jira/browse/HDFS-14657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16889763#comment-16889763
]
Chen Zhang commented on HDFS-14657:
-----------------------------------
Thanks [~xkrogen] for reminding me of the batch IBR processing feature.
This patch is initially applied to our 2.6 cluster which don't have batch IBR
processing feature, so I ignored this.
Considering batch IBR processing, I suggest we can change the
DatanodeDescriptor.reportLock to *BlockManager.blockReportLock*, and each time
before processing a FBR or IBR, acquire the blockReportLock first, other
behavior is not change.
I think this solution might be feasible for the following reasons:
# FBR and IBR could only be proceed one by one, adding a blockReportLock won't
affect this behavior
# With blockReportLock, we can release FSNameSystem's write-lock safely
# BlockReportProcessingThread must acquire the blockReportLock first before
starting bach IBR processing
In additional, the original solution using DatanodeDescriptor.reportLock allows
multiple FBR being proceed at the same time, this change may introduce some
race condition. But with blockReportLock, the behavior is almost same as before.
> Refine NameSystem lock usage during processing FBR
> --------------------------------------------------
>
> Key: HDFS-14657
> URL: https://issues.apache.org/jira/browse/HDFS-14657
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Chen Zhang
> Assignee: Chen Zhang
> Priority: Major
> Attachments: HDFS-14657-001.patch
>
>
> The disk with 12TB capacity is very normal today, which means the FBR size is
> much larger than before, Namenode holds the NameSystemLock during processing
> block report for each storage, which might take quite a long time.
> On our production environment, processing large FBR usually cause a longer
> RPC queue time, which impacts client latency, so we did some simple work on
> refining the lock usage, which improved the p99 latency significantly.
> In our solution, BlockManager release the NameSystem write lock and request
> it again for every 5000 blocks(by default) during processing FBR, with the
> fair lock, all the RPC request can be processed before BlockManager
> re-acquire the write lock.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
