[
https://issues.apache.org/jira/browse/HDFS-14657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16888936#comment-16888936
]
Erik Krogen commented on HDFS-14657:
------------------------------------
This seems like a nice middle ground between the current behavior and
HDFS-11313, which is a larger development effort. In one of our environments,
we ended up having to make other unpleasant hacks to get around this issue in
lieu of HDFS-11313 being completed.
I haven't yet thought deeply about the patch, but one thing stood out to me so
far. Within {{BlockManager#processIncrementalBlockReport}}, previously we just
confirmed that the lock was held, now we release the lock and re-acquire it
(twice). IIRC, currently there is behavior within the IBR processing logic to
batch many IBRs within the same write lock acquisition, to decrease the
overhead of locking on each IBR. So before, we had something like one lock
acquisition per 1000 IBRs, now we have 2 lock acquisitions per IBR. I'm
wondering if this will introduce undesirable overheads?
> Refine NameSystem lock usage during processing FBR
> --------------------------------------------------
>
> Key: HDFS-14657
> URL: https://issues.apache.org/jira/browse/HDFS-14657
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Chen Zhang
> Assignee: Chen Zhang
> Priority: Major
> Attachments: HDFS-14657-001.patch
>
>
> The disk with 12TB capacity is very normal today, which means the FBR size is
> much larger than before, Namenode holds the NameSystemLock during processing
> block report for each storage, which might take quite a long time.
> On our production environment, processing large FBR usually cause a longer
> RPC queue time, which impacts client latency, so we did some simple work on
> refining the lock usage, which improved the p99 latency significantly.
> In our solution, BlockManager release the NameSystem write lock and request
> it again for every 5000 blocks(by default) during processing FBR, with the
> fair lock, all the RPC request can be processed before BlockManager
> re-acquire the write lock.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
