[
https://issues.apache.org/jira/browse/HDFS-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13837875#comment-13837875
]
Daryn Sharp commented on HDFS-5453:
-----------------------------------
[~sureshms] Sorry for the delay in response, I've been on vacation. The
initial simplistic implementation, w/o any fsdir lock changes, and with direct
fsn access (no RPC), has a modest throughput improvement of ~2-15% depending on
various read/write workloads of only listStatus/mkdir/delete with an ideal
scenario of low path contention in the namesystem. In practice other
subsystems unnecessarily write locking the namesystem will probably negate most
gains until they are addressed too.
A gain is achieved if handler threads have passed through the fsn lock(s),
resolved their path, checked permissions, and are blocked on the fsdir lock -
as opposed to all read/write handlers being blocked on the global fsn lock
during any write op.
I don't have the numbers handy, but with complete removal of the fsdir lock
(not yet feasible due the non-thread safe datastructures it protects) and
desync of a few other methods such as UGI.getCurrentUser produced a multiplier
of throughput.
At the moment, I only intend to lay the groundwork for larger changes.
> Support fine grain locking in FSNamesystem
> ------------------------------------------
>
> Key: HDFS-5453
> URL: https://issues.apache.org/jira/browse/HDFS-5453
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: namenode
> Affects Versions: 2.0.0-alpha, 3.0.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
>
> The namesystem currently uses a course grain lock to control access. This
> prevents concurrent writers in different branches of the tree, and prevents
> readers from accessing branches that writers aren't using.
> Features that introduce latency to namesystem operations, such as cold
> storage of inodes, will need fine grain locking to avoid degrading the entire
> namesystem's throughput.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
