[
https://issues.apache.org/jira/browse/HDFS-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13856257#comment-13856257
]
Amir Langer commented on HDFS-5453:
-----------------------------------
Hi [~sureshms]
I worked on the async simulation with [~ebortnik] and can further explain the
results.
First the caveats / general comments:
* It is a simulation, not an implementation. The objective was just to get a
rough estimate of how much potential is there in having such an approach.
* We realise an async approach will also introduce new issues which are
avoided by the current architecture. This is a purely theoretical idea at this
stage.
* As Eddie said - It is orthogonal to this issue. We might want to move this
discussion elsewhere (or in other words... apologies to anyone who was only
interested in fine grained locking and have read so far :) )
As for the results:
BASE is the existing locking mechanism.
ASYNC is the same code without any lock or synchronisation but with added queue
and a scheduler thread that polls requests and assigns those to worker threads.
Before assigning it, the scheduler thread also resolves the path for every
request (as we assume this should form part of the decision to which worker to
assign the request - This is where it is somewhat similar to the fine grained
locking idea).
Edit logs are not written in the ASYNC simulation - The idea is to create a
radically different architecture where the requests are Asynchronous, the edit
log would be the persistent representation of the work queue and only a client
adapter would turn this into a synchronous API. (Again, above caveats apply
here)
The benchmark bypasses the RPC layer and measures the namespace alone.
The number of clients/workers represents the number of threads in BASE (Number
of clients accessing the namespace concurrently).
In ASYNC it is the number of worker threads and also the number of threads
acting as clients (Feeding the queue with requests). Remember - ASYNC also has
an extra thread acting as scheduler.
As for the GET_BLOCK_LOCATIONS results - It does benefit from the read lock to
an extent (notice a steady rise in throughput up to 4 clients which is half the
number of cores we used).
I believe it does not benefit fully from it due to two reasons:
# Other synchronisation points which synchronises all threads like -
#* FSPermissionChecker uses several bits of synchronised code especially
org.apache.hadoop.security.UserGroupInformation#getCurrentUser which seems to
frequently block threads (This may be different in other setups but that's how
our benchmark works).
#* audit logs
#* DNSToSwitchMapping#resolve drills down to synchronised code in InetAddress
# Several Background threads which periodically grab a write lock and block
all other threads
I'll be happy to discuss further / attach a patch but perhaps this is too
off-topic for this specific JIRA.
> Support fine grain locking in FSNamesystem
> ------------------------------------------
>
> Key: HDFS-5453
> URL: https://issues.apache.org/jira/browse/HDFS-5453
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: namenode
> Affects Versions: 2.0.0-alpha, 3.0.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: async_simulation.xlsx
>
>
> The namesystem currently uses a course grain lock to control access. This
> prevents concurrent writers in different branches of the tree, and prevents
> readers from accessing branches that writers aren't using.
> Features that introduce latency to namesystem operations, such as cold
> storage of inodes, will need fine grain locking to avoid degrading the entire
> namesystem's throughput.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
