[
https://issues.apache.org/jira/browse/HDFS-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13858664#comment-13858664
]
Amir Langer commented on HDFS-5453:
-----------------------------------
The ASYNC performance of get block locations with 15, 16 threads is poor
because of CPU thrashing.
Remember that these results were obtained on an 8 core machine. With 16 worker
threads, 16 clients and one very important scheduler thread all competing for
the same cores - It is not that surprising that it performed so poorly.
The impact on the scheduler thread is especially critical because any time it
is swapped, it impacts the throughput of all.
On a real async service I would expect much tighter control of the resources
using patterns such as core isolation, thread affinity or at least setting a
different thread priority.
Back to this JIRA :) -
I think you're right and have linked HDFS-5477 to this JIRA.
The plan in HDFS-5477 is not to redesign the whole thing - so consistency will
still rely on locking and therefore this JIRA is required. On a different test
we conducted, fine grained locking outperformed the current global lock by
quite a margin when an (artificial) 1ms delay was introduced into a create
operation. This showed us that without it, we can not introduce an RPC call
into the locked parts of the namesystem.
> Support fine grain locking in FSNamesystem
> ------------------------------------------
>
> Key: HDFS-5453
> URL: https://issues.apache.org/jira/browse/HDFS-5453
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: namenode
> Affects Versions: 2.0.0-alpha, 3.0.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: async_simulation.xlsx
>
>
> The namesystem currently uses a course grain lock to control access. This
> prevents concurrent writers in different branches of the tree, and prevents
> readers from accessing branches that writers aren't using.
> Features that introduce latency to namesystem operations, such as cold
> storage of inodes, will need fine grain locking to avoid degrading the entire
> namesystem's throughput.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
