[
https://issues.apache.org/jira/browse/HDFS-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907662#comment-16907662
]
Konstantin Shvachko commented on HDFS-14703:
--------------------------------------------
Attaching the POC patch. It consists of 4 commits. Apply using {{git am
001-partitioned-inodeMap-POC/*}} command.
# 0001 patch is an investigation to verify that FSN lock is used together with
dirLock. I just ran unit tests with this patch. Most of them pass the
verification, but some don't.
# 0002 patch disables dirLock.
# 0003 introduces PartitionedGSet, LatchLock. It implements dynamic
partitioning based on inodeId key (see INodeIdComparator)
# 0004 introduces two-level key and implements static partitioning based on
that key.
With 0003 and 0004 patches I ran NNThroughputBenchmark creating 2 million
directories with 200 concurrent threads and 128 subdirectories.
So the POC implements new locking for one operation only - mkdir.
The benchmark command: {{NNThroughputBenchmark -fs file:/// -op mkdirs -threads
200 -dirs 2000000 -dirsPerDir 128}}
> NameNode Fine-Grained Locking via Metadata Partitioning
> -------------------------------------------------------
>
> Key: HDFS-14703
> URL: https://issues.apache.org/jira/browse/HDFS-14703
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs, namenode
> Reporter: Konstantin Shvachko
> Priority: Major
> Attachments: 001-partitioned-inodeMap-POC.tar.gz, NameNode
> Fine-Grained Locking.pdf
>
>
> We target to enable fine-grained locking by splitting the in-memory namespace
> into multiple partitions each having a separate lock. Intended to improve
> performance of NameNode write operations.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
