[
https://issues.apache.org/jira/browse/HDFS-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17387797#comment-17387797
]
Xing Lin edited comment on HDFS-14703 at 7/27/21, 6:07 AM:
-----------------------------------------------------------
[~daryn] Thanks for your comments. I will address your last question and leave
other questions to [~shv]. :)
Regarding the results, we used the standard NNThroughputBenchmark, with
commands like the following.
{code:java}
./bin/hadoop org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -fs
file:/// -op mkdirs -threads 200 -dirs 10000000 -dirsPerDir 512{code}
Here are a result from [~prasad-acit], since his QPS numbers are higher than
what I got.
{code:java}
BASE:
common/hadoop-hdfs-32021-05-17 11:17:36,973 INFO
namenode.NNThroughputBenchmark: — mkdirs inputs —
2021-05-17 11:17:36,973 INFO namenode.NNThroughputBenchmark: nrDirs = 1000000
2021-05-17 11:17:36,973 INFO namenode.NNThroughputBenchmark: nrThreads = 200
2021-05-17 11:17:36,973 INFO namenode.NNThroughputBenchmark: nrDirsPerDir = 32
2021-05-17 11:17:36,973 INFO namenode.NNThroughputBenchmark: — mkdirs stats —
2021-05-17 11:17:36,973 INFO namenode.NNThroughputBenchmark: # operations:
1000000
2021-05-17 11:17:36,973 INFO namenode.NNThroughputBenchmark: Elapsed Time:
17718
2021-05-17 11:17:36,973 INFO namenode.NNThroughputBenchmark: Ops per sec:
56439.77875606727
2021-05-17 11:17:36,973 INFO namenode.NNThroughputBenchmark: Average Time: 3
2021-05-17 11:17:36,973 INFO namenode.FSEditLog: Ending log segment 1, 1031254
PATCH:
2021-05-17 11:11:09,321 INFO namenode.NNThroughputBenchmark: — mkdirs inputs —
2021-05-17 11:11:09,321 INFO namenode.NNThroughputBenchmark: nrDirs = 1000000
2021-05-17 11:11:09,321 INFO namenode.NNThroughputBenchmark: nrThreads = 200
2021-05-17 11:11:09,321 INFO namenode.NNThroughputBenchmark: nrDirsPerDir = 32
2021-05-17 11:11:09,321 INFO namenode.NNThroughputBenchmark: — mkdirs stats —
2021-05-17 11:11:09,321 INFO namenode.NNThroughputBenchmark: # operations:
1000000
2021-05-17 11:11:09,321 INFO namenode.NNThroughputBenchmark: Elapsed Time:
15010
2021-05-17 11:11:09,321 INFO namenode.NNThroughputBenchmark: Ops per sec:
66622.25183211193
2021-05-17 11:11:09,321 INFO namenode.NNThroughputBenchmark: Average Time: 2
2021-05-17 11:11:09,331 INFO namenode.FSEditLog: Ending log segment 1, 1031254
{code}
was (Author: xinglin):
[~daryn] Thanks for your comments. I will address your last question and leave
other questions to [~shv]. :)
Regarding the results, we used the standard NNThroughputBenchmark, with
commands like the following.
./bin/hadoop org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark *-fs*
[*file:///*|file:///*] -op mkdirs -threads 200 -dirs 10000000 -dirsPerDir 512
Here are a result from [~prasad-acit], since his QPS numbers are higher than
what I got.
BASE:
common/hadoop-hdfs-32021-05-17 11:17:36,973 INFO
namenode.NNThroughputBenchmark: --- mkdirs inputs ---
2021-05-17 11:17:36,973 INFO namenode.NNThroughputBenchmark: nrDirs = 1000000
2021-05-17 11:17:36,973 INFO namenode.NNThroughputBenchmark: nrThreads = 200
2021-05-17 11:17:36,973 INFO namenode.NNThroughputBenchmark: nrDirsPerDir = 32
2021-05-17 11:17:36,973 INFO namenode.NNThroughputBenchmark: --- mkdirs stats
---
2021-05-17 11:17:36,973 INFO namenode.NNThroughputBenchmark: # operations:
1000000
2021-05-17 11:17:36,973 INFO namenode.NNThroughputBenchmark: Elapsed Time: 17718
2021-05-17 11:17:36,973 INFO namenode.NNThroughputBenchmark: Ops per sec:
56439.77875606727
2021-05-17 11:17:36,973 INFO namenode.NNThroughputBenchmark: Average Time: 3
2021-05-17 11:17:36,973 INFO namenode.FSEditLog: Ending log segment 1, 1031254
PATCH:
2021-05-17 11:11:09,321 INFO namenode.NNThroughputBenchmark: --- mkdirs inputs
---
2021-05-17 11:11:09,321 INFO namenode.NNThroughputBenchmark: nrDirs = 1000000
2021-05-17 11:11:09,321 INFO namenode.NNThroughputBenchmark: nrThreads = 200
2021-05-17 11:11:09,321 INFO namenode.NNThroughputBenchmark: nrDirsPerDir = 32
2021-05-17 11:11:09,321 INFO namenode.NNThroughputBenchmark: --- mkdirs stats
---
2021-05-17 11:11:09,321 INFO namenode.NNThroughputBenchmark: # operations:
1000000
2021-05-17 11:11:09,321 INFO namenode.NNThroughputBenchmark: Elapsed Time: 15010
2021-05-17 11:11:09,321 INFO namenode.NNThroughputBenchmark: Ops per sec:
66622.25183211193
2021-05-17 11:11:09,321 INFO namenode.NNThroughputBenchmark: Average Time: 2
2021-05-17 11:11:09,331 INFO namenode.FSEditLog: Ending log segment 1, 1031254
> NameNode Fine-Grained Locking via Metadata Partitioning
> -------------------------------------------------------
>
> Key: HDFS-14703
> URL: https://issues.apache.org/jira/browse/HDFS-14703
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs, namenode
> Reporter: Konstantin Shvachko
> Priority: Major
> Attachments: 001-partitioned-inodeMap-POC.tar.gz,
> 002-partitioned-inodeMap-POC.tar.gz, 003-partitioned-inodeMap-POC.tar.gz,
> NameNode Fine-Grained Locking.pdf, NameNode Fine-Grained Locking.pdf
>
>
> We target to enable fine-grained locking by splitting the in-memory namespace
> into multiple partitions each having a separate lock. Intended to improve
> performance of NameNode write operations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
