[
https://issues.apache.org/jira/browse/HDFS-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17344975#comment-17344975
]
Renukaprasad C commented on HDFS-14703:
---------------------------------------
Thanks [~xinglin],
I tried with 8 core on laptop as well as in VM.
{code:java}
Here is my VM Configuration:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 1
Core(s) per socket: 8
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 62
Model name: Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz
Stepping: 4
CPU MHz: 3000.079
BogoMIPS: 6000.22
Hypervisor vendor: Xen
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 25600K
NUMA node0 CPU(s): 0-7
Laptop:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 2
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 142
Model name: Intel(R) Core(TM) i5-8265U CPU @ 1.60GHz
Stepping: 11
CPU MHz: 998.040
CPU max MHz: 3900.0000
CPU min MHz: 400.0000
BogoMIPS: 3600.00
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 6144K
NUMA node0 CPU(s): 0-7
{code}
I got better throughput in my VM, still Ops count with & without patch changes
remain same.
[root@00956 bin]# ./hadoop jar ./hadoop-hdfs-3.1.1-hw-ei-SNAPSHOT-tests.jar
org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -fs
hdfs://XX.XX.XX.XX:65110 -op mkdirs -threads 1000 -dirs 1000000 -dirsPerDir 128
2021-05-15 14:18:25,641 INFO namenode.NNThroughputBenchmark: Starting
benchmark: mkdirs
2021-05-15 14:18:25,682 INFO namenode.NNThroughputBenchmark: Generate 1000000
inputs for mkdirs
2021-05-15 14:18:26,209 FATAL namenode.NNThroughputBenchmark: Log level = ERROR
2021-05-15 14:18:26,298 INFO namenode.NNThroughputBenchmark: Starting 1000000
mkdirs(s) with 1000 threads.
2021-05-15 14:20:25,475 INFO namenode.NNThroughputBenchmark:
2021-05-15 14:20:25,475 INFO namenode.NNThroughputBenchmark: --- mkdirs inputs
---
2021-05-15 14:20:25,475 INFO namenode.NNThroughputBenchmark: nrDirs = 1000000
2021-05-15 14:20:25,475 INFO namenode.NNThroughputBenchmark: nrThreads = 1000
2021-05-15 14:20:25,475 INFO namenode.NNThroughputBenchmark: nrDirsPerDir = 128
2021-05-15 14:20:25,475 INFO namenode.NNThroughputBenchmark: --- mkdirs stats
---
2021-05-15 14:20:25,475 INFO namenode.NNThroughputBenchmark: # operations:
1000000
2021-05-15 14:20:25,475 INFO namenode.NNThroughputBenchmark: Elapsed Time:
118570
2021-05-15 14:20:25,476 INFO namenode.NNThroughputBenchmark: Ops per sec:
8433.836552247618
2021-05-15 14:20:25,476 INFO namenode.NNThroughputBenchmark: Average Time: 116
I will also try to test on some high end environment. Could you share me the
command you run and the partition size you have set?
> NameNode Fine-Grained Locking via Metadata Partitioning
> -------------------------------------------------------
>
> Key: HDFS-14703
> URL: https://issues.apache.org/jira/browse/HDFS-14703
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs, namenode
> Reporter: Konstantin Shvachko
> Priority: Major
> Attachments: 001-partitioned-inodeMap-POC.tar.gz,
> 002-partitioned-inodeMap-POC.tar.gz, 003-partitioned-inodeMap-POC.tar.gz,
> NameNode Fine-Grained Locking.pdf, NameNode Fine-Grained Locking.pdf
>
>
> We target to enable fine-grained locking by splitting the in-memory namespace
> into multiple partitions each having a separate lock. Intended to improve
> performance of NameNode write operations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
