[
https://issues.apache.org/jira/browse/HDFS-7784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15796809#comment-15796809
]
Gang Xie commented on HDFS-7784:
--------------------------------
The hardware info:
CPU:
Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz with 24 Cores
Mem:
cat /proc/meminfo
MemTotal: 131749888 kB
MemFree: 9390596 kB
Buffers: 171080 kB
Cached: 23657816 kB
SwapCached: 0 kB
Active: 119711620 kB
Inactive: 381236 kB
Active(anon): 96186924 kB
Inactive(anon): 81452 kB
Active(file): 23524696 kB
Inactive(file): 299784 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 108 kB
Writeback: 0 kB
AnonPages: 96264056 kB
Mapped: 26604 kB
Shmem: 4412 kB
Slab: 728272 kB
SReclaimable: 673344 kB
SUnreclaim: 54928 kB
KernelStack: 5392 kB
PageTables: 192256 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 65874944 kB
Committed_AS: 107921484 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 488704 kB
VmallocChunk: 34289747040 kB
HardwareCorrupted: 4 kB
AnonHugePages: 90095616 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 8192 kB
DirectMap2M: 2015232 kB
DirectMap1G: 132120576 kB
And it's hdd.
> load fsimage in parallel
> ------------------------
>
> Key: HDFS-7784
> URL: https://issues.apache.org/jira/browse/HDFS-7784
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Reporter: Walter Su
> Assignee: Walter Su
> Priority: Minor
> Labels: BB2015-05-TBR
> Attachments: HDFS-7784.001.patch, test-20150213.pdf
>
>
> When single Namenode has huge amount of files, without using federation, the
> startup/restart speed is slow. The fsimage loading step takes the most of the
> time. fsimage loading can seperate to two parts, deserialization and object
> construction(mostly map insertion). Deserialization takes the most of CPU
> time. So we can do deserialization in parallel, and add to hashmap in serial.
> It will significantly reduce the NN start time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
