[
https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16186011#comment-16186011
]
Xiao Chen commented on HDFS-8865:
---------------------------------
hm, https://builds.apache.org/job/PreCommit-hdfs-Build/21423/ seems to confused
the branch-2.7 patch and branch-2.6 patch. I see messages saying both of them
getting applied... Removing 2.6 and retriggered, hopefully faster than looking
at why yetus failed.
bq. Xiao Chen would you please also add TestDiskspaceQuotaUpdate into the 2.6
patch?
IIRC I tried very briefly, but need other changes to make it compile, so
stopped there.
> Improve quota initialization performance
> ----------------------------------------
>
> Key: HDFS-8865
> URL: https://issues.apache.org/jira/browse/HDFS-8865
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Kihwal Lee
> Assignee: Kihwal Lee
> Fix For: 2.8.0, 3.0.0-alpha1
>
> Attachments: HDFS-8865_branch-2.6.patch, HDFS-8865_branch-2.7.patch,
> HDFS-8865.branch-2.7.patch, HDFS-8865.patch, HDFS-8865.v2.checkstyle.patch,
> HDFS-8865.v2.patch, HDFS-8865.v3.patch
>
>
> After replaying edits, the whole file system tree is recursively scanned in
> order to initialize the quota. For big name space, this can take a very long
> time. Since this is done during namenode failover, it also affects failover
> latency.
> By using the Fork-Join framework, I was able to greatly reduce the
> initialization time. The following is the test result using the fsimage from
> one of the big name nodes we have.
> || threads || seconds||
> | 1 (existing) | 55|
> | 1 (fork-join) | 68 |
> | 4 | 16 |
> | 8 | 8 |
> | 12 | 6 |
> | 16 | 5 |
> | 20 | 4 |
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
