[
https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14720452#comment-14720452
]
Hudson commented on HDFS-8865:
------------------------------
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #317 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/317/])
HDFS-8865. Improve quota initialization performance. Contributed by Kihwal Lee.
(kihwal: rev b6ceee9bf42eec15891f60a014bbfa47e03f563c)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/QuotaCounts.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
> Improve quota initialization performance
> ----------------------------------------
>
> Key: HDFS-8865
> URL: https://issues.apache.org/jira/browse/HDFS-8865
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Kihwal Lee
> Assignee: Kihwal Lee
> Fix For: 3.0.0, 2.8.0
>
> Attachments: HDFS-8865.patch, HDFS-8865.v2.checkstyle.patch,
> HDFS-8865.v2.patch, HDFS-8865.v3.patch
>
>
> After replaying edits, the whole file system tree is recursively scanned in
> order to initialize the quota. For big name space, this can take a very long
> time. Since this is done during namenode failover, it also affects failover
> latency.
> By using the Fork-Join framework, I was able to greatly reduce the
> initialization time. The following is the test result using the fsimage from
> one of the big name nodes we have.
> || threads || seconds||
> | 1 (existing) | 55|
> | 1 (fork-join) | 68 |
> | 4 | 16 |
> | 8 | 8 |
> | 12 | 6 |
> | 16 | 5 |
> | 20 | 4 |
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
