[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15503952#comment-15503952 ]
Nicolas Fraison commented on HDFS-8865: --------------------------------------- [~kihwal] I would appreciate to be added as a contributor. Even if it is not pushed to any release the backport will be available here for anyone needing it. > Improve quota initialization performance > ---------------------------------------- > > Key: HDFS-8865 > URL: https://issues.apache.org/jira/browse/HDFS-8865 > Project: Hadoop HDFS > Issue Type: Improvement > Reporter: Kihwal Lee > Assignee: Kihwal Lee > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: HDFS-8865.patch, HDFS-8865.v2.checkstyle.patch, > HDFS-8865.v2.patch, HDFS-8865.v3.patch, HDFS-8865_branch-2.7.patch > > > After replaying edits, the whole file system tree is recursively scanned in > order to initialize the quota. For big name space, this can take a very long > time. Since this is done during namenode failover, it also affects failover > latency. > By using the Fork-Join framework, I was able to greatly reduce the > initialization time. The following is the test result using the fsimage from > one of the big name nodes we have. > || threads || seconds|| > | 1 (existing) | 55| > | 1 (fork-join) | 68 | > | 4 | 16 | > | 8 | 8 | > | 12 | 6 | > | 16 | 5 | > | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org