[ 
https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15503755#comment-15503755
 ] 

Nicolas Fraison commented on HDFS-8865:
---------------------------------------

We have backported this patch to our 2.6 builds.
Could you please let me knwo how I can provide it, I don't have any rights to 
attach the git diff file.

> Improve quota initialization performance
> ----------------------------------------
>
>                 Key: HDFS-8865
>                 URL: https://issues.apache.org/jira/browse/HDFS-8865
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Kihwal Lee
>            Assignee: Kihwal Lee
>             Fix For: 2.8.0, 3.0.0-alpha1
>
>         Attachments: HDFS-8865.patch, HDFS-8865.v2.checkstyle.patch, 
> HDFS-8865.v2.patch, HDFS-8865.v3.patch, HDFS-8865_branch-2.7.patch
>
>
> After replaying edits, the whole file system tree is recursively scanned in 
> order to initialize the quota. For big name space, this can take a very long 
> time.  Since this is done during namenode failover, it also affects failover 
> latency.
> By using the Fork-Join framework, I was able to greatly reduce the 
> initialization time.  The following is the test result using the fsimage from 
> one of the big name nodes we have.
> || threads || seconds||
> | 1 (existing) | 55|
> | 1 (fork-join) | 68 |
> | 4 | 16 |
> | 8 | 8 |
> | 12 | 6 |
> | 16 | 5 |
> | 20 | 4 |



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

Reply via email to