[jira] [Updated] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-8865: Resolution: Fixed Fix Version/s: 2.7.5 2.6.6 Status: Resolved (was: Patch Available) Thanks a lot [~yzhangal] for review. Pushed to branch-2.7 and branch-2.6. Compiled again before pushing. Also pushed the said series of jira to fix branch-2.6 test, and filed HDFS-12578 for branch-2.7. Thanks [~kihwal] [~nfraison.criteo] for contributing the patches, and [~jojochuang] for initial backport/review. > Improve quota initialization performance > > > Key: HDFS-8865 > URL: https://issues.apache.org/jira/browse/HDFS-8865 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Fix For: 2.6.6, 2.7.5, 3.0.0-alpha1, 2.8.0 > > Attachments: HDFS-8865.branch-2.6.01.patch, > HDFS-8865_branch-2.6.patch, HDFS-8865.branch-2.6.patch, > HDFS-8865_branch-2.7.patch, HDFS-8865.branch-2.7.patch, HDFS-8865.patch, > HDFS-8865.v2.checkstyle.patch, HDFS-8865.v2.patch, HDFS-8865.v3.patch > > > After replaying edits, the whole file system tree is recursively scanned in > order to initialize the quota. For big name space, this can take a very long > time. Since this is done during namenode failover, it also affects failover > latency. > By using the Fork-Join framework, I was able to greatly reduce the > initialization time. The following is the test result using the fsimage from > one of the big name nodes we have. > || threads || seconds|| > | 1 (existing) | 55| > | 1 (fork-join) | 68 | > | 4 | 16 | > | 8 | 8 | > | 12 | 6 | > | 16 | 5 | > | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-8865: Attachment: HDFS-8865.branch-2.6.01.patch Thanks [~yzhangal], attached new 2.6 patch. > Improve quota initialization performance > > > Key: HDFS-8865 > URL: https://issues.apache.org/jira/browse/HDFS-8865 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: HDFS-8865.branch-2.6.01.patch, > HDFS-8865_branch-2.6.patch, HDFS-8865.branch-2.6.patch, > HDFS-8865_branch-2.7.patch, HDFS-8865.branch-2.7.patch, HDFS-8865.patch, > HDFS-8865.v2.checkstyle.patch, HDFS-8865.v2.patch, HDFS-8865.v3.patch > > > After replaying edits, the whole file system tree is recursively scanned in > order to initialize the quota. For big name space, this can take a very long > time. Since this is done during namenode failover, it also affects failover > latency. > By using the Fork-Join framework, I was able to greatly reduce the > initialization time. The following is the test result using the fsimage from > one of the big name nodes we have. > || threads || seconds|| > | 1 (existing) | 55| > | 1 (fork-join) | 68 | > | 4 | 16 | > | 8 | 8 | > | 12 | 6 | > | 16 | 5 | > | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-8865: Attachment: (was: HDFS-8865.branch-2.7.patch) > Improve quota initialization performance > > > Key: HDFS-8865 > URL: https://issues.apache.org/jira/browse/HDFS-8865 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: HDFS-8865_branch-2.6.patch, HDFS-8865.branch-2.6.patch, > HDFS-8865_branch-2.7.patch, HDFS-8865.branch-2.7.patch, HDFS-8865.patch, > HDFS-8865.v2.checkstyle.patch, HDFS-8865.v2.patch, HDFS-8865.v3.patch > > > After replaying edits, the whole file system tree is recursively scanned in > order to initialize the quota. For big name space, this can take a very long > time. Since this is done during namenode failover, it also affects failover > latency. > By using the Fork-Join framework, I was able to greatly reduce the > initialization time. The following is the test result using the fsimage from > one of the big name nodes we have. > || threads || seconds|| > | 1 (existing) | 55| > | 1 (fork-join) | 68 | > | 4 | 16 | > | 8 | 8 | > | 12 | 6 | > | 16 | 5 | > | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-8865: Attachment: HDFS-8865.branch-2.7.patch > Improve quota initialization performance > > > Key: HDFS-8865 > URL: https://issues.apache.org/jira/browse/HDFS-8865 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: HDFS-8865_branch-2.6.patch, HDFS-8865.branch-2.6.patch, > HDFS-8865_branch-2.7.patch, HDFS-8865.branch-2.7.patch, HDFS-8865.patch, > HDFS-8865.v2.checkstyle.patch, HDFS-8865.v2.patch, HDFS-8865.v3.patch > > > After replaying edits, the whole file system tree is recursively scanned in > order to initialize the quota. For big name space, this can take a very long > time. Since this is done during namenode failover, it also affects failover > latency. > By using the Fork-Join framework, I was able to greatly reduce the > initialization time. The following is the test result using the fsimage from > one of the big name nodes we have. > || threads || seconds|| > | 1 (existing) | 55| > | 1 (fork-join) | 68 | > | 4 | 16 | > | 8 | 8 | > | 12 | 6 | > | 16 | 5 | > | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-8865: Attachment: HDFS-8865.branch-2.6.patch bq. IIRC I tried very briefly, but need other changes to make it compile, so stopped there. Turns out I was misremembering that with some internal reviews. branch-2.6 patch attached. > Improve quota initialization performance > > > Key: HDFS-8865 > URL: https://issues.apache.org/jira/browse/HDFS-8865 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: HDFS-8865_branch-2.6.patch, HDFS-8865.branch-2.6.patch, > HDFS-8865_branch-2.7.patch, HDFS-8865.branch-2.7.patch, HDFS-8865.patch, > HDFS-8865.v2.checkstyle.patch, HDFS-8865.v2.patch, HDFS-8865.v3.patch > > > After replaying edits, the whole file system tree is recursively scanned in > order to initialize the quota. For big name space, this can take a very long > time. Since this is done during namenode failover, it also affects failover > latency. > By using the Fork-Join framework, I was able to greatly reduce the > initialization time. The following is the test result using the fsimage from > one of the big name nodes we have. > || threads || seconds|| > | 1 (existing) | 55| > | 1 (fork-join) | 68 | > | 4 | 16 | > | 8 | 8 | > | 12 | 6 | > | 16 | 5 | > | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-8865: Attachment: (was: HDFS-8865.branch-2.6.patch) > Improve quota initialization performance > > > Key: HDFS-8865 > URL: https://issues.apache.org/jira/browse/HDFS-8865 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: HDFS-8865_branch-2.6.patch, HDFS-8865_branch-2.7.patch, > HDFS-8865.branch-2.7.patch, HDFS-8865.patch, HDFS-8865.v2.checkstyle.patch, > HDFS-8865.v2.patch, HDFS-8865.v3.patch > > > After replaying edits, the whole file system tree is recursively scanned in > order to initialize the quota. For big name space, this can take a very long > time. Since this is done during namenode failover, it also affects failover > latency. > By using the Fork-Join framework, I was able to greatly reduce the > initialization time. The following is the test result using the fsimage from > one of the big name nodes we have. > || threads || seconds|| > | 1 (existing) | 55| > | 1 (fork-join) | 68 | > | 4 | 16 | > | 8 | 8 | > | 12 | 6 | > | 16 | 5 | > | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-8865: Attachment: HDFS-8865.branch-2.6.patch The 2.6 patch provided by [~nfraison.criteo] still applies. Re-attaching with minor change in patch name to trigger 2.6 pre-commits. > Improve quota initialization performance > > > Key: HDFS-8865 > URL: https://issues.apache.org/jira/browse/HDFS-8865 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: HDFS-8865_branch-2.6.patch, HDFS-8865.branch-2.6.patch, > HDFS-8865_branch-2.7.patch, HDFS-8865.branch-2.7.patch, HDFS-8865.patch, > HDFS-8865.v2.checkstyle.patch, HDFS-8865.v2.patch, HDFS-8865.v3.patch > > > After replaying edits, the whole file system tree is recursively scanned in > order to initialize the quota. For big name space, this can take a very long > time. Since this is done during namenode failover, it also affects failover > latency. > By using the Fork-Join framework, I was able to greatly reduce the > initialization time. The following is the test result using the fsimage from > one of the big name nodes we have. > || threads || seconds|| > | 1 (existing) | 55| > | 1 (fork-join) | 68 | > | 4 | 16 | > | 8 | 8 | > | 12 | 6 | > | 16 | 5 | > | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-8865: Status: Patch Available (was: Reopened) > Improve quota initialization performance > > > Key: HDFS-8865 > URL: https://issues.apache.org/jira/browse/HDFS-8865 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Fix For: 3.0.0-alpha1, 2.8.0 > > Attachments: HDFS-8865_branch-2.6.patch, HDFS-8865_branch-2.7.patch, > HDFS-8865.branch-2.7.patch, HDFS-8865.patch, HDFS-8865.v2.checkstyle.patch, > HDFS-8865.v2.patch, HDFS-8865.v3.patch > > > After replaying edits, the whole file system tree is recursively scanned in > order to initialize the quota. For big name space, this can take a very long > time. Since this is done during namenode failover, it also affects failover > latency. > By using the Fork-Join framework, I was able to greatly reduce the > initialization time. The following is the test result using the fsimage from > one of the big name nodes we have. > || threads || seconds|| > | 1 (existing) | 55| > | 1 (fork-join) | 68 | > | 4 | 16 | > | 8 | 8 | > | 12 | 6 | > | 16 | 5 | > | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-8865: Attachment: HDFS-8865.branch-2.7.patch > Improve quota initialization performance > > > Key: HDFS-8865 > URL: https://issues.apache.org/jira/browse/HDFS-8865 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: HDFS-8865_branch-2.6.patch, HDFS-8865_branch-2.7.patch, > HDFS-8865.branch-2.7.patch, HDFS-8865.patch, HDFS-8865.v2.checkstyle.patch, > HDFS-8865.v2.patch, HDFS-8865.v3.patch > > > After replaying edits, the whole file system tree is recursively scanned in > order to initialize the quota. For big name space, this can take a very long > time. Since this is done during namenode failover, it also affects failover > latency. > By using the Fork-Join framework, I was able to greatly reduce the > initialization time. The following is the test result using the fsimage from > one of the big name nodes we have. > || threads || seconds|| > | 1 (existing) | 55| > | 1 (fork-join) | 68 | > | 4 | 16 | > | 8 | 8 | > | 12 | 6 | > | 16 | 5 | > | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Fraison updated HDFS-8865: -- Attachment: HDFS-8865_branch-2.6.patch Thanks [~kihwal] Patch for 2.6 branch added > Improve quota initialization performance > > > Key: HDFS-8865 > URL: https://issues.apache.org/jira/browse/HDFS-8865 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: HDFS-8865.patch, HDFS-8865.v2.checkstyle.patch, > HDFS-8865.v2.patch, HDFS-8865.v3.patch, HDFS-8865_branch-2.6.patch, > HDFS-8865_branch-2.7.patch > > > After replaying edits, the whole file system tree is recursively scanned in > order to initialize the quota. For big name space, this can take a very long > time. Since this is done during namenode failover, it also affects failover > latency. > By using the Fork-Join framework, I was able to greatly reduce the > initialization time. The following is the test result using the fsimage from > one of the big name nodes we have. > || threads || seconds|| > | 1 (existing) | 55| > | 1 (fork-join) | 68 | > | 4 | 16 | > | 8 | 8 | > | 12 | 6 | > | 16 | 5 | > | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-8865: - Attachment: HDFS-8865_branch-2.7.patch We are going to apply this to our 2.7 builds. Here is the patch, in case any one is interested. > Improve quota initialization performance > > > Key: HDFS-8865 > URL: https://issues.apache.org/jira/browse/HDFS-8865 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Fix For: 3.0.0, 2.8.0 > > Attachments: HDFS-8865.patch, HDFS-8865.v2.checkstyle.patch, > HDFS-8865.v2.patch, HDFS-8865.v3.patch, HDFS-8865_branch-2.7.patch > > > After replaying edits, the whole file system tree is recursively scanned in > order to initialize the quota. For big name space, this can take a very long > time. Since this is done during namenode failover, it also affects failover > latency. > By using the Fork-Join framework, I was able to greatly reduce the > initialization time. The following is the test result using the fsimage from > one of the big name nodes we have. > || threads || seconds|| > | 1 (existing) | 55| > | 1 (fork-join) | 68 | > | 4 | 16 | > | 8 | 8 | > | 12 | 6 | > | 16 | 5 | > | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-8865: - Hadoop Flags: Reviewed Improve quota initialization performance Key: HDFS-8865 URL: https://issues.apache.org/jira/browse/HDFS-8865 Project: Hadoop HDFS Issue Type: Improvement Reporter: Kihwal Lee Assignee: Kihwal Lee Fix For: 3.0.0, 2.8.0 Attachments: HDFS-8865.patch, HDFS-8865.v2.checkstyle.patch, HDFS-8865.v2.patch, HDFS-8865.v3.patch After replaying edits, the whole file system tree is recursively scanned in order to initialize the quota. For big name space, this can take a very long time. Since this is done during namenode failover, it also affects failover latency. By using the Fork-Join framework, I was able to greatly reduce the initialization time. The following is the test result using the fsimage from one of the big name nodes we have. || threads || seconds|| | 1 (existing) | 55| | 1 (fork-join) | 68 | | 4 | 16 | | 8 | 8 | | 12 | 6 | | 16 | 5 | | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-8865: - Status: In Progress (was: Patch Available) Improve quota initialization performance Key: HDFS-8865 URL: https://issues.apache.org/jira/browse/HDFS-8865 Project: Hadoop HDFS Issue Type: Improvement Reporter: Kihwal Lee Assignee: Kihwal Lee Attachments: HDFS-8865.patch, HDFS-8865.v2.checkstyle.patch, HDFS-8865.v2.patch, HDFS-8865.v3.patch After replaying edits, the whole file system tree is recursively scanned in order to initialize the quota. For big name space, this can take a very long time. Since this is done during namenode failover, it also affects failover latency. By using the Fork-Join framework, I was able to greatly reduce the initialization time. The following is the test result using the fsimage from one of the big name nodes we have. || threads || seconds|| | 1 (existing) | 55| | 1 (fork-join) | 68 | | 4 | 16 | | 8 | 8 | | 12 | 6 | | 16 | 5 | | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-8865: - Attachment: HDFS-8865.v3.patch Improve quota initialization performance Key: HDFS-8865 URL: https://issues.apache.org/jira/browse/HDFS-8865 Project: Hadoop HDFS Issue Type: Improvement Reporter: Kihwal Lee Assignee: Kihwal Lee Attachments: HDFS-8865.patch, HDFS-8865.v2.checkstyle.patch, HDFS-8865.v2.patch, HDFS-8865.v3.patch After replaying edits, the whole file system tree is recursively scanned in order to initialize the quota. For big name space, this can take a very long time. Since this is done during namenode failover, it also affects failover latency. By using the Fork-Join framework, I was able to greatly reduce the initialization time. The following is the test result using the fsimage from one of the big name nodes we have. || threads || seconds|| | 1 (existing) | 55| | 1 (fork-join) | 68 | | 4 | 16 | | 8 | 8 | | 12 | 6 | | 16 | 5 | | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-8865: - Attachment: HDFS-8865.v2.checkstyle.patch Missed the one checkstyle warning. Improve quota initialization performance Key: HDFS-8865 URL: https://issues.apache.org/jira/browse/HDFS-8865 Project: Hadoop HDFS Issue Type: Improvement Reporter: Kihwal Lee Assignee: Kihwal Lee Attachments: HDFS-8865.patch, HDFS-8865.v2.checkstyle.patch, HDFS-8865.v2.patch After replaying edits, the whole file system tree is recursively scanned in order to initialize the quota. For big name space, this can take a very long time. Since this is done during namenode failover, it also affects failover latency. By using the Fork-Join framework, I was able to greatly reduce the initialization time. The following is the test result using the fsimage from one of the big name nodes we have. || threads || seconds|| | 1 (existing) | 55| | 1 (fork-join) | 68 | | 4 | 16 | | 8 | 8 | | 12 | 6 | | 16 | 5 | | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-8865: - Attachment: HDFS-8865.v2.patch Improve quota initialization performance Key: HDFS-8865 URL: https://issues.apache.org/jira/browse/HDFS-8865 Project: Hadoop HDFS Issue Type: Improvement Reporter: Kihwal Lee Assignee: Kihwal Lee Attachments: HDFS-8865.patch, HDFS-8865.v2.patch After replaying edits, the whole file system tree is recursively scanned in order to initialize the quota. For big name space, this can take a very long time. Since this is done during namenode failover, it also affects failover latency. By using the Fork-Join framework, I was able to greatly reduce the initialization time. The following is the test result using the fsimage from one of the big name nodes we have. || threads || seconds|| | 1 (existing) | 55| | 1 (fork-join) | 68 | | 4 | 16 | | 8 | 8 | | 12 | 6 | | 16 | 5 | | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-8865: - Attachment: HDFS-8865.patch Improve quota initialization performance Key: HDFS-8865 URL: https://issues.apache.org/jira/browse/HDFS-8865 Project: Hadoop HDFS Issue Type: Improvement Reporter: Kihwal Lee Assignee: Kihwal Lee Attachments: HDFS-8865.patch After replaying edits, the whole file system tree is recursively scanned in order to initialize the quota. For big name space, this can take a very long time. Since this is done during namenode failover, it also affects failover latency. By using the Fork-Join framework, I was able to greatly reduce the initialization time. The following is the test result using the fsimage from one of the big name nodes we have. || threads || seconds|| | 1 (existing) | 55| | 1 (fork-join) | 68 | | 4 | 16 | | 8 | 8 | | 12 | 6 | | 16 | 5 | | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8865) Improve quota initialization performance
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-8865: - Status: Patch Available (was: Open) Improve quota initialization performance Key: HDFS-8865 URL: https://issues.apache.org/jira/browse/HDFS-8865 Project: Hadoop HDFS Issue Type: Improvement Reporter: Kihwal Lee Assignee: Kihwal Lee Attachments: HDFS-8865.patch After replaying edits, the whole file system tree is recursively scanned in order to initialize the quota. For big name space, this can take a very long time. Since this is done during namenode failover, it also affects failover latency. By using the Fork-Join framework, I was able to greatly reduce the initialization time. The following is the test result using the fsimage from one of the big name nodes we have. || threads || seconds|| | 1 (existing) | 55| | 1 (fork-join) | 68 | | 4 | 16 | | 8 | 8 | | 12 | 6 | | 16 | 5 | | 20 | 4 | -- This message was sent by Atlassian JIRA (v6.3.4#6332)
