Kihwal Lee created HDFS-8865:
--------------------------------
Summary: Improve quota initialization performance
Key: HDFS-8865
URL: https://issues.apache.org/jira/browse/HDFS-8865
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Kihwal Lee
After replaying edits, the whole file system tree is recursively scanned in
order to initialize the quota. For big name space, this can take a very long
time. Since this is done during namenode failover, it also affects failover
latency.
By using the Fork-Join framework, I was able to greatly reduce the
initialization time. The following is the test result using the fsimage from
one of the big name nodes we have.
|| threads || seconds||
| 1 (existing) | 55|
| 1 (fork-join) | 68 |
| 4 | 16 |
| 8 | 8 |
| 12 | 6 |
| 16 | 5 |
| 20 | 4 |
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
