[
https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14660904#comment-14660904
]
Kihwal Lee commented on HDFS-8865:
----------------------------------
While porting the patch from 2.6 to trunk, I noticed something odd in
{{FSImagte.java}}.
{code}
final EnumCounters<StorageType> typeSpaces =
new EnumCounters<StorageType>(StorageType.class);
for (StorageType t : StorageType.getTypesSupportingQuota()) {
final long typeSpace = counts.getTypeSpaces().get(t) -
parentTypeSpaces.get(t);
final long typeQuota = q.getTypeSpaces().get(t);
if (Quota.isViolated(typeQuota, typeSpace)) {
LOG.warn("Storage type quota violation in image for "
+ dir.getFullPathName()
+ " type = " + t.toString() + " quota = "
+ typeQuota + " < consumed " + typeSpace);
}
}
dir.getDirectoryWithQuotaFeature().setSpaceConsumed(namespace, ssConsumed,
typeSpaces);
{code}
{{typeSpaces}} is passed to {{setSpaceConsumed}}, but it is empty. On restart
of NN, storage type-based quota usage tracking will be reset.
> Improve quota initialization performance
> ----------------------------------------
>
> Key: HDFS-8865
> URL: https://issues.apache.org/jira/browse/HDFS-8865
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Kihwal Lee
> Assignee: Kihwal Lee
>
> After replaying edits, the whole file system tree is recursively scanned in
> order to initialize the quota. For big name space, this can take a very long
> time. Since this is done during namenode failover, it also affects failover
> latency.
> By using the Fork-Join framework, I was able to greatly reduce the
> initialization time. The following is the test result using the fsimage from
> one of the big name nodes we have.
> || threads || seconds||
> | 1 (existing) | 55|
> | 1 (fork-join) | 68 |
> | 4 | 16 |
> | 8 | 8 |
> | 12 | 6 |
> | 16 | 5 |
> | 20 | 4 |
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
