[
https://issues.apache.org/jira/browse/HDFS-15942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen O'Donnell updated HDFS-15942:
-------------------------------------
Component/s: namenode
Target Version/s: 3.4.0
> Increase Quota initialization threads
> -------------------------------------
>
> Key: HDFS-15942
> URL: https://issues.apache.org/jira/browse/HDFS-15942
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Reporter: Stephen O'Donnell
> Assignee: Stephen O'Donnell
> Priority: Major
>
> On large namespaces, the quota initialization at started can take a long time
> with the default 4 threads. Also on NN failover, often the quota needs to be
> calculated before the failover can completed, delaying the failover.
> I performed some benchmarks some time back on a large image (316M inodes 35GB
> on disk), the quota load takes:
> {code}
> quota - 4 threads 39 seconds
> quota - 8 threads 23 seconds
> quota - 12 threads 20 seconds
> quota - 16 threads 15 seconds
> {code}
> As the quota is calculated when the NN is starting up (and hence doing no
> other work) or at failover time before the new standby becomes active, I
> think the quota should use as many threads as possible.
> I proposed we change the default to 8 or 12 on at least trunk and branch-3.3
> so we have a better default going forward.
> Has anyone got any other thoughts?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
