[
https://issues.apache.org/jira/browse/HDFS-12545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16180960#comment-16180960
]
Daryn Sharp commented on HDFS-12545:
------------------------------------
More threads don't always mean more throughput. Increasing the handlers
increases lock contention, context switches, gc pressure, etc. If anything, the
tuning should probably be based on number of cores.
It must also be truly configurable. The proposed formula appears to force a
minimum of 200 handlers on 1k+ clusters. We run with 100 handlers even on 5k+
clusters. Rather than yet another conf key, perhaps "0" should mean autotune.
> Autotune NameNode RPC handler threads according to number of datanodes in
> cluster
> ---------------------------------------------------------------------------------
>
> Key: HDFS-12545
> URL: https://issues.apache.org/jira/browse/HDFS-12545
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Ajay Kumar
> Assignee: Ajay Kumar
>
> Autotune NameNode RPC handler threads according to number of datanodes in
> cluster. Currently rpc handler are controlled by
> {{dfs.namenode.handler.count}} on cluster start. Jira is to discuss best way
> to auto tune it according to no of datanodes. Updating this to
> {{max(dfs.namenode.handler.count, min(200,20 * log2(no of datanodes)))}} on
> NameNode start is one possible way.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
