[
https://issues.apache.org/jira/browse/HDFS-12545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16181305#comment-16181305
]
Ajay Kumar commented on HDFS-12545:
-----------------------------------
[~msingh] I think we should address both of those properties together.Open to
suggestions
[~shahrs87] that heuristic is from [Hadoop
Operations|http://shop.oreilly.com/product/0636920025085.do] book.
[~daryn]
bq. More threads don't always mean more throughput. Increasing the handlers
increases lock contention, context switches, gc pressure, etc.
Agree, but number of cores alone might not be enough to find the right balue. I
think cluster size is also an important factor . Good configuration NN on small
cluster might work well with default rpc handlers while same is not true for
large clusters.
bq. Rather than yet another conf key, perhaps "0" should mean autotune.
agree.
> Autotune NameNode RPC handler threads according to number of datanodes in
> cluster
> ---------------------------------------------------------------------------------
>
> Key: HDFS-12545
> URL: https://issues.apache.org/jira/browse/HDFS-12545
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Ajay Kumar
> Assignee: Ajay Kumar
>
> Autotune NameNode RPC handler threads according to number of datanodes in
> cluster.
> Currently rpc handler are controlled by {{dfs.namenode.handler.count}} on
> cluster start. Jira is to discuss best way to auto tune it according to no of
> datanodes and any other relevant input. Updating this to
> {{max(dfs.namenode.handler.count, min(200,20 * log2(number of DataNodes)))}}
> on NameNode start is one possible way. (This heuristic is from [Hadoop
> Operations|http://shop.oreilly.com/product/0636920025085.do] book.)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
