[
https://issues.apache.org/jira/browse/HDFS-9940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15196482#comment-15196482
]
John Zhuge commented on HDFS-9940:
----------------------------------
Balancer uses config {{dfs.datanode.balance.max.concurrent.moves}} to set field
{{Dispatcher$maxConcurrentMovesPerNode}} that sets the size of the thread pool
{{moveExecutor}} local to {{Dispatcher$executePendingMove}}.
* If the value is higher than {{dfs.datanode.balance.max.concurrent.moves}} on
the Datanode, Balancer may send more requests than DN can handle; DN will log
"Not able to copy block ... because threads quota is exceeded" and return ERROR
to Balancer. Thus some Balancer threads are wasted.
* if the value is smaller, the potential of the DN is not reached.
I can understand the original author's decision to use the same config name.
How about {{dfs.balancer.max.concurrent.moves.per.datanode}}?
> Rename dfs.balancer.max.concurrent.moves to avoid confusion
> -----------------------------------------------------------
>
> Key: HDFS-9940
> URL: https://issues.apache.org/jira/browse/HDFS-9940
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
> Affects Versions: 2.6.0
> Reporter: John Zhuge
> Assignee: John Zhuge
> Priority: Minor
> Labels: supportability
> Fix For: 2.8.0
>
>
> It is very confusing for both Balancer and Datanode to use the same property
> {{dfs.datanode.balance.max.concurrent.moves}}. It is especially so for the
> Balancer because the property has "datanode" in the name string. Many
> customers forget to set the property for the Balancer.
> Change the Balancer to use a new property
> {{dfs.balancer.max.concurrent.moves}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
