[
https://issues.apache.org/jira/browse/HDFS-10297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HDFS-10297:
------------------------------
Attachment: HDFS-10297.001.patch
Patch 001:
* Change values in {{hdfs-default.xml}}
* Change values of {{DFS_DATANODE_BALANCE_BANDWIDTHPERSEC_DEFAULT}} and
{{DFS_DATANODE_BALANCE_MAX_NUM_CONCURRENT_MOVES_DEFAULT}}
Test output:
{noformat}
$ ( cd hadoop-hdfs-project/hadoop-hdfs ; mvn test
-Dtest=TestBalancer#testBalancer2 )
$ grep 'concurrent\.moves\|Number threads for balancing'
hadoop-hdfs-project/hadoop-hdfs/target/surefire-reports/*output*
2016-04-15 10:46:58,563 [Thread-0] INFO datanode.DataNode
(DataXceiverServer.java:<init>(79)) - Number threads for balancing is 50
2016-04-15 10:46:58,781 [Thread-0] INFO datanode.DataNode
(DataXceiverServer.java:<init>(79)) - Number threads for balancing is 50
2016-04-15 10:46:59,942 [Thread-0] INFO datanode.DataNode
(DataXceiverServer.java:<init>(79)) - Number threads for balancing is 50
2016-04-15 10:47:00,153 [Thread-0] INFO balancer.Balancer
(Balancer.java:getInt(240)) - dfs.datanode.balance.max.concurrent.moves = 50
(default=50)
2016-04-15 10:47:04,205 [Thread-0] INFO balancer.Balancer
(Balancer.java:getInt(240)) - dfs.datanode.balance.max.concurrent.moves = 50
(default=50)
[jzhuge@jzhuge-MBP hadoop](trunk *)$ grep 'bandwidthPer\|Balancing bandwidth'
hadoop-hdfs-project/hadoop-hdfs/target/surefire-reports/*output*
2016-04-15 10:46:58,563 [Thread-0] INFO datanode.DataNode
(DataXceiverServer.java:<init>(78)) - Balancing bandwidth is 10485760 bytes/s
2016-04-15 10:46:58,781 [Thread-0] INFO datanode.DataNode
(DataXceiverServer.java:<init>(78)) - Balancing bandwidth is 10485760 bytes/s
2016-04-15 10:46:59,942 [Thread-0] INFO datanode.DataNode
(DataXceiverServer.java:<init>(78)) - Balancing bandwidth is 10485760 bytes/s
{noformat}
> Increase default balance bandwidth and concurrent moves
> -------------------------------------------------------
>
> Key: HDFS-10297
> URL: https://issues.apache.org/jira/browse/HDFS-10297
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
> Affects Versions: 2.6.0
> Reporter: John Zhuge
> Assignee: John Zhuge
> Priority: Minor
> Labels: supportability
> Attachments: HDFS-10297.001.patch
>
>
> Adjust the default values to better support the current level of customer
> host and network configurations.
> Increase the default for property {{dfs.datanode.balance.bandwidthPerSec}}
> from 1 to 10 MB. Apply to DN. 10 MB/s is about 10% of the GbE network.
> Increase the default for property
> {{dfs.datanode.balance.max.concurrent.moves}} from 5 to 50. Apply to DN and
> Balancer. The default number of DN receiver threads is 4096. The default
> number of balancer mover threads is 1000.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
