[jira] [Updated] (HDFS-10297) Increase default balance bandwidth and concurrent moves

2016-04-27 Thread Andrew Wang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-10297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang updated HDFS-10297:
---
   Resolution: Fixed
Fix Version/s: (was: 2.9.0)
   2.8.0
   Status: Resolved  (was: Patch Available)

Pushed, thanks John!

> Increase default balance bandwidth and concurrent moves
> ---
>
> Key: HDFS-10297
> URL: https://issues.apache.org/jira/browse/HDFS-10297
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
>  Labels: supportability
> Fix For: 2.8.0
>
> Attachments: HDFS-10297.001.patch, HDFS-10297.002.patch, 
> HDFS-10297.003.branch-2.8.patch, HDFS-10297.003.patch
>
>
> Adjust the default values to better support the current level of customer 
> host and network configurations.
> Increase the default for property {{dfs.datanode.balance.bandwidthPerSec}} 
> from 1 to 10 MB. Apply to DN. 10 MB/s is about 10% of the GbE network.
> Increase the default for property 
> {{dfs.datanode.balance.max.concurrent.moves}} from 5 to 50. Apply to DN and 
> Balancer. The default number of DN receiver threads is 4096. The default 
> number of balancer mover threads is 1000.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-10297) Increase default balance bandwidth and concurrent moves

2016-04-26 Thread John Zhuge (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-10297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge updated HDFS-10297:
--
Attachment: HDFS-10297.003.branch-2.8.patch

Upload HDFS-10297.003.branch-2.8.patch for branch 2.8.

> Increase default balance bandwidth and concurrent moves
> ---
>
> Key: HDFS-10297
> URL: https://issues.apache.org/jira/browse/HDFS-10297
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
>  Labels: supportability
> Fix For: 2.9.0
>
> Attachments: HDFS-10297.001.patch, HDFS-10297.002.patch, 
> HDFS-10297.003.branch-2.8.patch, HDFS-10297.003.patch
>
>
> Adjust the default values to better support the current level of customer 
> host and network configurations.
> Increase the default for property {{dfs.datanode.balance.bandwidthPerSec}} 
> from 1 to 10 MB. Apply to DN. 10 MB/s is about 10% of the GbE network.
> Increase the default for property 
> {{dfs.datanode.balance.max.concurrent.moves}} from 5 to 50. Apply to DN and 
> Balancer. The default number of DN receiver threads is 4096. The default 
> number of balancer mover threads is 1000.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-10297) Increase default balance bandwidth and concurrent moves

2016-04-26 Thread Andrew Wang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-10297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang updated HDFS-10297:
---
Fix Version/s: 2.9.0

Committed to trunk and branch-2, branch-2.8 wasn't clean. John, if you want it 
in 2.8, do you mind providing a branch-2.8 patch? Else we can just resolve.

> Increase default balance bandwidth and concurrent moves
> ---
>
> Key: HDFS-10297
> URL: https://issues.apache.org/jira/browse/HDFS-10297
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
>  Labels: supportability
> Fix For: 2.9.0
>
> Attachments: HDFS-10297.001.patch, HDFS-10297.002.patch, 
> HDFS-10297.003.patch
>
>
> Adjust the default values to better support the current level of customer 
> host and network configurations.
> Increase the default for property {{dfs.datanode.balance.bandwidthPerSec}} 
> from 1 to 10 MB. Apply to DN. 10 MB/s is about 10% of the GbE network.
> Increase the default for property 
> {{dfs.datanode.balance.max.concurrent.moves}} from 5 to 50. Apply to DN and 
> Balancer. The default number of DN receiver threads is 4096. The default 
> number of balancer mover threads is 1000.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-10297) Increase default balance bandwidth and concurrent moves

2016-04-16 Thread John Zhuge (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-10297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge updated HDFS-10297:
--
Attachment: HDFS-10297.003.patch

Patch 003:
* Fix checkstyle

> Increase default balance bandwidth and concurrent moves
> ---
>
> Key: HDFS-10297
> URL: https://issues.apache.org/jira/browse/HDFS-10297
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
>  Labels: supportability
> Attachments: HDFS-10297.001.patch, HDFS-10297.002.patch, 
> HDFS-10297.003.patch
>
>
> Adjust the default values to better support the current level of customer 
> host and network configurations.
> Increase the default for property {{dfs.datanode.balance.bandwidthPerSec}} 
> from 1 to 10 MB. Apply to DN. 10 MB/s is about 10% of the GbE network.
> Increase the default for property 
> {{dfs.datanode.balance.max.concurrent.moves}} from 5 to 50. Apply to DN and 
> Balancer. The default number of DN receiver threads is 4096. The default 
> number of balancer mover threads is 1000.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-10297) Increase default balance bandwidth and concurrent moves

2016-04-16 Thread John Zhuge (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-10297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge updated HDFS-10297:
--
Attachment: HDFS-10297.002.patch

Patch 002:
* Fix checkstyle errors

> Increase default balance bandwidth and concurrent moves
> ---
>
> Key: HDFS-10297
> URL: https://issues.apache.org/jira/browse/HDFS-10297
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
>  Labels: supportability
> Attachments: HDFS-10297.001.patch, HDFS-10297.002.patch
>
>
> Adjust the default values to better support the current level of customer 
> host and network configurations.
> Increase the default for property {{dfs.datanode.balance.bandwidthPerSec}} 
> from 1 to 10 MB. Apply to DN. 10 MB/s is about 10% of the GbE network.
> Increase the default for property 
> {{dfs.datanode.balance.max.concurrent.moves}} from 5 to 50. Apply to DN and 
> Balancer. The default number of DN receiver threads is 4096. The default 
> number of balancer mover threads is 1000.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-10297) Increase default balance bandwidth and concurrent moves

2016-04-15 Thread Jing Zhao (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-10297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jing Zhao updated HDFS-10297:
-
Fix Version/s: (was: 2.8.0)

> Increase default balance bandwidth and concurrent moves
> ---
>
> Key: HDFS-10297
> URL: https://issues.apache.org/jira/browse/HDFS-10297
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
>  Labels: supportability
> Attachments: HDFS-10297.001.patch
>
>
> Adjust the default values to better support the current level of customer 
> host and network configurations.
> Increase the default for property {{dfs.datanode.balance.bandwidthPerSec}} 
> from 1 to 10 MB. Apply to DN. 10 MB/s is about 10% of the GbE network.
> Increase the default for property 
> {{dfs.datanode.balance.max.concurrent.moves}} from 5 to 50. Apply to DN and 
> Balancer. The default number of DN receiver threads is 4096. The default 
> number of balancer mover threads is 1000.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-10297) Increase default balance bandwidth and concurrent moves

2016-04-15 Thread John Zhuge (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-10297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge updated HDFS-10297:
--
Fix Version/s: 2.8.0
   Status: Patch Available  (was: Open)

> Increase default balance bandwidth and concurrent moves
> ---
>
> Key: HDFS-10297
> URL: https://issues.apache.org/jira/browse/HDFS-10297
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
>  Labels: supportability
> Fix For: 2.8.0
>
> Attachments: HDFS-10297.001.patch
>
>
> Adjust the default values to better support the current level of customer 
> host and network configurations.
> Increase the default for property {{dfs.datanode.balance.bandwidthPerSec}} 
> from 1 to 10 MB. Apply to DN. 10 MB/s is about 10% of the GbE network.
> Increase the default for property 
> {{dfs.datanode.balance.max.concurrent.moves}} from 5 to 50. Apply to DN and 
> Balancer. The default number of DN receiver threads is 4096. The default 
> number of balancer mover threads is 1000.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-10297) Increase default balance bandwidth and concurrent moves

2016-04-15 Thread John Zhuge (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-10297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge updated HDFS-10297:
--
Attachment: HDFS-10297.001.patch

Patch 001:
* Change values in {{hdfs-default.xml}}
* Change values of {{DFS_DATANODE_BALANCE_BANDWIDTHPERSEC_DEFAULT}} and 
{{DFS_DATANODE_BALANCE_MAX_NUM_CONCURRENT_MOVES_DEFAULT}}

Test output:
{noformat}
$ ( cd hadoop-hdfs-project/hadoop-hdfs ; mvn test 
-Dtest=TestBalancer#testBalancer2 )
$ grep 'concurrent\.moves\|Number threads for balancing' 
hadoop-hdfs-project/hadoop-hdfs/target/surefire-reports/*output*
2016-04-15 10:46:58,563 [Thread-0] INFO  datanode.DataNode 
(DataXceiverServer.java:(79)) - Number threads for balancing is 50
2016-04-15 10:46:58,781 [Thread-0] INFO  datanode.DataNode 
(DataXceiverServer.java:(79)) - Number threads for balancing is 50
2016-04-15 10:46:59,942 [Thread-0] INFO  datanode.DataNode 
(DataXceiverServer.java:(79)) - Number threads for balancing is 50
2016-04-15 10:47:00,153 [Thread-0] INFO  balancer.Balancer 
(Balancer.java:getInt(240)) - dfs.datanode.balance.max.concurrent.moves = 50 
(default=50)
2016-04-15 10:47:04,205 [Thread-0] INFO  balancer.Balancer 
(Balancer.java:getInt(240)) - dfs.datanode.balance.max.concurrent.moves = 50 
(default=50)
[jzhuge@jzhuge-MBP hadoop](trunk *)$ grep 'bandwidthPer\|Balancing bandwidth' 
hadoop-hdfs-project/hadoop-hdfs/target/surefire-reports/*output*
2016-04-15 10:46:58,563 [Thread-0] INFO  datanode.DataNode 
(DataXceiverServer.java:(78)) - Balancing bandwidth is 10485760 bytes/s
2016-04-15 10:46:58,781 [Thread-0] INFO  datanode.DataNode 
(DataXceiverServer.java:(78)) - Balancing bandwidth is 10485760 bytes/s
2016-04-15 10:46:59,942 [Thread-0] INFO  datanode.DataNode 
(DataXceiverServer.java:(78)) - Balancing bandwidth is 10485760 bytes/s
{noformat}

> Increase default balance bandwidth and concurrent moves
> ---
>
> Key: HDFS-10297
> URL: https://issues.apache.org/jira/browse/HDFS-10297
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
>  Labels: supportability
> Attachments: HDFS-10297.001.patch
>
>
> Adjust the default values to better support the current level of customer 
> host and network configurations.
> Increase the default for property {{dfs.datanode.balance.bandwidthPerSec}} 
> from 1 to 10 MB. Apply to DN. 10 MB/s is about 10% of the GbE network.
> Increase the default for property 
> {{dfs.datanode.balance.max.concurrent.moves}} from 5 to 50. Apply to DN and 
> Balancer. The default number of DN receiver threads is 4096. The default 
> number of balancer mover threads is 1000.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)