[jira] [Commented] (HDFS-10297) Increase default balance bandwidth and concurrent moves
[
https://issues.apache.org/jira/browse/HDFS-10297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15259368#comment-15259368
]
John Zhuge commented on HDFS-10297:
---
[~andrew.wang], I uploaded a patch for branch-2.8, please merge.
> Increase default balance bandwidth and concurrent moves
> ---
>
> Key: HDFS-10297
> URL: https://issues.apache.org/jira/browse/HDFS-10297
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Labels: supportability
> Fix For: 2.9.0
>
> Attachments: HDFS-10297.001.patch, HDFS-10297.002.patch,
> HDFS-10297.003.branch-2.8.patch, HDFS-10297.003.patch
>
>
> Adjust the default values to better support the current level of customer
> host and network configurations.
> Increase the default for property {{dfs.datanode.balance.bandwidthPerSec}}
> from 1 to 10 MB. Apply to DN. 10 MB/s is about 10% of the GbE network.
> Increase the default for property
> {{dfs.datanode.balance.max.concurrent.moves}} from 5 to 50. Apply to DN and
> Balancer. The default number of DN receiver threads is 4096. The default
> number of balancer mover threads is 1000.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-10297) Increase default balance bandwidth and concurrent moves
[
https://issues.apache.org/jira/browse/HDFS-10297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15259009#comment-15259009
]
John Zhuge commented on HDFS-10297:
---
Thanks [~rchiang].
> Increase default balance bandwidth and concurrent moves
> ---
>
> Key: HDFS-10297
> URL: https://issues.apache.org/jira/browse/HDFS-10297
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Labels: supportability
> Fix For: 2.9.0
>
> Attachments: HDFS-10297.001.patch, HDFS-10297.002.patch,
> HDFS-10297.003.patch
>
>
> Adjust the default values to better support the current level of customer
> host and network configurations.
> Increase the default for property {{dfs.datanode.balance.bandwidthPerSec}}
> from 1 to 10 MB. Apply to DN. 10 MB/s is about 10% of the GbE network.
> Increase the default for property
> {{dfs.datanode.balance.max.concurrent.moves}} from 5 to 50. Apply to DN and
> Balancer. The default number of DN receiver threads is 4096. The default
> number of balancer mover threads is 1000.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-10297) Increase default balance bandwidth and concurrent moves
[
https://issues.apache.org/jira/browse/HDFS-10297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15259005#comment-15259005
]
Ray Chiang commented on HDFS-10297:
---
Yeah, I only did trunk and branch-2 versions for HDFS-8356. I didn't go
backwards compatible with all the branch-2 minor versions.
> Increase default balance bandwidth and concurrent moves
> ---
>
> Key: HDFS-10297
> URL: https://issues.apache.org/jira/browse/HDFS-10297
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Labels: supportability
> Fix For: 2.9.0
>
> Attachments: HDFS-10297.001.patch, HDFS-10297.002.patch,
> HDFS-10297.003.patch
>
>
> Adjust the default values to better support the current level of customer
> host and network configurations.
> Increase the default for property {{dfs.datanode.balance.bandwidthPerSec}}
> from 1 to 10 MB. Apply to DN. 10 MB/s is about 10% of the GbE network.
> Increase the default for property
> {{dfs.datanode.balance.max.concurrent.moves}} from 5 to 50. Apply to DN and
> Balancer. The default number of DN receiver threads is 4096. The default
> number of balancer mover threads is 1000.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-10297) Increase default balance bandwidth and concurrent moves
[
https://issues.apache.org/jira/browse/HDFS-10297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15258988#comment-15258988
]
John Zhuge commented on HDFS-10297:
---
[~rchiang]'s HDFS-8356 added the missing properties. Is it applicable to 2.8
and 2.7? My guess is no.
> Increase default balance bandwidth and concurrent moves
> ---
>
> Key: HDFS-10297
> URL: https://issues.apache.org/jira/browse/HDFS-10297
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Labels: supportability
> Fix For: 2.9.0
>
> Attachments: HDFS-10297.001.patch, HDFS-10297.002.patch,
> HDFS-10297.003.patch
>
>
> Adjust the default values to better support the current level of customer
> host and network configurations.
> Increase the default for property {{dfs.datanode.balance.bandwidthPerSec}}
> from 1 to 10 MB. Apply to DN. 10 MB/s is about 10% of the GbE network.
> Increase the default for property
> {{dfs.datanode.balance.max.concurrent.moves}} from 5 to 50. Apply to DN and
> Balancer. The default number of DN receiver threads is 4096. The default
> number of balancer mover threads is 1000.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-10297) Increase default balance bandwidth and concurrent moves
[
https://issues.apache.org/jira/browse/HDFS-10297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15258978#comment-15258978
]
John Zhuge commented on HDFS-10297:
---
Wow, so many properties of file {{hdfs-default.xml}} in branch-2 (line 2972 to
3984) are NOT in branch-2.8 or 2.7 including
{{dfs.datanode.balance.max.concurrent.moves}}?!
> Increase default balance bandwidth and concurrent moves
> ---
>
> Key: HDFS-10297
> URL: https://issues.apache.org/jira/browse/HDFS-10297
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Labels: supportability
> Fix For: 2.9.0
>
> Attachments: HDFS-10297.001.patch, HDFS-10297.002.patch,
> HDFS-10297.003.patch
>
>
> Adjust the default values to better support the current level of customer
> host and network configurations.
> Increase the default for property {{dfs.datanode.balance.bandwidthPerSec}}
> from 1 to 10 MB. Apply to DN. 10 MB/s is about 10% of the GbE network.
> Increase the default for property
> {{dfs.datanode.balance.max.concurrent.moves}} from 5 to 50. Apply to DN and
> Balancer. The default number of DN receiver threads is 4096. The default
> number of balancer mover threads is 1000.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-10297) Increase default balance bandwidth and concurrent moves
[
https://issues.apache.org/jira/browse/HDFS-10297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15258924#comment-15258924
]
John Zhuge commented on HDFS-10297:
---
Thank you [~andrew.wang]. I will create a patch for 2.8.
> Increase default balance bandwidth and concurrent moves
> ---
>
> Key: HDFS-10297
> URL: https://issues.apache.org/jira/browse/HDFS-10297
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Labels: supportability
> Fix For: 2.9.0
>
> Attachments: HDFS-10297.001.patch, HDFS-10297.002.patch,
> HDFS-10297.003.patch
>
>
> Adjust the default values to better support the current level of customer
> host and network configurations.
> Increase the default for property {{dfs.datanode.balance.bandwidthPerSec}}
> from 1 to 10 MB. Apply to DN. 10 MB/s is about 10% of the GbE network.
> Increase the default for property
> {{dfs.datanode.balance.max.concurrent.moves}} from 5 to 50. Apply to DN and
> Balancer. The default number of DN receiver threads is 4096. The default
> number of balancer mover threads is 1000.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-10297) Increase default balance bandwidth and concurrent moves
[
https://issues.apache.org/jira/browse/HDFS-10297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15258859#comment-15258859
]
Hudson commented on HDFS-10297:
---
FAILURE: Integrated in Hadoop-trunk-Commit #9672 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/9672/])
HDFS-10297. Increase default balance bandwidth and concurrent moves. (wang: rev
6be22ddbf1b6f3c19ec70c63ddb8f5519d18dd72)
* hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
> Increase default balance bandwidth and concurrent moves
> ---
>
> Key: HDFS-10297
> URL: https://issues.apache.org/jira/browse/HDFS-10297
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Labels: supportability
> Fix For: 2.9.0
>
> Attachments: HDFS-10297.001.patch, HDFS-10297.002.patch,
> HDFS-10297.003.patch
>
>
> Adjust the default values to better support the current level of customer
> host and network configurations.
> Increase the default for property {{dfs.datanode.balance.bandwidthPerSec}}
> from 1 to 10 MB. Apply to DN. 10 MB/s is about 10% of the GbE network.
> Increase the default for property
> {{dfs.datanode.balance.max.concurrent.moves}} from 5 to 50. Apply to DN and
> Balancer. The default number of DN receiver threads is 4096. The default
> number of balancer mover threads is 1000.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-10297) Increase default balance bandwidth and concurrent moves
[
https://issues.apache.org/jira/browse/HDFS-10297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15258821#comment-15258821
]
Andrew Wang commented on HDFS-10297:
LGTM +1, will commit shortly.
> Increase default balance bandwidth and concurrent moves
> ---
>
> Key: HDFS-10297
> URL: https://issues.apache.org/jira/browse/HDFS-10297
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Labels: supportability
> Attachments: HDFS-10297.001.patch, HDFS-10297.002.patch,
> HDFS-10297.003.patch
>
>
> Adjust the default values to better support the current level of customer
> host and network configurations.
> Increase the default for property {{dfs.datanode.balance.bandwidthPerSec}}
> from 1 to 10 MB. Apply to DN. 10 MB/s is about 10% of the GbE network.
> Increase the default for property
> {{dfs.datanode.balance.max.concurrent.moves}} from 5 to 50. Apply to DN and
> Balancer. The default number of DN receiver threads is 4096. The default
> number of balancer mover threads is 1000.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-10297) Increase default balance bandwidth and concurrent moves
[
https://issues.apache.org/jira/browse/HDFS-10297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15251006#comment-15251006
]
John Zhuge commented on HDFS-10297:
---
Could someone kindly review this simple patch?
> Increase default balance bandwidth and concurrent moves
> ---
>
> Key: HDFS-10297
> URL: https://issues.apache.org/jira/browse/HDFS-10297
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Labels: supportability
> Attachments: HDFS-10297.001.patch, HDFS-10297.002.patch,
> HDFS-10297.003.patch
>
>
> Adjust the default values to better support the current level of customer
> host and network configurations.
> Increase the default for property {{dfs.datanode.balance.bandwidthPerSec}}
> from 1 to 10 MB. Apply to DN. 10 MB/s is about 10% of the GbE network.
> Increase the default for property
> {{dfs.datanode.balance.max.concurrent.moves}} from 5 to 50. Apply to DN and
> Balancer. The default number of DN receiver threads is 4096. The default
> number of balancer mover threads is 1000.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-10297) Increase default balance bandwidth and concurrent moves
[
https://issues.apache.org/jira/browse/HDFS-10297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15244475#comment-15244475
]
Hadoop QA commented on HDFS-10297:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
9s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 53s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 0s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 14s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 55s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s {color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: patch generated 0
new + 392 unchanged - 3 fixed = 392 total (was 395) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 0s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 14s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 50s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 90m 51s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_77. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 78m 59s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 197m 1s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_77 Failed junit tests |
hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl |
| | hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead |
| | hadoop.hdfs.TestFileAppend |
| | hadoop.hdfs.server.datanode.TestDataNodeMX
[jira] [Commented] (HDFS-10297) Increase default balance bandwidth and concurrent moves
[
https://issues.apache.org/jira/browse/HDFS-10297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15244213#comment-15244213
]
Hadoop QA commented on HDFS-10297:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 24s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
18s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 48s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 18s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 12s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
39s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 20s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 21s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 46s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 46s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 17s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 17s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 39s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs: patch generated 1 new +
392 unchanged - 3 fixed = 393 total (was 395) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 11s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 19s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 54s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 110m 57s
{color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_77.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 84m 45s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 235m 15s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_77 Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeUUID |
| | hadoop.hdfs.TestDFSUpgradeFromImage |
| | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.hdfs.TestPersistBlocks |
| | hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead |
| | hadoop.hdfs.
[jira] [Commented] (HDFS-10297) Increase default balance bandwidth and concurrent moves
[
https://issues.apache.org/jira/browse/HDFS-10297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15244108#comment-15244108
]
John Zhuge commented on HDFS-10297:
---
Unit test failures look unrelated.
> Increase default balance bandwidth and concurrent moves
> ---
>
> Key: HDFS-10297
> URL: https://issues.apache.org/jira/browse/HDFS-10297
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Labels: supportability
> Attachments: HDFS-10297.001.patch, HDFS-10297.002.patch
>
>
> Adjust the default values to better support the current level of customer
> host and network configurations.
> Increase the default for property {{dfs.datanode.balance.bandwidthPerSec}}
> from 1 to 10 MB. Apply to DN. 10 MB/s is about 10% of the GbE network.
> Increase the default for property
> {{dfs.datanode.balance.max.concurrent.moves}} from 5 to 50. Apply to DN and
> Balancer. The default number of DN receiver threads is 4096. The default
> number of balancer mover threads is 1000.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-10297) Increase default balance bandwidth and concurrent moves
[
https://issues.apache.org/jira/browse/HDFS-10297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15243691#comment-15243691
]
Hadoop QA commented on HDFS-10297:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
53s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 51s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 0s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 46s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 23s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs: patch generated 2 new +
393 unchanged - 2 fixed = 395 total (was 395) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 43s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 56m 42s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_77. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 53m 18s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 135m 38s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_77 Failed junit tests | hadoop.hdfs.TestHFlush |
| | hadoop.hdfs.server.namenode.TestEditLog |
| | hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead |
| JDK v1.7.0_95 Failed junit tests |
hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead |
\\
\\
|| Subsystem || Report/Notes ||
| Docker
[jira] [Commented] (HDFS-10297) Increase default balance bandwidth and concurrent moves
[
https://issues.apache.org/jira/browse/HDFS-10297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15243522#comment-15243522
]
John Zhuge commented on HDFS-10297:
---
Thanks [~jingzhao].
> Increase default balance bandwidth and concurrent moves
> ---
>
> Key: HDFS-10297
> URL: https://issues.apache.org/jira/browse/HDFS-10297
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Labels: supportability
> Attachments: HDFS-10297.001.patch
>
>
> Adjust the default values to better support the current level of customer
> host and network configurations.
> Increase the default for property {{dfs.datanode.balance.bandwidthPerSec}}
> from 1 to 10 MB. Apply to DN. 10 MB/s is about 10% of the GbE network.
> Increase the default for property
> {{dfs.datanode.balance.max.concurrent.moves}} from 5 to 50. Apply to DN and
> Balancer. The default number of DN receiver threads is 4096. The default
> number of balancer mover threads is 1000.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-10297) Increase default balance bandwidth and concurrent moves
[
https://issues.apache.org/jira/browse/HDFS-10297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15243340#comment-15243340
]
Jing Zhao commented on HDFS-10297:
--
[~jzhuge], thanks for reporting the issue and providing a patch. I noticed you
set the "Fix Version" to 2.8.0. We usually update "Fix Version" only after the
patch has been committed. So currently I removed that field for you.
> Increase default balance bandwidth and concurrent moves
> ---
>
> Key: HDFS-10297
> URL: https://issues.apache.org/jira/browse/HDFS-10297
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Labels: supportability
> Attachments: HDFS-10297.001.patch
>
>
> Adjust the default values to better support the current level of customer
> host and network configurations.
> Increase the default for property {{dfs.datanode.balance.bandwidthPerSec}}
> from 1 to 10 MB. Apply to DN. 10 MB/s is about 10% of the GbE network.
> Increase the default for property
> {{dfs.datanode.balance.max.concurrent.moves}} from 5 to 50. Apply to DN and
> Balancer. The default number of DN receiver threads is 4096. The default
> number of balancer mover threads is 1000.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
