[jira] [Commented] (HDFS-13188) Disk Balancer: Support multiple block pools during block move
[ https://issues.apache.org/jira/browse/HDFS-13188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388203#comment-16388203 ] Bharat Viswanadham commented on HDFS-13188: --- Thank You [~elgoiri] for review and committing the patch. > Disk Balancer: Support multiple block pools during block move > - > > Key: HDFS-13188 > URL: https://issues.apache.org/jira/browse/HDFS-13188 > Project: Hadoop HDFS > Issue Type: Bug > Components: diskbalancer >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Fix For: 3.2.0 > > Attachments: HDFS-13188.01.patch, HDFS-13188.02.patch, > HDFS-13188.03.patch, HDFS-13188.04.patch, HDFS-13188.05.patch > > > During execute plan: > *Federated setup:* > When multiple block pools are there, it will only copy from blocks from first > block pool to destination disk, when balancing. > We want to distribute the blocks from all block pools on source disk to > destination disk during balancing. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13188) Disk Balancer: Support multiple block pools during block move
[ https://issues.apache.org/jira/browse/HDFS-13188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388147#comment-16388147 ] Hudson commented on HDFS-13188: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13779 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13779/]) HDFS-13188. Disk Balancer: Support multiple block pools during block (inigoiri: rev 7060725662cb3317ff2f0fcc38f965fd23e8e6aa) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/diskbalancer/DiskBalancerTestUtil.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DiskBalancer.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/diskbalancer/TestDiskBalancer.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/diskbalancer/TestDiskBalancerRPC.java > Disk Balancer: Support multiple block pools during block move > - > > Key: HDFS-13188 > URL: https://issues.apache.org/jira/browse/HDFS-13188 > Project: Hadoop HDFS > Issue Type: Bug > Components: diskbalancer >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Fix For: 3.2.0 > > Attachments: HDFS-13188.01.patch, HDFS-13188.02.patch, > HDFS-13188.03.patch, HDFS-13188.04.patch, HDFS-13188.05.patch > > > During execute plan: > *Federated setup:* > When multiple block pools are there, it will only copy from blocks from first > block pool to destination disk, when balancing. > We want to distribute the blocks from all block pools on source disk to > destination disk during balancing. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13188) Disk Balancer: Support multiple block pools during block move
[ https://issues.apache.org/jira/browse/HDFS-13188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388113#comment-16388113 ] Íñigo Goiri commented on HDFS-13188: Thanks [~bharatviswa] for the ping, I had forgotten. Committed to trunk. > Disk Balancer: Support multiple block pools during block move > - > > Key: HDFS-13188 > URL: https://issues.apache.org/jira/browse/HDFS-13188 > Project: Hadoop HDFS > Issue Type: Bug > Components: diskbalancer >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Fix For: 3.2.0 > > Attachments: HDFS-13188.01.patch, HDFS-13188.02.patch, > HDFS-13188.03.patch, HDFS-13188.04.patch, HDFS-13188.05.patch > > > During execute plan: > *Federated setup:* > When multiple block pools are there, it will only copy from blocks from first > block pool to destination disk, when balancing. > We want to distribute the blocks from all block pools on source disk to > destination disk during balancing. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13188) Disk Balancer: Support multiple block pools during block move
[ https://issues.apache.org/jira/browse/HDFS-13188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387260#comment-16387260 ] Bharat Viswanadham commented on HDFS-13188: --- [~elgoiri] Could you help in committing the patch? > Disk Balancer: Support multiple block pools during block move > - > > Key: HDFS-13188 > URL: https://issues.apache.org/jira/browse/HDFS-13188 > Project: Hadoop HDFS > Issue Type: Bug > Components: diskbalancer >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HDFS-13188.01.patch, HDFS-13188.02.patch, > HDFS-13188.03.patch, HDFS-13188.04.patch, HDFS-13188.05.patch > > > During execute plan: > *Federated setup:* > When multiple block pools are there, it will only copy from blocks from first > block pool to destination disk, when balancing. > We want to distribute the blocks from all block pools on source disk to > destination disk during balancing. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13188) Disk Balancer: Support multiple block pools during block move
[
https://issues.apache.org/jira/browse/HDFS-13188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377982#comment-16377982
]
Íñigo Goiri commented on HDFS-13188:
Thanks [~bharatviswa] for [^HDFS-13188.05.patch].
If nobody has anymore comments, I'll commit to {{trunk}} tomorrow.
> Disk Balancer: Support multiple block pools during block move
> -
>
> Key: HDFS-13188
> URL: https://issues.apache.org/jira/browse/HDFS-13188
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: diskbalancer
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: HDFS-13188.01.patch, HDFS-13188.02.patch,
> HDFS-13188.03.patch, HDFS-13188.04.patch, HDFS-13188.05.patch
>
>
> During execute plan:
> *Federated setup:*
> When multiple block pools are there, it will only copy from blocks from first
> block pool to destination disk, when balancing.
> We want to distribute the blocks from all block pools on source disk to
> destination disk during balancing.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13188) Disk Balancer: Support multiple block pools during block move
[
https://issues.apache.org/jira/browse/HDFS-13188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377910#comment-16377910
]
genericqa commented on HDFS-13188:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
46s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 4s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
55s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 54s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}119m 29s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}167m 25s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13188 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12912156/HDFS-13188.05.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux eacd288aff3c 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 6ce9f79 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HDFS-Build/23208/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/23208/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/23208/testReport/ |
| Max. process+thread count | 4101 (vs. ulimit of 1)
[jira] [Commented] (HDFS-13188) Disk Balancer: Support multiple block pools during block move
[ https://issues.apache.org/jira/browse/HDFS-13188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377857#comment-16377857 ] Bharat Viswanadham commented on HDFS-13188: --- [~elgoiri] Thanks for info and review. > Disk Balancer: Support multiple block pools during block move > - > > Key: HDFS-13188 > URL: https://issues.apache.org/jira/browse/HDFS-13188 > Project: Hadoop HDFS > Issue Type: Bug > Components: diskbalancer >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HDFS-13188.01.patch, HDFS-13188.02.patch, > HDFS-13188.03.patch, HDFS-13188.04.patch, HDFS-13188.05.patch > > > During execute plan: > *Federated setup:* > When multiple block pools are there, it will only copy from blocks from first > block pool to destination disk, when balancing. > We want to distribute the blocks from all block pools on source disk to > destination disk during balancing. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13188) Disk Balancer: Support multiple block pools during block move
[
https://issues.apache.org/jira/browse/HDFS-13188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377848#comment-16377848
]
genericqa commented on HDFS-13188:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 0s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 27s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}130m 14s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}179m 19s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13188 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12912128/HDFS-13188.04.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 9889e5056cea 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / e851881 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/23207/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/23207/testReport/ |
| Max. process+thread count | 3665 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
[jira] [Commented] (HDFS-13188) Disk Balancer: Support multiple block pools during block move
[ https://issues.apache.org/jira/browse/HDFS-13188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377827#comment-16377827 ] Íñigo Goiri commented on HDFS-13188: bq. Just want to ask, we have line limit length to 80, is there any requirement that if it is less than 80, we should fit it in a single line? No requirement but I personally think it's good practice to keep the code as clean as possible. Obviously there are plenty of exceptions and Yetus doesn't even mark it as checkstyle warning. The unit tests are good in my opinion; I'll wait for Yetus just in case. +1 on [^HDFS-13188.05.patch]. If nobody has issues I'll commit this. > Disk Balancer: Support multiple block pools during block move > - > > Key: HDFS-13188 > URL: https://issues.apache.org/jira/browse/HDFS-13188 > Project: Hadoop HDFS > Issue Type: Bug > Components: diskbalancer >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HDFS-13188.01.patch, HDFS-13188.02.patch, > HDFS-13188.03.patch, HDFS-13188.04.patch, HDFS-13188.05.patch > > > During execute plan: > *Federated setup:* > When multiple block pools are there, it will only copy from blocks from first > block pool to destination disk, when balancing. > We want to distribute the blocks from all block pools on source disk to > destination disk during balancing. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13188) Disk Balancer: Support multiple block pools during block move
[ https://issues.apache.org/jira/browse/HDFS-13188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377794#comment-16377794 ] Bharat Viswanadham commented on HDFS-13188: --- Addressed [~elgoiri] Just want to ask, we have line limit length to 80, is there any requirement that if it is less than 80, we should fit it in a single line? > Disk Balancer: Support multiple block pools during block move > - > > Key: HDFS-13188 > URL: https://issues.apache.org/jira/browse/HDFS-13188 > Project: Hadoop HDFS > Issue Type: Bug > Components: diskbalancer >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HDFS-13188.01.patch, HDFS-13188.02.patch, > HDFS-13188.03.patch, HDFS-13188.04.patch, HDFS-13188.05.patch > > > During execute plan: > *Federated setup:* > When multiple block pools are there, it will only copy from blocks from first > block pool to destination disk, when balancing. > We want to distribute the blocks from all block pools on source disk to > destination disk during balancing. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13188) Disk Balancer: Support multiple block pools during block move
[
https://issues.apache.org/jira/browse/HDFS-13188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377732#comment-16377732
]
Íñigo Goiri commented on HDFS-13188:
Thanks [~bharatviswa]. For [^HDFS-13188.04.patch], {{TestDiskBalancer:678}}
should fit in one line.
> Disk Balancer: Support multiple block pools during block move
> -
>
> Key: HDFS-13188
> URL: https://issues.apache.org/jira/browse/HDFS-13188
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: diskbalancer
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: HDFS-13188.01.patch, HDFS-13188.02.patch,
> HDFS-13188.03.patch, HDFS-13188.04.patch
>
>
> During execute plan:
> *Federated setup:*
> When multiple block pools are there, it will only copy from blocks from first
> block pool to destination disk, when balancing.
> We want to distribute the blocks from all block pools on source disk to
> destination disk during balancing.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13188) Disk Balancer: Support multiple block pools during block move
[ https://issues.apache.org/jira/browse/HDFS-13188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377619#comment-16377619 ] Bharat Viswanadham commented on HDFS-13188: --- Hi [~elgoiri] Addressed your review comments in patch v04. Imported the hadoop checkstyle rules in to intellij, which will do break lines according to hadoop rules. And also as some of the code is just modified, the code was like that before, i have not changed. > Disk Balancer: Support multiple block pools during block move > - > > Key: HDFS-13188 > URL: https://issues.apache.org/jira/browse/HDFS-13188 > Project: Hadoop HDFS > Issue Type: Bug > Components: diskbalancer >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HDFS-13188.01.patch, HDFS-13188.02.patch, > HDFS-13188.03.patch, HDFS-13188.04.patch > > > During execute plan: > *Federated setup:* > When multiple block pools are there, it will only copy from blocks from first > block pool to destination disk, when balancing. > We want to distribute the blocks from all block pools on source disk to > destination disk during balancing. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13188) Disk Balancer: Support multiple block pools during block move
[
https://issues.apache.org/jira/browse/HDFS-13188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16376294#comment-16376294
]
genericqa commented on HDFS-13188:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
35s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 40s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 52s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}112m 30s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}174m 20s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13188 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12911966/HDFS-13188.03.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 7e6a2b16b8e5 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 2fa7963 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/23196/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/23196/testReport/ |
| Max. process+thread count | 3361 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/23196/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT
[jira] [Commented] (HDFS-13188) Disk Balancer: Support multiple block pools during block move
[
https://issues.apache.org/jira/browse/HDFS-13188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16376230#comment-16376230
]
Íñigo Goiri commented on HDFS-13188:
Thanks [~bharatviswa]. In [^HDFS-13188.03.patch] the following break lines in
{{TestDiskBalancer}}:
* 659
* 662
* 681
{{DiskBalancerTestUtil}}:
* 336
* 340
Other than that +1.
> Disk Balancer: Support multiple block pools during block move
> -
>
> Key: HDFS-13188
> URL: https://issues.apache.org/jira/browse/HDFS-13188
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: diskbalancer
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: HDFS-13188.01.patch, HDFS-13188.02.patch,
> HDFS-13188.03.patch
>
>
> During execute plan:
> *Federated setup:*
> When multiple block pools are there, it will only copy from blocks from first
> block pool to destination disk, when balancing.
> We want to distribute the blocks from all block pools on source disk to
> destination disk during balancing.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13188) Disk Balancer: Support multiple block pools during block move
[ https://issues.apache.org/jira/browse/HDFS-13188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16376227#comment-16376227 ] Bharat Viswanadham commented on HDFS-13188: --- Hi [~goiri] Thanks for review. Addressed review comments in patch v03. > Disk Balancer: Support multiple block pools during block move > - > > Key: HDFS-13188 > URL: https://issues.apache.org/jira/browse/HDFS-13188 > Project: Hadoop HDFS > Issue Type: Bug > Components: diskbalancer >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HDFS-13188.01.patch, HDFS-13188.02.patch, > HDFS-13188.03.patch > > > During execute plan: > *Federated setup:* > When multiple block pools are there, it will only copy from blocks from first > block pool to destination disk, when balancing. > We want to distribute the blocks from all block pools on source disk to > destination disk during balancing. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13188) Disk Balancer: Support multiple block pools during block move
[
https://issues.apache.org/jira/browse/HDFS-13188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16376200#comment-16376200
]
Íñigo Goiri commented on HDFS-13188:
The unit test seem to
[pass|https://builds.apache.org/job/PreCommit-HDFS-Build/23183/testReport/org.apache.hadoop.hdfs.server.diskbalancer/].
A few minor comments (some already come from the previous code but we could fix
here):
* The lines break in {{DiskBalancerTestUtil#getBlockCount}} are weird.
* Space missing in {{if(checkblockPoolCount)}}
* Don't compact {{import org.apache.hadoop.hdfs.*;}}
* We have {{assertTrue(DiskBalancerTestUtil.getBlockCount(volume, true) > 0);}}
but we could go for assertEquals(0, DiskBalancerTestUtil.getBlockCount(volume,
true));}}.
* {{TestDiskBalancer}} uses {{Logger}} so you could go for {{LOG.info("{}:
Block Count : {}", refs, DiskBalancerTestUtil.getBlockCount(volume));}}
> Disk Balancer: Support multiple block pools during block move
> -
>
> Key: HDFS-13188
> URL: https://issues.apache.org/jira/browse/HDFS-13188
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: diskbalancer
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: HDFS-13188.01.patch, HDFS-13188.02.patch
>
>
> During execute plan:
> *Federated setup:*
> When multiple block pools are there, it will only copy from blocks from first
> block pool to destination disk, when balancing.
> We want to distribute the blocks from all block pools on source disk to
> destination disk during balancing.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13188) Disk Balancer: Support multiple block pools during block move
[
https://issues.apache.org/jira/browse/HDFS-13188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16375287#comment-16375287
]
genericqa commented on HDFS-13188:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
25s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 16s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 50s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}118m 13s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}179m 8s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
| | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13188 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12911850/HDFS-13188.02.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux ee5acec78c23 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 033f9c6 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/23183/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/23183/testReport/ |
| Max. process+thread count | 2985 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/23183/console |
| Powered by |
[jira] [Commented] (HDFS-13188) Disk Balancer: Support multiple block pools during block move
[
https://issues.apache.org/jira/browse/HDFS-13188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16375240#comment-16375240
]
genericqa commented on HDFS-13188:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 3s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 1s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}104m 49s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}163m 0s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13188 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12911825/HDFS-13188.01.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 1e7285035467 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 68ce193 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/23182/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/23182/testReport/ |
| Max. process+thread count | 4183 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
[jira] [Commented] (HDFS-13188) Disk Balancer: Support multiple block pools during block move
[ https://issues.apache.org/jira/browse/HDFS-13188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16375171#comment-16375171 ] Bharat Viswanadham commented on HDFS-13188: --- [~elgoiri] Attached the patch v01 to add a federated test case, to test the behavior. > Disk Balancer: Support multiple block pools during block move > - > > Key: HDFS-13188 > URL: https://issues.apache.org/jira/browse/HDFS-13188 > Project: Hadoop HDFS > Issue Type: Bug > Components: diskbalancer >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HDFS-13188.01.patch > > > During execute plan: > *Federated setup:* > When multiple block pools are there, it will only copy from blocks from first > block pool to destination disk, when balancing. > We want to distribute the blocks from all block pools on source disk to > destination disk during balancing. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
