[jira] [Commented] (HDFS-8147) Mover should not schedule two replicas to the same DN storage
[
https://issues.apache.org/jira/browse/HDFS-8147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14510847#comment-14510847
]
Hudson commented on HDFS-8147:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #2105 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2105/])
HDFS-8147. StorageGroup in Dispatcher should override equals nad hashCode.
Contributed by surendra singh lilhore (szetszwo: rev
416b84354ebcc6f6f23598a5a254e62eb606680d)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
Mover should not schedule two replicas to the same DN storage
-
Key: HDFS-8147
URL: https://issues.apache.org/jira/browse/HDFS-8147
Project: Hadoop HDFS
Issue Type: Bug
Components: balancer mover
Affects Versions: 2.6.0
Reporter: surendra singh lilhore
Assignee: surendra singh lilhore
Fix For: 2.7.1
Attachments: HDFS-8147.patch, HDFS-8147_1.patch, HDFS-8147_2.patch,
HDFS-8147_3.patch, HDFS-8147_4.patch
*Scenario:*
1. Three DN cluster. For DNs storage type is like this.
DN1 : DISK,ARCHIVE
DN2 : DISK
DN3 : DISK,ARCHIVE (All DNs are in same rack)
2. One file with two replicas (In DN1 and DN2)
3. Set file storage policy COLD
4. Now execute Mover.
*Expected Result:* File blocks should move in DN1:ARCHIVE and DN3:ARCHIVE
*Actual Result:* {{chooseTargetInSameNode()}} move D1:DISK block to
D1:ARCHIVE, but in next iteration {{chooseTarget()}} for same rake is
selecting again DN1:ARCHIVE for target where already same block exists.
{{chooseTargetInSameNode()}} and {{chooseTarget()}} should not select the
node as target where already same replica exists. The dispatcher will fail
to move block as shown in the log below. Then, the Mover will try again in
next iteration.
*Logs*
{code}
15/04/15 10:47:17 WARN balancer.Dispatcher: Failed to move
blk_1073741852_1028 with size=11990 from 10.19.92.74:50010:DISK to
10.19.92.73:50010:ARCHIVE through 10.19.92.73:50010: Got error, status
message opReplaceBlock
BP-1258709199-10.19.92.74-1428292615636:blk_1073741852_1028 received
exception
org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Replica
FinalizedReplica, blk_1073741852_1028, FINALIZED
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8147) Mover should not schedule two replicas to the same DN storage
[
https://issues.apache.org/jira/browse/HDFS-8147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14510864#comment-14510864
]
Hudson commented on HDFS-8147:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #164 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/164/])
HDFS-8147. StorageGroup in Dispatcher should override equals nad hashCode.
Contributed by surendra singh lilhore (szetszwo: rev
416b84354ebcc6f6f23598a5a254e62eb606680d)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
Mover should not schedule two replicas to the same DN storage
-
Key: HDFS-8147
URL: https://issues.apache.org/jira/browse/HDFS-8147
Project: Hadoop HDFS
Issue Type: Bug
Components: balancer mover
Affects Versions: 2.6.0
Reporter: surendra singh lilhore
Assignee: surendra singh lilhore
Fix For: 2.7.1
Attachments: HDFS-8147.patch, HDFS-8147_1.patch, HDFS-8147_2.patch,
HDFS-8147_3.patch, HDFS-8147_4.patch
*Scenario:*
1. Three DN cluster. For DNs storage type is like this.
DN1 : DISK,ARCHIVE
DN2 : DISK
DN3 : DISK,ARCHIVE (All DNs are in same rack)
2. One file with two replicas (In DN1 and DN2)
3. Set file storage policy COLD
4. Now execute Mover.
*Expected Result:* File blocks should move in DN1:ARCHIVE and DN3:ARCHIVE
*Actual Result:* {{chooseTargetInSameNode()}} move D1:DISK block to
D1:ARCHIVE, but in next iteration {{chooseTarget()}} for same rake is
selecting again DN1:ARCHIVE for target where already same block exists.
{{chooseTargetInSameNode()}} and {{chooseTarget()}} should not select the
node as target where already same replica exists. The dispatcher will fail
to move block as shown in the log below. Then, the Mover will try again in
next iteration.
*Logs*
{code}
15/04/15 10:47:17 WARN balancer.Dispatcher: Failed to move
blk_1073741852_1028 with size=11990 from 10.19.92.74:50010:DISK to
10.19.92.73:50010:ARCHIVE through 10.19.92.73:50010: Got error, status
message opReplaceBlock
BP-1258709199-10.19.92.74-1428292615636:blk_1073741852_1028 received
exception
org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Replica
FinalizedReplica, blk_1073741852_1028, FINALIZED
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8147) Mover should not schedule two replicas to the same DN storage
[
https://issues.apache.org/jira/browse/HDFS-8147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14510922#comment-14510922
]
Hudson commented on HDFS-8147:
--
SUCCESS: Integrated in Hadoop-Yarn-trunk #907 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/907/])
HDFS-8147. StorageGroup in Dispatcher should override equals nad hashCode.
Contributed by surendra singh lilhore (szetszwo: rev
416b84354ebcc6f6f23598a5a254e62eb606680d)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java
Mover should not schedule two replicas to the same DN storage
-
Key: HDFS-8147
URL: https://issues.apache.org/jira/browse/HDFS-8147
Project: Hadoop HDFS
Issue Type: Bug
Components: balancer mover
Affects Versions: 2.6.0
Reporter: surendra singh lilhore
Assignee: surendra singh lilhore
Fix For: 2.7.1
Attachments: HDFS-8147.patch, HDFS-8147_1.patch, HDFS-8147_2.patch,
HDFS-8147_3.patch, HDFS-8147_4.patch
*Scenario:*
1. Three DN cluster. For DNs storage type is like this.
DN1 : DISK,ARCHIVE
DN2 : DISK
DN3 : DISK,ARCHIVE (All DNs are in same rack)
2. One file with two replicas (In DN1 and DN2)
3. Set file storage policy COLD
4. Now execute Mover.
*Expected Result:* File blocks should move in DN1:ARCHIVE and DN3:ARCHIVE
*Actual Result:* {{chooseTargetInSameNode()}} move D1:DISK block to
D1:ARCHIVE, but in next iteration {{chooseTarget()}} for same rake is
selecting again DN1:ARCHIVE for target where already same block exists.
{{chooseTargetInSameNode()}} and {{chooseTarget()}} should not select the
node as target where already same replica exists. The dispatcher will fail
to move block as shown in the log below. Then, the Mover will try again in
next iteration.
*Logs*
{code}
15/04/15 10:47:17 WARN balancer.Dispatcher: Failed to move
blk_1073741852_1028 with size=11990 from 10.19.92.74:50010:DISK to
10.19.92.73:50010:ARCHIVE through 10.19.92.73:50010: Got error, status
message opReplaceBlock
BP-1258709199-10.19.92.74-1428292615636:blk_1073741852_1028 received
exception
org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Replica
FinalizedReplica, blk_1073741852_1028, FINALIZED
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8147) Mover should not schedule two replicas to the same DN storage
[
https://issues.apache.org/jira/browse/HDFS-8147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14510883#comment-14510883
]
Hudson commented on HDFS-8147:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #173 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/173/])
HDFS-8147. StorageGroup in Dispatcher should override equals nad hashCode.
Contributed by surendra singh lilhore (szetszwo: rev
416b84354ebcc6f6f23598a5a254e62eb606680d)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
Mover should not schedule two replicas to the same DN storage
-
Key: HDFS-8147
URL: https://issues.apache.org/jira/browse/HDFS-8147
Project: Hadoop HDFS
Issue Type: Bug
Components: balancer mover
Affects Versions: 2.6.0
Reporter: surendra singh lilhore
Assignee: surendra singh lilhore
Fix For: 2.7.1
Attachments: HDFS-8147.patch, HDFS-8147_1.patch, HDFS-8147_2.patch,
HDFS-8147_3.patch, HDFS-8147_4.patch
*Scenario:*
1. Three DN cluster. For DNs storage type is like this.
DN1 : DISK,ARCHIVE
DN2 : DISK
DN3 : DISK,ARCHIVE (All DNs are in same rack)
2. One file with two replicas (In DN1 and DN2)
3. Set file storage policy COLD
4. Now execute Mover.
*Expected Result:* File blocks should move in DN1:ARCHIVE and DN3:ARCHIVE
*Actual Result:* {{chooseTargetInSameNode()}} move D1:DISK block to
D1:ARCHIVE, but in next iteration {{chooseTarget()}} for same rake is
selecting again DN1:ARCHIVE for target where already same block exists.
{{chooseTargetInSameNode()}} and {{chooseTarget()}} should not select the
node as target where already same replica exists. The dispatcher will fail
to move block as shown in the log below. Then, the Mover will try again in
next iteration.
*Logs*
{code}
15/04/15 10:47:17 WARN balancer.Dispatcher: Failed to move
blk_1073741852_1028 with size=11990 from 10.19.92.74:50010:DISK to
10.19.92.73:50010:ARCHIVE through 10.19.92.73:50010: Got error, status
message opReplaceBlock
BP-1258709199-10.19.92.74-1428292615636:blk_1073741852_1028 received
exception
org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Replica
FinalizedReplica, blk_1073741852_1028, FINALIZED
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8147) Mover should not schedule two replicas to the same DN storage
[
https://issues.apache.org/jira/browse/HDFS-8147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14511025#comment-14511025
]
Hudson commented on HDFS-8147:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #174 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/174/])
HDFS-8147. StorageGroup in Dispatcher should override equals nad hashCode.
Contributed by surendra singh lilhore (szetszwo: rev
416b84354ebcc6f6f23598a5a254e62eb606680d)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java
Mover should not schedule two replicas to the same DN storage
-
Key: HDFS-8147
URL: https://issues.apache.org/jira/browse/HDFS-8147
Project: Hadoop HDFS
Issue Type: Bug
Components: balancer mover
Affects Versions: 2.6.0
Reporter: surendra singh lilhore
Assignee: surendra singh lilhore
Fix For: 2.7.1
Attachments: HDFS-8147.patch, HDFS-8147_1.patch, HDFS-8147_2.patch,
HDFS-8147_3.patch, HDFS-8147_4.patch
*Scenario:*
1. Three DN cluster. For DNs storage type is like this.
DN1 : DISK,ARCHIVE
DN2 : DISK
DN3 : DISK,ARCHIVE (All DNs are in same rack)
2. One file with two replicas (In DN1 and DN2)
3. Set file storage policy COLD
4. Now execute Mover.
*Expected Result:* File blocks should move in DN1:ARCHIVE and DN3:ARCHIVE
*Actual Result:* {{chooseTargetInSameNode()}} move D1:DISK block to
D1:ARCHIVE, but in next iteration {{chooseTarget()}} for same rake is
selecting again DN1:ARCHIVE for target where already same block exists.
{{chooseTargetInSameNode()}} and {{chooseTarget()}} should not select the
node as target where already same replica exists. The dispatcher will fail
to move block as shown in the log below. Then, the Mover will try again in
next iteration.
*Logs*
{code}
15/04/15 10:47:17 WARN balancer.Dispatcher: Failed to move
blk_1073741852_1028 with size=11990 from 10.19.92.74:50010:DISK to
10.19.92.73:50010:ARCHIVE through 10.19.92.73:50010: Got error, status
message opReplaceBlock
BP-1258709199-10.19.92.74-1428292615636:blk_1073741852_1028 received
exception
org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Replica
FinalizedReplica, blk_1073741852_1028, FINALIZED
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8147) Mover should not schedule two replicas to the same DN storage
[
https://issues.apache.org/jira/browse/HDFS-8147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14511064#comment-14511064
]
Hudson commented on HDFS-8147:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2123 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2123/])
HDFS-8147. StorageGroup in Dispatcher should override equals nad hashCode.
Contributed by surendra singh lilhore (szetszwo: rev
416b84354ebcc6f6f23598a5a254e62eb606680d)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java
Mover should not schedule two replicas to the same DN storage
-
Key: HDFS-8147
URL: https://issues.apache.org/jira/browse/HDFS-8147
Project: Hadoop HDFS
Issue Type: Bug
Components: balancer mover
Affects Versions: 2.6.0
Reporter: surendra singh lilhore
Assignee: surendra singh lilhore
Fix For: 2.7.1
Attachments: HDFS-8147.patch, HDFS-8147_1.patch, HDFS-8147_2.patch,
HDFS-8147_3.patch, HDFS-8147_4.patch
*Scenario:*
1. Three DN cluster. For DNs storage type is like this.
DN1 : DISK,ARCHIVE
DN2 : DISK
DN3 : DISK,ARCHIVE (All DNs are in same rack)
2. One file with two replicas (In DN1 and DN2)
3. Set file storage policy COLD
4. Now execute Mover.
*Expected Result:* File blocks should move in DN1:ARCHIVE and DN3:ARCHIVE
*Actual Result:* {{chooseTargetInSameNode()}} move D1:DISK block to
D1:ARCHIVE, but in next iteration {{chooseTarget()}} for same rake is
selecting again DN1:ARCHIVE for target where already same block exists.
{{chooseTargetInSameNode()}} and {{chooseTarget()}} should not select the
node as target where already same replica exists. The dispatcher will fail
to move block as shown in the log below. Then, the Mover will try again in
next iteration.
*Logs*
{code}
15/04/15 10:47:17 WARN balancer.Dispatcher: Failed to move
blk_1073741852_1028 with size=11990 from 10.19.92.74:50010:DISK to
10.19.92.73:50010:ARCHIVE through 10.19.92.73:50010: Got error, status
message opReplaceBlock
BP-1258709199-10.19.92.74-1428292615636:blk_1073741852_1028 received
exception
org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Replica
FinalizedReplica, blk_1073741852_1028, FINALIZED
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8147) Mover should not schedule two replicas to the same DN storage
[
https://issues.apache.org/jira/browse/HDFS-8147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14509599#comment-14509599
]
Hudson commented on HDFS-8147:
--
FAILURE: Integrated in Hadoop-trunk-Commit #7652 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/7652/])
HDFS-8147. StorageGroup in Dispatcher should override equals nad hashCode.
Contributed by surendra singh lilhore (szetszwo: rev
416b84354ebcc6f6f23598a5a254e62eb606680d)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
Mover should not schedule two replicas to the same DN storage
-
Key: HDFS-8147
URL: https://issues.apache.org/jira/browse/HDFS-8147
Project: Hadoop HDFS
Issue Type: Bug
Components: balancer mover
Affects Versions: 2.6.0
Reporter: surendra singh lilhore
Assignee: surendra singh lilhore
Fix For: 2.7.1
Attachments: HDFS-8147.patch, HDFS-8147_1.patch, HDFS-8147_2.patch,
HDFS-8147_3.patch, HDFS-8147_4.patch
*Scenario:*
1. Three DN cluster. For DNs storage type is like this.
DN1 : DISK,ARCHIVE
DN2 : DISK
DN3 : DISK,ARCHIVE (All DNs are in same rack)
2. One file with two replicas (In DN1 and DN2)
3. Set file storage policy COLD
4. Now execute Mover.
*Expected Result:* File blocks should move in DN1:ARCHIVE and DN3:ARCHIVE
*Actual Result:* {{chooseTargetInSameNode()}} move D1:DISK block to
D1:ARCHIVE, but in next iteration {{chooseTarget()}} for same rake is
selecting again DN1:ARCHIVE for target where already same block exists.
{{chooseTargetInSameNode()}} and {{chooseTarget()}} should not select the
node as target where already same replica exists. The dispatcher will fail
to move block as shown in the log below. Then, the Mover will try again in
next iteration.
*Logs*
{code}
15/04/15 10:47:17 WARN balancer.Dispatcher: Failed to move
blk_1073741852_1028 with size=11990 from 10.19.92.74:50010:DISK to
10.19.92.73:50010:ARCHIVE through 10.19.92.73:50010: Got error, status
message opReplaceBlock
BP-1258709199-10.19.92.74-1428292615636:blk_1073741852_1028 received
exception
org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Replica
FinalizedReplica, blk_1073741852_1028, FINALIZED
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8147) Mover should not schedule two replicas to the same DN storage
[
https://issues.apache.org/jira/browse/HDFS-8147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14509401#comment-14509401
]
Hadoop QA commented on HDFS-8147:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 37s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | javac | 7m 26s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 34s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 7m 47s | The applied patch generated 1
additional checkstyle issues. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 4s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | native | 3m 12s | Pre-build of native portion |
| {color:green}+1{color} | hdfs tests | 186m 38s | Tests passed in hadoop-hdfs.
|
| | | 234m 50s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12727606/HDFS-8147_4.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / baf8bc6 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/10358/artifact/patchprocess/checkstyle-result-diff.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/10358/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/10358/testReport/ |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10358/console |
This message was automatically generated.
Mover should not schedule two replicas to the same DN storage
-
Key: HDFS-8147
URL: https://issues.apache.org/jira/browse/HDFS-8147
Project: Hadoop HDFS
Issue Type: Bug
Components: balancer mover
Affects Versions: 2.6.0
Reporter: surendra singh lilhore
Assignee: surendra singh lilhore
Attachments: HDFS-8147.patch, HDFS-8147_1.patch, HDFS-8147_2.patch,
HDFS-8147_3.patch, HDFS-8147_4.patch
*Scenario:*
1. Three DN cluster. For DNs storage type is like this.
DN1 : DISK,ARCHIVE
DN2 : DISK
DN3 : DISK,ARCHIVE (All DNs are in same rack)
2. One file with two replicas (In DN1 and DN2)
3. Set file storage policy COLD
4. Now execute Mover.
*Expected Result:* File blocks should move in DN1:ARCHIVE and DN3:ARCHIVE
*Actual Result:* {{chooseTargetInSameNode()}} move D1:DISK block to
D1:ARCHIVE, but in next iteration {{chooseTarget()}} for same rake is
selecting again DN1:ARCHIVE for target where already same block exists.
{{chooseTargetInSameNode()}} and {{chooseTarget()}} should not select the
node as target where already same replica exists. The dispatcher will fail
to move block as shown in the log below. Then, the Mover will try again in
next iteration.
*Logs*
{code}
15/04/15 10:47:17 WARN balancer.Dispatcher: Failed to move
blk_1073741852_1028 with size=11990 from 10.19.92.74:50010:DISK to
10.19.92.73:50010:ARCHIVE through 10.19.92.73:50010: Got error, status
message opReplaceBlock
BP-1258709199-10.19.92.74-1428292615636:blk_1073741852_1028 received
exception
org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Replica
FinalizedReplica, blk_1073741852_1028, FINALIZED
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8147) Mover should not schedule two replicas to the same DN storage
[
https://issues.apache.org/jira/browse/HDFS-8147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14509040#comment-14509040
]
surendra singh lilhore commented on HDFS-8147:
--
[~szetszwo] Thanks for review.
Fixed review comments and attached new patch, Please review.
Mover should not schedule two replicas to the same DN storage
-
Key: HDFS-8147
URL: https://issues.apache.org/jira/browse/HDFS-8147
Project: Hadoop HDFS
Issue Type: Bug
Components: balancer mover
Affects Versions: 2.6.0
Reporter: surendra singh lilhore
Assignee: surendra singh lilhore
Attachments: HDFS-8147.patch, HDFS-8147_1.patch, HDFS-8147_2.patch,
HDFS-8147_3.patch, HDFS-8147_4.patch
*Scenario:*
1. Three DN cluster. For DNs storage type is like this.
DN1 : DISK,ARCHIVE
DN2 : DISK
DN3 : DISK,ARCHIVE (All DNs are in same rack)
2. One file with two replicas (In DN1 and DN2)
3. Set file storage policy COLD
4. Now execute Mover.
*Expected Result:* File blocks should move in DN1:ARCHIVE and DN3:ARCHIVE
*Actual Result:* {{chooseTargetInSameNode()}} move D1:DISK block to
D1:ARCHIVE, but in next iteration {{chooseTarget()}} for same rake is
selecting again DN1:ARCHIVE for target where already same block exists.
{{chooseTargetInSameNode()}} and {{chooseTarget()}} should not select the
node as target where already same replica exists. The dispatcher will fail
to move block as shown in the log below. Then, the Mover will try again in
next iteration.
*Logs*
{code}
15/04/15 10:47:17 WARN balancer.Dispatcher: Failed to move
blk_1073741852_1028 with size=11990 from 10.19.92.74:50010:DISK to
10.19.92.73:50010:ARCHIVE through 10.19.92.73:50010: Got error, status
message opReplaceBlock
BP-1258709199-10.19.92.74-1428292615636:blk_1073741852_1028 received
exception
org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Replica
FinalizedReplica, blk_1073741852_1028, FINALIZED
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
