[
https://issues.apache.org/jira/browse/HDFS-8147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo Nicholas Sze updated HDFS-8147:
--------------------------------------
Description:
*Scenario:*
1. Three DN cluster. For DNs storage type is like this.
DN1 : DISK,ARCHIVE
DN2 : DISK
DN3 : DISK,ARCHIVE (All DNs are in same rack)
2. One file with two replicas (In DN1 and DN2)
3. Set file storage policy COLD
4. Now execute Mover.
*Expected Result:* File blocks should move in DN1:ARCHIVE and DN3:ARCHIVE
*Actual Result:* {{chooseTargetInSameNode()}} move D1:DISK block to D1:ARCHIVE,
but in next iteration {{chooseTarget()}} for same rake is selecting again
DN1:ARCHIVE for target where already same block exists.
{{chooseTargetInSameNode()}} and {{chooseTarget()}} should not select the node
as target where already same replica exists. The dispatcher will fail to move
block as shown in the log below. Then, the Mover will try again in next
iteration.
*Logs*
{code}
15/04/15 10:47:17 WARN balancer.Dispatcher: Failed to move blk_1073741852_1028
with size=11990 from 10.19.92.74:50010:DISK to 10.19.92.73:50010:ARCHIVE
through 10.19.92.73:50010: Got error, status message opReplaceBlock
BP-1258709199-10.19.92.74-1428292615636:blk_1073741852_1028 received exception
org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Replica
FinalizedReplica, blk_1073741852_1028, FINALIZED
{code}
was:
*Scenario:*
1. Three DN cluster. For DNs storage type is like this.
DN1 : DISK,ARCHIVE
DN2 : DISK
DN3 : DISK,ARCHIVE (All DNs are in same rack)
2. One file with two replicas (In DN1 and DN2)
3. Set file storage policy COLD
4. Now execute Mover.
*Expected Result:* File blocks should move in DN1:ARCHIVE and DN3:ARCHIVE
*Actual Result:* {{chooseTargetInSameNode()}} move D1:DISK block to D1:ARCHIVE,
but in next iteration {{chooseTarget()}} for same rake is selecting again
DN1:ARCHIVE for target where already same block exists.
{{chooseTargetInSameNode()}} and {{chooseTarget()}} should not select the node
as target where already same replica exists.
*Logs*
{code}
15/04/15 10:47:17 WARN balancer.Dispatcher: Failed to move blk_1073741852_1028
with size=11990 from 10.19.92.74:50010:DISK to 10.19.92.73:50010:ARCHIVE
through 10.19.92.73:50010: Got error, status message opReplaceBlock
BP-1258709199-10.19.92.74-1428292615636:blk_1073741852_1028 received exception
org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Replica
FinalizedReplica, blk_1073741852_1028, FINALIZED
{code}
Summary: Mover should not schedule two replicas to the same DN storage
(was: Mover should not select the DN storage as target where already same
replica exists.)
Revised summary and description.
> Mover should not schedule two replicas to the same DN storage
> -------------------------------------------------------------
>
> Key: HDFS-8147
> URL: https://issues.apache.org/jira/browse/HDFS-8147
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
> Affects Versions: 2.6.0
> Reporter: surendra singh lilhore
> Assignee: surendra singh lilhore
> Attachments: HDFS-8147.patch, HDFS-8147_1.patch, HDFS-8147_2.patch,
> HDFS-8147_3.patch
>
>
> *Scenario:*
> 1. Three DN cluster. For DNs storage type is like this.
> DN1 : DISK,ARCHIVE
> DN2 : DISK
> DN3 : DISK,ARCHIVE (All DNs are in same rack)
> 2. One file with two replicas (In DN1 and DN2)
> 3. Set file storage policy COLD
> 4. Now execute Mover.
> *Expected Result:* File blocks should move in DN1:ARCHIVE and DN3:ARCHIVE
> *Actual Result:* {{chooseTargetInSameNode()}} move D1:DISK block to
> D1:ARCHIVE, but in next iteration {{chooseTarget()}} for same rake is
> selecting again DN1:ARCHIVE for target where already same block exists.
> {{chooseTargetInSameNode()}} and {{chooseTarget()}} should not select the
> node as target where already same replica exists. The dispatcher will fail
> to move block as shown in the log below. Then, the Mover will try again in
> next iteration.
> *Logs*
> {code}
> 15/04/15 10:47:17 WARN balancer.Dispatcher: Failed to move
> blk_1073741852_1028 with size=11990 from 10.19.92.74:50010:DISK to
> 10.19.92.73:50010:ARCHIVE through 10.19.92.73:50010: Got error, status
> message opReplaceBlock
> BP-1258709199-10.19.92.74-1428292615636:blk_1073741852_1028 received
> exception
> org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Replica
> FinalizedReplica, blk_1073741852_1028, FINALIZED
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
