[jira] [Updated] (HDFS-8147) Mover should not schedule two replicas to the same DN storage

2017-01-05 Thread Junping Du (JIRA)

 [ 
https://issues.apache.org/jira/browse/HDFS-8147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Junping Du updated HDFS-8147:
-
Fix Version/s: 2.8.0

> Mover should not schedule two replicas to the same DN storage
> -
>
> Key: HDFS-8147
> URL: https://issues.apache.org/jira/browse/HDFS-8147
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Fix For: 2.8.0, 2.7.1, 3.0.0-alpha1
>
> Attachments: HDFS-8147.patch, HDFS-8147_1.patch, HDFS-8147_2.patch, 
> HDFS-8147_3.patch, HDFS-8147_4.patch
>
>
> *Scenario:*
> 1. Three DN cluster.  For DNs storage type is like this.
> DN1 : DISK,ARCHIVE
> DN2 : DISK
> DN3 : DISK,ARCHIVE (All DNs are in same rack)
> 2. One file with two replicas (In DN1 and DN2)
> 3. Set file storage policy COLD
> 4. Now execute Mover.
> *Expected Result:* File blocks should move in DN1:ARCHIVE and DN3:ARCHIVE
> *Actual Result:* {{chooseTargetInSameNode()}} move D1:DISK block to 
> D1:ARCHIVE, but in next iteration {{chooseTarget()}} for same rake is 
> selecting again DN1:ARCHIVE for target where already same block exists.
> {{chooseTargetInSameNode()}} and {{chooseTarget()}} should not select the 
> node as target where already same replica exists.  The dispatcher will fail 
> to move block as shown in the log below.  Then, the Mover will try again in 
> next iteration.
> *Logs*
> {code}
> 15/04/15 10:47:17 WARN balancer.Dispatcher: Failed to move 
> blk_1073741852_1028 with size=11990 from 10.19.92.74:50010:DISK to 
> 10.19.92.73:50010:ARCHIVE through 10.19.92.73:50010: Got error, status 
> message opReplaceBlock 
> BP-1258709199-10.19.92.74-1428292615636:blk_1073741852_1028 received 
> exception 
> org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Replica 
> FinalizedReplica, blk_1073741852_1028, FINALIZED
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-8147) Mover should not schedule two replicas to the same DN storage

2015-04-23 Thread Tsz Wo Nicholas Sze (JIRA)

 [ 
https://issues.apache.org/jira/browse/HDFS-8147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsz Wo Nicholas Sze updated HDFS-8147:
--
Hadoop Flags: Reviewed

+1 the new patch looks good.

 Mover should not schedule two replicas to the same DN storage
 -

 Key: HDFS-8147
 URL: https://issues.apache.org/jira/browse/HDFS-8147
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: balancer  mover
Affects Versions: 2.6.0
Reporter: surendra singh lilhore
Assignee: surendra singh lilhore
 Attachments: HDFS-8147.patch, HDFS-8147_1.patch, HDFS-8147_2.patch, 
 HDFS-8147_3.patch, HDFS-8147_4.patch


 *Scenario:*
 1. Three DN cluster.  For DNs storage type is like this.
 DN1 : DISK,ARCHIVE
 DN2 : DISK
 DN3 : DISK,ARCHIVE (All DNs are in same rack)
 2. One file with two replicas (In DN1 and DN2)
 3. Set file storage policy COLD
 4. Now execute Mover.
 *Expected Result:* File blocks should move in DN1:ARCHIVE and DN3:ARCHIVE
 *Actual Result:* {{chooseTargetInSameNode()}} move D1:DISK block to 
 D1:ARCHIVE, but in next iteration {{chooseTarget()}} for same rake is 
 selecting again DN1:ARCHIVE for target where already same block exists.
 {{chooseTargetInSameNode()}} and {{chooseTarget()}} should not select the 
 node as target where already same replica exists.  The dispatcher will fail 
 to move block as shown in the log below.  Then, the Mover will try again in 
 next iteration.
 *Logs*
 {code}
 15/04/15 10:47:17 WARN balancer.Dispatcher: Failed to move 
 blk_1073741852_1028 with size=11990 from 10.19.92.74:50010:DISK to 
 10.19.92.73:50010:ARCHIVE through 10.19.92.73:50010: Got error, status 
 message opReplaceBlock 
 BP-1258709199-10.19.92.74-1428292615636:blk_1073741852_1028 received 
 exception 
 org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Replica 
 FinalizedReplica, blk_1073741852_1028, FINALIZED
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HDFS-8147) Mover should not schedule two replicas to the same DN storage

2015-04-23 Thread Tsz Wo Nicholas Sze (JIRA)

 [ 
https://issues.apache.org/jira/browse/HDFS-8147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsz Wo Nicholas Sze updated HDFS-8147:
--
   Resolution: Fixed
Fix Version/s: 2.7.1
   Status: Resolved  (was: Patch Available)

I have committed this.  Thanks, surendra!

 Mover should not schedule two replicas to the same DN storage
 -

 Key: HDFS-8147
 URL: https://issues.apache.org/jira/browse/HDFS-8147
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: balancer  mover
Affects Versions: 2.6.0
Reporter: surendra singh lilhore
Assignee: surendra singh lilhore
 Fix For: 2.7.1

 Attachments: HDFS-8147.patch, HDFS-8147_1.patch, HDFS-8147_2.patch, 
 HDFS-8147_3.patch, HDFS-8147_4.patch


 *Scenario:*
 1. Three DN cluster.  For DNs storage type is like this.
 DN1 : DISK,ARCHIVE
 DN2 : DISK
 DN3 : DISK,ARCHIVE (All DNs are in same rack)
 2. One file with two replicas (In DN1 and DN2)
 3. Set file storage policy COLD
 4. Now execute Mover.
 *Expected Result:* File blocks should move in DN1:ARCHIVE and DN3:ARCHIVE
 *Actual Result:* {{chooseTargetInSameNode()}} move D1:DISK block to 
 D1:ARCHIVE, but in next iteration {{chooseTarget()}} for same rake is 
 selecting again DN1:ARCHIVE for target where already same block exists.
 {{chooseTargetInSameNode()}} and {{chooseTarget()}} should not select the 
 node as target where already same replica exists.  The dispatcher will fail 
 to move block as shown in the log below.  Then, the Mover will try again in 
 next iteration.
 *Logs*
 {code}
 15/04/15 10:47:17 WARN balancer.Dispatcher: Failed to move 
 blk_1073741852_1028 with size=11990 from 10.19.92.74:50010:DISK to 
 10.19.92.73:50010:ARCHIVE through 10.19.92.73:50010: Got error, status 
 message opReplaceBlock 
 BP-1258709199-10.19.92.74-1428292615636:blk_1073741852_1028 received 
 exception 
 org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Replica 
 FinalizedReplica, blk_1073741852_1028, FINALIZED
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HDFS-8147) Mover should not schedule two replicas to the same DN storage

2015-04-23 Thread surendra singh lilhore (JIRA)

 [ 
https://issues.apache.org/jira/browse/HDFS-8147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

surendra singh lilhore updated HDFS-8147:
-
Attachment: HDFS-8147_4.patch

 Mover should not schedule two replicas to the same DN storage
 -

 Key: HDFS-8147
 URL: https://issues.apache.org/jira/browse/HDFS-8147
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: balancer  mover
Affects Versions: 2.6.0
Reporter: surendra singh lilhore
Assignee: surendra singh lilhore
 Attachments: HDFS-8147.patch, HDFS-8147_1.patch, HDFS-8147_2.patch, 
 HDFS-8147_3.patch, HDFS-8147_4.patch


 *Scenario:*
 1. Three DN cluster.  For DNs storage type is like this.
 DN1 : DISK,ARCHIVE
 DN2 : DISK
 DN3 : DISK,ARCHIVE (All DNs are in same rack)
 2. One file with two replicas (In DN1 and DN2)
 3. Set file storage policy COLD
 4. Now execute Mover.
 *Expected Result:* File blocks should move in DN1:ARCHIVE and DN3:ARCHIVE
 *Actual Result:* {{chooseTargetInSameNode()}} move D1:DISK block to 
 D1:ARCHIVE, but in next iteration {{chooseTarget()}} for same rake is 
 selecting again DN1:ARCHIVE for target where already same block exists.
 {{chooseTargetInSameNode()}} and {{chooseTarget()}} should not select the 
 node as target where already same replica exists.  The dispatcher will fail 
 to move block as shown in the log below.  Then, the Mover will try again in 
 next iteration.
 *Logs*
 {code}
 15/04/15 10:47:17 WARN balancer.Dispatcher: Failed to move 
 blk_1073741852_1028 with size=11990 from 10.19.92.74:50010:DISK to 
 10.19.92.73:50010:ARCHIVE through 10.19.92.73:50010: Got error, status 
 message opReplaceBlock 
 BP-1258709199-10.19.92.74-1428292615636:blk_1073741852_1028 received 
 exception 
 org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Replica 
 FinalizedReplica, blk_1073741852_1028, FINALIZED
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HDFS-8147) Mover should not schedule two replicas to the same DN storage

2015-04-22 Thread Tsz Wo Nicholas Sze (JIRA)

 [ 
https://issues.apache.org/jira/browse/HDFS-8147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsz Wo Nicholas Sze updated HDFS-8147:
--
Description: 
*Scenario:*
1. Three DN cluster.  For DNs storage type is like this.
DN1 : DISK,ARCHIVE
DN2 : DISK
DN3 : DISK,ARCHIVE (All DNs are in same rack)
2. One file with two replicas (In DN1 and DN2)
3. Set file storage policy COLD
4. Now execute Mover.

*Expected Result:* File blocks should move in DN1:ARCHIVE and DN3:ARCHIVE

*Actual Result:* {{chooseTargetInSameNode()}} move D1:DISK block to D1:ARCHIVE, 
but in next iteration {{chooseTarget()}} for same rake is selecting again 
DN1:ARCHIVE for target where already same block exists.
{{chooseTargetInSameNode()}} and {{chooseTarget()}} should not select the node 
as target where already same replica exists.  The dispatcher will fail to move 
block as shown in the log below.  Then, the Mover will try again in next 
iteration.

*Logs*
{code}
15/04/15 10:47:17 WARN balancer.Dispatcher: Failed to move blk_1073741852_1028 
with size=11990 from 10.19.92.74:50010:DISK to 10.19.92.73:50010:ARCHIVE 
through 10.19.92.73:50010: Got error, status message opReplaceBlock 
BP-1258709199-10.19.92.74-1428292615636:blk_1073741852_1028 received exception 
org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Replica 
FinalizedReplica, blk_1073741852_1028, FINALIZED
{code}


  was:
*Scenario:*
1. Three DN cluster.  For DNs storage type is like this.
DN1 : DISK,ARCHIVE
DN2 : DISK
DN3 : DISK,ARCHIVE (All DNs are in same rack)
2. One file with two replicas (In DN1 and DN2)
3. Set file storage policy COLD
4. Now execute Mover.

*Expected Result:* File blocks should move in DN1:ARCHIVE and DN3:ARCHIVE

*Actual Result:* {{chooseTargetInSameNode()}} move D1:DISK block to D1:ARCHIVE, 
but in next iteration {{chooseTarget()}} for same rake is selecting again 
DN1:ARCHIVE for target where already same block exists.
{{chooseTargetInSameNode()}} and {{chooseTarget()}} should not select the node 
as target where already same replica exists.

*Logs*
{code}
15/04/15 10:47:17 WARN balancer.Dispatcher: Failed to move blk_1073741852_1028 
with size=11990 from 10.19.92.74:50010:DISK to 10.19.92.73:50010:ARCHIVE 
through 10.19.92.73:50010: Got error, status message opReplaceBlock 
BP-1258709199-10.19.92.74-1428292615636:blk_1073741852_1028 received exception 
org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Replica 
FinalizedReplica, blk_1073741852_1028, FINALIZED
{code}


Summary: Mover should not schedule two replicas to the same DN storage  
(was: Mover should not select the DN storage as target where already same 
replica exists.)

Revised summary and description.

 Mover should not schedule two replicas to the same DN storage
 -

 Key: HDFS-8147
 URL: https://issues.apache.org/jira/browse/HDFS-8147
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: balancer  mover
Affects Versions: 2.6.0
Reporter: surendra singh lilhore
Assignee: surendra singh lilhore
 Attachments: HDFS-8147.patch, HDFS-8147_1.patch, HDFS-8147_2.patch, 
 HDFS-8147_3.patch


 *Scenario:*
 1. Three DN cluster.  For DNs storage type is like this.
 DN1 : DISK,ARCHIVE
 DN2 : DISK
 DN3 : DISK,ARCHIVE (All DNs are in same rack)
 2. One file with two replicas (In DN1 and DN2)
 3. Set file storage policy COLD
 4. Now execute Mover.
 *Expected Result:* File blocks should move in DN1:ARCHIVE and DN3:ARCHIVE
 *Actual Result:* {{chooseTargetInSameNode()}} move D1:DISK block to 
 D1:ARCHIVE, but in next iteration {{chooseTarget()}} for same rake is 
 selecting again DN1:ARCHIVE for target where already same block exists.
 {{chooseTargetInSameNode()}} and {{chooseTarget()}} should not select the 
 node as target where already same replica exists.  The dispatcher will fail 
 to move block as shown in the log below.  Then, the Mover will try again in 
 next iteration.
 *Logs*
 {code}
 15/04/15 10:47:17 WARN balancer.Dispatcher: Failed to move 
 blk_1073741852_1028 with size=11990 from 10.19.92.74:50010:DISK to 
 10.19.92.73:50010:ARCHIVE through 10.19.92.73:50010: Got error, status 
 message opReplaceBlock 
 BP-1258709199-10.19.92.74-1428292615636:blk_1073741852_1028 received 
 exception 
 org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Replica 
 FinalizedReplica, blk_1073741852_1028, FINALIZED
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)