[jira] [Updated] (HDFS-8147) Mover should not schedule two replicas to the same DN storage
[ https://issues.apache.org/jira/browse/HDFS-8147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-8147: - Fix Version/s: 2.8.0 > Mover should not schedule two replicas to the same DN storage > - > > Key: HDFS-8147 > URL: https://issues.apache.org/jira/browse/HDFS-8147 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer & mover >Affects Versions: 2.6.0 >Reporter: Surendra Singh Lilhore >Assignee: Surendra Singh Lilhore > Fix For: 2.8.0, 2.7.1, 3.0.0-alpha1 > > Attachments: HDFS-8147.patch, HDFS-8147_1.patch, HDFS-8147_2.patch, > HDFS-8147_3.patch, HDFS-8147_4.patch > > > *Scenario:* > 1. Three DN cluster. For DNs storage type is like this. > DN1 : DISK,ARCHIVE > DN2 : DISK > DN3 : DISK,ARCHIVE (All DNs are in same rack) > 2. One file with two replicas (In DN1 and DN2) > 3. Set file storage policy COLD > 4. Now execute Mover. > *Expected Result:* File blocks should move in DN1:ARCHIVE and DN3:ARCHIVE > *Actual Result:* {{chooseTargetInSameNode()}} move D1:DISK block to > D1:ARCHIVE, but in next iteration {{chooseTarget()}} for same rake is > selecting again DN1:ARCHIVE for target where already same block exists. > {{chooseTargetInSameNode()}} and {{chooseTarget()}} should not select the > node as target where already same replica exists. The dispatcher will fail > to move block as shown in the log below. Then, the Mover will try again in > next iteration. > *Logs* > {code} > 15/04/15 10:47:17 WARN balancer.Dispatcher: Failed to move > blk_1073741852_1028 with size=11990 from 10.19.92.74:50010:DISK to > 10.19.92.73:50010:ARCHIVE through 10.19.92.73:50010: Got error, status > message opReplaceBlock > BP-1258709199-10.19.92.74-1428292615636:blk_1073741852_1028 received > exception > org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Replica > FinalizedReplica, blk_1073741852_1028, FINALIZED > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8147) Mover should not schedule two replicas to the same DN storage
[ https://issues.apache.org/jira/browse/HDFS-8147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-8147: -- Hadoop Flags: Reviewed +1 the new patch looks good. Mover should not schedule two replicas to the same DN storage - Key: HDFS-8147 URL: https://issues.apache.org/jira/browse/HDFS-8147 Project: Hadoop HDFS Issue Type: Bug Components: balancer mover Affects Versions: 2.6.0 Reporter: surendra singh lilhore Assignee: surendra singh lilhore Attachments: HDFS-8147.patch, HDFS-8147_1.patch, HDFS-8147_2.patch, HDFS-8147_3.patch, HDFS-8147_4.patch *Scenario:* 1. Three DN cluster. For DNs storage type is like this. DN1 : DISK,ARCHIVE DN2 : DISK DN3 : DISK,ARCHIVE (All DNs are in same rack) 2. One file with two replicas (In DN1 and DN2) 3. Set file storage policy COLD 4. Now execute Mover. *Expected Result:* File blocks should move in DN1:ARCHIVE and DN3:ARCHIVE *Actual Result:* {{chooseTargetInSameNode()}} move D1:DISK block to D1:ARCHIVE, but in next iteration {{chooseTarget()}} for same rake is selecting again DN1:ARCHIVE for target where already same block exists. {{chooseTargetInSameNode()}} and {{chooseTarget()}} should not select the node as target where already same replica exists. The dispatcher will fail to move block as shown in the log below. Then, the Mover will try again in next iteration. *Logs* {code} 15/04/15 10:47:17 WARN balancer.Dispatcher: Failed to move blk_1073741852_1028 with size=11990 from 10.19.92.74:50010:DISK to 10.19.92.73:50010:ARCHIVE through 10.19.92.73:50010: Got error, status message opReplaceBlock BP-1258709199-10.19.92.74-1428292615636:blk_1073741852_1028 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Replica FinalizedReplica, blk_1073741852_1028, FINALIZED {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8147) Mover should not schedule two replicas to the same DN storage
[ https://issues.apache.org/jira/browse/HDFS-8147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-8147: -- Resolution: Fixed Fix Version/s: 2.7.1 Status: Resolved (was: Patch Available) I have committed this. Thanks, surendra! Mover should not schedule two replicas to the same DN storage - Key: HDFS-8147 URL: https://issues.apache.org/jira/browse/HDFS-8147 Project: Hadoop HDFS Issue Type: Bug Components: balancer mover Affects Versions: 2.6.0 Reporter: surendra singh lilhore Assignee: surendra singh lilhore Fix For: 2.7.1 Attachments: HDFS-8147.patch, HDFS-8147_1.patch, HDFS-8147_2.patch, HDFS-8147_3.patch, HDFS-8147_4.patch *Scenario:* 1. Three DN cluster. For DNs storage type is like this. DN1 : DISK,ARCHIVE DN2 : DISK DN3 : DISK,ARCHIVE (All DNs are in same rack) 2. One file with two replicas (In DN1 and DN2) 3. Set file storage policy COLD 4. Now execute Mover. *Expected Result:* File blocks should move in DN1:ARCHIVE and DN3:ARCHIVE *Actual Result:* {{chooseTargetInSameNode()}} move D1:DISK block to D1:ARCHIVE, but in next iteration {{chooseTarget()}} for same rake is selecting again DN1:ARCHIVE for target where already same block exists. {{chooseTargetInSameNode()}} and {{chooseTarget()}} should not select the node as target where already same replica exists. The dispatcher will fail to move block as shown in the log below. Then, the Mover will try again in next iteration. *Logs* {code} 15/04/15 10:47:17 WARN balancer.Dispatcher: Failed to move blk_1073741852_1028 with size=11990 from 10.19.92.74:50010:DISK to 10.19.92.73:50010:ARCHIVE through 10.19.92.73:50010: Got error, status message opReplaceBlock BP-1258709199-10.19.92.74-1428292615636:blk_1073741852_1028 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Replica FinalizedReplica, blk_1073741852_1028, FINALIZED {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8147) Mover should not schedule two replicas to the same DN storage
[ https://issues.apache.org/jira/browse/HDFS-8147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] surendra singh lilhore updated HDFS-8147: - Attachment: HDFS-8147_4.patch Mover should not schedule two replicas to the same DN storage - Key: HDFS-8147 URL: https://issues.apache.org/jira/browse/HDFS-8147 Project: Hadoop HDFS Issue Type: Bug Components: balancer mover Affects Versions: 2.6.0 Reporter: surendra singh lilhore Assignee: surendra singh lilhore Attachments: HDFS-8147.patch, HDFS-8147_1.patch, HDFS-8147_2.patch, HDFS-8147_3.patch, HDFS-8147_4.patch *Scenario:* 1. Three DN cluster. For DNs storage type is like this. DN1 : DISK,ARCHIVE DN2 : DISK DN3 : DISK,ARCHIVE (All DNs are in same rack) 2. One file with two replicas (In DN1 and DN2) 3. Set file storage policy COLD 4. Now execute Mover. *Expected Result:* File blocks should move in DN1:ARCHIVE and DN3:ARCHIVE *Actual Result:* {{chooseTargetInSameNode()}} move D1:DISK block to D1:ARCHIVE, but in next iteration {{chooseTarget()}} for same rake is selecting again DN1:ARCHIVE for target where already same block exists. {{chooseTargetInSameNode()}} and {{chooseTarget()}} should not select the node as target where already same replica exists. The dispatcher will fail to move block as shown in the log below. Then, the Mover will try again in next iteration. *Logs* {code} 15/04/15 10:47:17 WARN balancer.Dispatcher: Failed to move blk_1073741852_1028 with size=11990 from 10.19.92.74:50010:DISK to 10.19.92.73:50010:ARCHIVE through 10.19.92.73:50010: Got error, status message opReplaceBlock BP-1258709199-10.19.92.74-1428292615636:blk_1073741852_1028 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Replica FinalizedReplica, blk_1073741852_1028, FINALIZED {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8147) Mover should not schedule two replicas to the same DN storage
[ https://issues.apache.org/jira/browse/HDFS-8147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-8147: -- Description: *Scenario:* 1. Three DN cluster. For DNs storage type is like this. DN1 : DISK,ARCHIVE DN2 : DISK DN3 : DISK,ARCHIVE (All DNs are in same rack) 2. One file with two replicas (In DN1 and DN2) 3. Set file storage policy COLD 4. Now execute Mover. *Expected Result:* File blocks should move in DN1:ARCHIVE and DN3:ARCHIVE *Actual Result:* {{chooseTargetInSameNode()}} move D1:DISK block to D1:ARCHIVE, but in next iteration {{chooseTarget()}} for same rake is selecting again DN1:ARCHIVE for target where already same block exists. {{chooseTargetInSameNode()}} and {{chooseTarget()}} should not select the node as target where already same replica exists. The dispatcher will fail to move block as shown in the log below. Then, the Mover will try again in next iteration. *Logs* {code} 15/04/15 10:47:17 WARN balancer.Dispatcher: Failed to move blk_1073741852_1028 with size=11990 from 10.19.92.74:50010:DISK to 10.19.92.73:50010:ARCHIVE through 10.19.92.73:50010: Got error, status message opReplaceBlock BP-1258709199-10.19.92.74-1428292615636:blk_1073741852_1028 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Replica FinalizedReplica, blk_1073741852_1028, FINALIZED {code} was: *Scenario:* 1. Three DN cluster. For DNs storage type is like this. DN1 : DISK,ARCHIVE DN2 : DISK DN3 : DISK,ARCHIVE (All DNs are in same rack) 2. One file with two replicas (In DN1 and DN2) 3. Set file storage policy COLD 4. Now execute Mover. *Expected Result:* File blocks should move in DN1:ARCHIVE and DN3:ARCHIVE *Actual Result:* {{chooseTargetInSameNode()}} move D1:DISK block to D1:ARCHIVE, but in next iteration {{chooseTarget()}} for same rake is selecting again DN1:ARCHIVE for target where already same block exists. {{chooseTargetInSameNode()}} and {{chooseTarget()}} should not select the node as target where already same replica exists. *Logs* {code} 15/04/15 10:47:17 WARN balancer.Dispatcher: Failed to move blk_1073741852_1028 with size=11990 from 10.19.92.74:50010:DISK to 10.19.92.73:50010:ARCHIVE through 10.19.92.73:50010: Got error, status message opReplaceBlock BP-1258709199-10.19.92.74-1428292615636:blk_1073741852_1028 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Replica FinalizedReplica, blk_1073741852_1028, FINALIZED {code} Summary: Mover should not schedule two replicas to the same DN storage (was: Mover should not select the DN storage as target where already same replica exists.) Revised summary and description. Mover should not schedule two replicas to the same DN storage - Key: HDFS-8147 URL: https://issues.apache.org/jira/browse/HDFS-8147 Project: Hadoop HDFS Issue Type: Bug Components: balancer mover Affects Versions: 2.6.0 Reporter: surendra singh lilhore Assignee: surendra singh lilhore Attachments: HDFS-8147.patch, HDFS-8147_1.patch, HDFS-8147_2.patch, HDFS-8147_3.patch *Scenario:* 1. Three DN cluster. For DNs storage type is like this. DN1 : DISK,ARCHIVE DN2 : DISK DN3 : DISK,ARCHIVE (All DNs are in same rack) 2. One file with two replicas (In DN1 and DN2) 3. Set file storage policy COLD 4. Now execute Mover. *Expected Result:* File blocks should move in DN1:ARCHIVE and DN3:ARCHIVE *Actual Result:* {{chooseTargetInSameNode()}} move D1:DISK block to D1:ARCHIVE, but in next iteration {{chooseTarget()}} for same rake is selecting again DN1:ARCHIVE for target where already same block exists. {{chooseTargetInSameNode()}} and {{chooseTarget()}} should not select the node as target where already same replica exists. The dispatcher will fail to move block as shown in the log below. Then, the Mover will try again in next iteration. *Logs* {code} 15/04/15 10:47:17 WARN balancer.Dispatcher: Failed to move blk_1073741852_1028 with size=11990 from 10.19.92.74:50010:DISK to 10.19.92.73:50010:ARCHIVE through 10.19.92.73:50010: Got error, status message opReplaceBlock BP-1258709199-10.19.92.74-1428292615636:blk_1073741852_1028 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Replica FinalizedReplica, blk_1073741852_1028, FINALIZED {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)