[
https://issues.apache.org/jira/browse/HDFS-9717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Li updated HDFS-9717:
-----------------------------
Component/s: namenode
> NameNode can not update the status of bad block
> -----------------------------------------------
>
> Key: HDFS-9717
> URL: https://issues.apache.org/jira/browse/HDFS-9717
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
> Affects Versions: 2.2.0
> Reporter: tangshangwen
> Assignee: tangshangwen
>
> In our cluster，some users set the number of replicas of file to 1, then back
> to 2，the file cannot be read，but the NameNode think it is healthy
> {noformat}
> /user/username/dt=2015-11-30/dp=16/part-r-00063.lzo 1513716944 bytes, 12
> block(s): Under replicated BP-1422437282658:blk_1897961957_824575827. Target
> Replicas is 2 but found 1 replica(s).
> Replica placement policy is violated for
> BP-1422437282658:blk_1897961957_824575827. Block should be additionally
> replicated on 1 more rack
> (s).
> 0. BP-1337805335-xxx.xxx.xxx.xxx-1422437282658:blk_1897961824_824575694
> len=134217728 repl=2 [host1:50010, host2:50010]
> 1. BP-1337805335-xxx.xxx.xxx.xxx-1422437282658:blk_1897961957_824575827
> len=134217728 repl=1 [host3:50010]
> 2. BP-1337805335-xxx.xxx.xxx.xxx-1422437282658:blk_1897962047_824575917
> len=134217728 repl=2 [host4:50010, host1:50010]
> ......
> Status: HEALTHY
> Total size: 1513716944 B
> Total dirs: 0
> Total files: 1
> Total symlinks: 0
> Total blocks (validated): 12 (avg. block size 126143078 B)
> Minimally replicated blocks: 12 (100.0 %)
> Over-replicated blocks: 0 (0.0 %)
> Under-replicated blocks: 1 (8.333333 %)
> Mis-replicated blocks: 1 (8.333333 %)
> Default replication factor: 3
> Average block replication: 1.9166666
> Corrupt blocks: 0
> Missing replicas: 1 (4.1666665 %)
> Number of data-nodes: xxxx
> Number of racks: xxx
> FSCK ended at Thu Jan 28 10:27:49 CST 2016 in 0 milliseconds
> {noformat}
> But the replica on the datanode has been damaged, can't read，this is
> datanode log
> {noformat}
> 2016-01-23 06:34:42,737 WARN
> org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: First
> Verification failed for
> BP-1337805335-xxx.xxx.xxx.xxx-1422437282658:blk_1897961957_824575827
>
>
> java.io.IOException: Input/output error
>
>
> at java.io.FileInputStream.readBytes(Native Method)
>
>
> at java.io.FileInputStream.read(FileInputStream.java:272)
>
>
> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:192)
>
>
> at
> org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:529)
>
>
> at
> org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:710)
>
>
> at
> org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.verifyBlock(BlockPoolSliceScanner.java:427)
>
> at
> org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.verifyFirstBlock(BlockPoolSliceScanner.java:506)
>
> at
> org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.scan(BlockPoolSliceScanner.java:667)
>
> at
> org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.scanBlockPoolSlice(BlockPoolSliceScanner.java:633)
>
> at
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:101)
>
> at java.lang.Thread.run(Thread.java:745)
> ------------------------------
> 2016-01-28 10:28:37,874 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> DatanodeRegistration(host1,
> storageID=DS-1450783279-xxx.xxx.xxx.xxx-50010-1432889625435
> , infoPort=50075, ipcPort=50020,
> storageInfo=lv=-47;cid=CID-3f36397d-b160-4414-b7e4-f37b72e96d53;nsid=1992344832;c=0):Failed
> to transfer BP-1337805335-xxx.xxx.xxx.xxx-142243
> 7282658:blk_1897961957_824575827 to xxx.xxx.xxx.xxx:50010 got
> java.net.SocketException: Original Exception : java.io.IOException:
> Input/output error
> at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
> at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433)
> at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565)
> at
> org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:223)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:546)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:710)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run(DataNode.java:1520)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Input/output error
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
