tangshangwen created HDFS-9717:
----------------------------------
Summary: NameNode can not update the status of bad block
Key: HDFS-9717
URL: https://issues.apache.org/jira/browse/HDFS-9717
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.2.0
Reporter: tangshangwen
Assignee: tangshangwen
In our cluster，some users set the number of replicas of file to 1, then back to
2，the file cannot be read，but the NameNode think it is healthy
{noformat}
/user/username/dt=2015-11-30/dp=16/part-r-00063.lzo 1513716944 bytes, 12
block(s): Under replicated BP-1422437282658:blk_1897961957_824575827. Target
Replicas is 2 but found 1 replica(s).
Replica placement policy is violated for
BP-1422437282658:blk_1897961957_824575827. Block should be additionally
replicated on 1 more rack
(s).
0. BP-1337805335-xxx.xxx.xxx.xxx-1422437282658:blk_1897961824_824575694
len=134217728 repl=2 [host1:50010, host2:50010]
1. BP-1337805335-xxx.xxx.xxx.xxx-1422437282658:blk_1897961957_824575827
len=134217728 repl=1 [host3:50010]
2. BP-1337805335-xxx.xxx.xxx.xxx-1422437282658:blk_1897962047_824575917
len=134217728 repl=2 [host4:50010, host1:50010]
......
Status: HEALTHY
Total size: 1513716944 B
Total dirs: 0
Total files: 1
Total symlinks: 0
Total blocks (validated): 12 (avg. block size 126143078 B)
Minimally replicated blocks: 12 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 1 (8.333333 %)
Mis-replicated blocks: 1 (8.333333 %)
Default replication factor: 3
Average block replication: 1.9166666
Corrupt blocks: 0
Missing replicas: 1 (4.1666665 %)
Number of data-nodes: xxxx
Number of racks: xxx
FSCK ended at Thu Jan 28 10:27:49 CST 2016 in 0 milliseconds
{noformat}
But the replica on the datanode has been damaged, can't read，this is datanode
log
{noformat}
2016-01-23 06:34:42,737 WARN
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: First
Verification failed for
BP-1337805335-xxx.xxx.xxx.xxx-1422437282658:blk_1897961957_824575827
java.io.IOException: Input/output error
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:272)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:192)
at
org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:529)
at
org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:710)
at
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.verifyBlock(BlockPoolSliceScanner.java:427)
at
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.verifyFirstBlock(BlockPoolSliceScanner.java:506)
at
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.scan(BlockPoolSliceScanner.java:667)
at
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.scanBlockPoolSlice(BlockPoolSliceScanner.java:633)
at
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:101)
at java.lang.Thread.run(Thread.java:745)
------------------------------
2016-01-28 10:28:37,874 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
DatanodeRegistration(host1,
storageID=DS-1450783279-xxx.xxx.xxx.xxx-50010-1432889625435
, infoPort=50075, ipcPort=50020,
storageInfo=lv=-47;cid=CID-3f36397d-b160-4414-b7e4-f37b72e96d53;nsid=1992344832;c=0):Failed
to transfer BP-1337805335-xxx.xxx.xxx.xxx-142243
7282658:blk_1897961957_824575827 to xxx.xxx.xxx.xxx:50010 got
java.net.SocketException: Original Exception : java.io.IOException:
Input/output error
at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433)
at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565)
at
org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:223)
at
org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:546)
at
org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:710)
at
org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run(DataNode.java:1520)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Input/output error
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
