[ 
https://issues.apache.org/jira/browse/HDFS-11022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15585313#comment-15585313
 ] 

Wei-Chiu Chuang commented on HDFS-11022:
----------------------------------------

FWIW, I am posting the relevent logs below:

bq. In the meantime, the replica is appended, updating its generation stamp 
from x to y.

2016-10-12 09:43:12,988 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: 
Receiving BP-1092022411-10.0.0.55-1474407949037:blk_1073803461_73556 src: 
/10.0.0.64:45516 dest: /10.0.0.58:50010

2016-10-12 09:43:12,988 INFO 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Appending 
to FinalizedReplica, blk_1073803461_73556, FINALIZED
  getNumBytes()     = 46585252
  getBytesOnDisk()  = 46585252
  getVisibleLength()= 46585252
  getVolume()       = /data/3/dfs/dn/current
  getBlockFile()    = 
/data/3/dfs/dn/current/BP-1092022411-10.0.0.55-1474407949037/current/finalized/subdir0/subdir240/blk_1073803461

2016-10-12 09:43:13,018 INFO 
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: 
/10.0.0.64:45516, dest: /10.0.0.58:50010, bytes: 47014152, op: HDFS_WRITE, 
cliID: DFSClient_NONMAPREDUCE_-957820327_243, offset: 0, srvID: 
bad3566e-51d0-4002-8504-a0a14df73450, blockid: 
BP-1092022411-10.0.0.55-1474407949037:blk_1073803461_73598, duration: 13117326

bq. The client tells NN to mark the replica blk_A_x corrupt.
2016-10-12 09:43:13,885 INFO BlockStateChange: BLOCK 
NameSystem.addToCorruptReplicasMap: blk_1073803461 added as corrupt on 
10.0.0.58:50010 by /10.0.0.58  because client machine reported it

bq. NN tells the data node to (1) delete replica blk_A_x and (2) replicate the 
newer replica blk_A_y from another datanode. Due to block placement policy, 
blk_A_y is replicated to the same node. (It's a small cluster)

2016-10-12 09:43:13,886 INFO BlockStateChange: BLOCK* invalidateBlock: 
blk_1073803461_73556(stored=blk_1073803461_73598) on 10.0.0.58:50010
 2016-10-12 09:43:13,886 INFO BlockStateChange: BLOCK* InvalidateBlocks: add 
blk_1073803461_73556 to 10.0.0.58:50010
 2016-10-12 09:43:16,789 INFO BlockStateChange: BLOCK* ask 10.0.0.64:50010 to 
replicate blk_1073803461_73598 to datanode(s) 10.0.0.58:50010
2016-10-12 09:43:16,789 INFO BlockStateChange: BLOCK* BlockManager: ask 
10.0.0.58:50010 to delete [blk_1073803461_73556]

bq. DN is unable to receive the newer replica blk_A_y, because the replica 
already exists.

(The source of transfer)
2016-10-12 09:43:13,022 INFO 
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: 
/10.0.0.54:45890, dest: /10.0.0.64:50010, bytes: 47014152, op: HDFS_WRITE, 
cliID: DFSClient_NONMAPREDUCE_-957820327_243, offset: 0, srvID: 
998d2a6f-048e-4829-9bc4-38d383631304, blockid: 
BP-1092022411-10.0.0.55-1474407949037:blk_1073803461_73598, duration: 13844523

2016-10-12 09:43:13,022 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: 
PacketResponder: BP-1092022411-10.0.0.55-1474407949037:blk_1073803461_73598, 
type=HAS_DOWNSTREAM_IN_PIPELINE terminating

2016-10-12 09:43:17,078 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: 
DatanodeRegistration(10.0.0.64, 
datanodeUuid=998d2a6f-048e-4829-9bc4-38d383631304, infoPort=50075, 
infoSecurePort=0, ipcPort=50020, 
storageInfo=lv=-56;cid=cluster16;nsid=2009955746;c=0) Starting thread to 
transfer BP-1092022411-10.0.0.55-1474407949037:blk_1073803461_73598 to 
10.0.0.58:5001010.0.0.

2016-10-12 09:43:17,082 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: 
DatanodeRegistration(10.0.0.64, 
datanodeUuid=998d2a6f-048e-4829-9bc4-38d383631304, infoPort=50075, 
infoSecurePort=0, ipcPort=50020, 
storageInfo=lv=-56;cid=cluster16;nsid=2009955746;c=0):Failed to transfer 
BP-1092022411-10.0.0.55-1474407949037:blk_1073803461_73598 to 10.0.0.58:50010 
got

java.net.SocketException: Original Exception : java.io.IOException: Connection 
reset by peer

       at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)

       at 
sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433)

       at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565)

       at 
org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:223)

       at 
org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:578)

       at 
org.apache.hadoop.hdfs.server.datanode.BlockSender.doSendBlock(BlockSender.java:758)

       at 
org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:705)

       at 
org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run(DataNode.java:2170)

       at java.lang.Thread.run(Thread.java:745)

Caused by: java.io.IOException: Connection reset by peer

       ... 9 more
(The destination of transfer)
2016-10-12 09:43:12,988 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: 
Receiving BP-1092022411-10.0.0.55-1474407949037:blk_1073803461_73556 src: 
/10.0.0.64:45516 dest: /10.0.0.58:50010

2016-10-12 09:43:12,988 INFO 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Appending 
to FinalizedReplica, blk_1073803461_73556, FINALIZED

 getNumBytes()     = 46585252

 getBytesOnDisk()  = 46585252

 getVisibleLength()= 46585252

 getVolume()       = /data/3/dfs/dn/current

 getBlockFile()    = 
/data/3/dfs/dn/current/BP-1092022411-10.0.0.55-1474407949037/current/finalized/subdir0/subdir240/blk_1073803461

2016-10-12 09:43:13,018 INFO 
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: 
/10.0.0.64:45516, dest: /10.0.0.58:50010, bytes: 47014152, op: HDFS_WRITE, 
cliID: DFSClient_NONMAPREDUCE_-957820327_243, offset: 0, srvID: 
bad3566e-51d0-4002-8504-a0a14df73450, blockid: 
BP-1092022411-10.0.0.55-1474407949037:blk_1073803461_73598, duration: 
1311732610.0.0.10.0.0.

2016-10-12 09:43:13,018 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: 
PacketResponder: BP-1092022411-10.0.0.55-1474407949037:blk_1073803461_73598, 
type=HAS_DOWNSTREAM_IN_PIPELINE terminating

2016-10-12 09:43:17,078 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: 
Receiving BP-1092022411-10.0.0.55-1474407949037:blk_1073803461_73598 src: 
/10.0.0.64:45526 dest: /10.0.0.58:5001010.0.0.10.0.0.

2016-10-12 09:43:17,078 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: 
opWriteBlock BP-1092022411-10.0.0.55-1474407949037:blk_1073803461_73598 
received exception 
org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Block 
BP-1092022411-10.0.0.55-1474407949037:blk_1073803461_73598 already exists in 
state FINALIZED and thus cannot be created.


2016-10-12 09:43:17,078 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: 
d03nappp0090.corp.chartercom.com:50010:DataXceiver error processing WRITE_BLOCK 
operation  src: /10.0.0.64:45526 dst: /10.0.0.58:50010; 
org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Block 
BP-1092022411-10.0.0.55-1474407949037:blk_1073803461_73598 already exists in 
state FINALIZED and thus cannot be created.10.0.0.10.0.0.10.0.0.


bq. DN is also unable to delete replica blk_A_y because blk_A_y does not exist.

2016-10-12 09:43:17,319 INFO 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Failed to 
delete replica blk_1073803461_73556: ReplicaInfo not found.

bq. The replica on the DN is not part of data pipeline, so it becomes stale.
2016-10-12 09:44:17,906 INFO 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: 
updatePipeline(block=BP-1092022411-10.0.0.55-1474407949037:blk_1073803461_73598,
 newGenerationStamp=73638, newLength=47014152, newNodes=[10.0.0.64:50010, 
10.0.0.63:50010, 10.0.0.56:50010], 
clientName=DFSClient_NONMAPREDUCE_1477780923_243)

> DataNode unable to remove corrupt block replica due to race condition
> ---------------------------------------------------------------------
>
>                 Key: HDFS-11022
>                 URL: https://issues.apache.org/jira/browse/HDFS-11022
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: datanode, namenode
>    Affects Versions: 2.6.0
>         Environment: CDH5.7.0
>            Reporter: Wei-Chiu Chuang
>            Priority: Critical
>         Attachments: HDFS-11022.png
>
>
> Scenario:
> # A client reads a replica blk_A_x from a data node and detected corruption.
> # In the meantime, the replica is appended, updating its generation stamp 
> from x to y.
> # The client tells NN to mark the replica blk_A_x corrupt.
> # NN tells the data node to (1) delete replica blk_A_x and (2) replicate the 
> newer replica blk_A_y from another datanode. Due to block placement policy, 
> blk_A_y is replicated to the same node. (It's a small cluster)
> # DN is unable to receive the newer replica blk_A_y, because the replica 
> already exists.
> # DN is also unable to delete replica blk_A_y because blk_A_y does not exist.
> # The replica on the DN is not part of data pipeline, so it becomes stale.
> If another replica becomes corrupt and NameNode wants to replicate a healthy 
> replica to this DataNode, it can't, because a stale replica exists. Because 
> this is a small cluster, soon enough (in a matter of a hour) no DataNode is 
> able to receive a healthy replica.
> This cluster also suffers from HDFS-11019, so even though DataNode later 
> detected data corruption, it was unable to report to NameNode.
> Note that we are still investigating the root cause of the corruption.
> The access pattern of client is through Httpfs, and it appended to finalized 
> blocks and then finalize the block quickly. It's not long running pipeline.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

Reply via email to