[jira] [Created] (HDFS-8208) Namenode reports stale block locations during decommissioning

James Mulcahy (JIRA) Tue, 21 Apr 2015 10:07:34 -0700

James Mulcahy created HDFS-8208:
-----------------------------------

             Summary: Namenode reports stale block locations during 
decommissioning 
                 Key: HDFS-8208
                 URL: https://issues.apache.org/jira/browse/HDFS-8208
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: namenode
    Affects Versions: 2.5.0
            Reporter: James Mulcahy



While decommissioning some datanodes, we observed OP_READ_BLOCK failures from 
DFSClient instances.  Tracing back the history in the logs reveals that the 
DFSClient instance was started after the block in question had been migrated 
off the DataNode that was being accessed at the time of the OP_READ_BLOCK 
failure.  It appears that the NameNode, despite having orchestrated the 
migration of the data, continued to provide a DataNode as a valid source of a 
block, despite it no longer being present.

Timeline:

The DN deleted the block at 09:43:20.
The DFSClient instance was started at 09:46:33.

DFS Client Error was:

15/03/25 09:46:35 WARN hdfs.DFSClient: Failed to connect to 
/IP.PARTLY.MASKED.192:50010 for block, add to deadNodes and continue. 
java.io.IOException: Got error for OP_READ_BLOCK, 
self=/IP.PARTLY.MASKED.24:37956, remote=/IP.PARTLY.MASKED.192:50010, for file 
/bundles/bundle-15edf17968d34583a7d24b30fb7e73c6, for pool 
BP-2091375182-IP.PARTLY.MASKED.219-1423771577555 block 1078309397_4569046
java.io.IOException: Got error for OP_READ_BLOCK, 
self=/IP.PARTLY.MASKED.24:37956, remote=/IP.PARTLY.MASKED.192:50010, for file 
/bundles/bundle-15edf17968d34583a7d24b30fb7e73c6, for pool 
BP-2091375182-17.220.247.219-1423771577555 block 1078309397_4569046
        at 
org.apache.hadoop.hdfs.RemoteBlockReader2.checkSuccess(RemoteBlockReader2.java:445)
        at 
org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:410)
        at 
org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReader(BlockReaderFactory.java:787)
        at 
org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:666)
        at 
org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:326)
        at 
org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:570)
        at 
org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:793)
        at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:840)
        at java.io.DataInputStream.read(DataInputStream.java:100)
        at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:84)
        at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:52)
        at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:112)
        at 
org.apache.hadoop.fs.shell.CommandWithDestination$TargetFileSystem.writeStreamToFile(CommandWithDestination.java:395)
        at 
org.apache.hadoop.fs.shell.CommandWithDestination.copyStreamToTarget(CommandWithDestination.java:327)
        at 
org.apache.hadoop.fs.shell.CommandWithDestination.copyFileToTarget(CommandWithDestination.java:303)
        at 
org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:243)
        at 
org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:228)
        at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:306)
        at 
org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:278)
        at 
org.apache.hadoop.fs.shell.CommandWithDestination.processPathArgument(CommandWithDestination.java:223)
        at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:260)
        at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:244)
        at 
org.apache.hadoop.fs.shell.CommandWithDestination.processArguments(CommandWithDestination.java:200)
        at 
org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:190)
        at org.apache.hadoop.fs.shell.Command.run(Command.java:154)
        at org.apache.hadoop.fs.FsShell.run(FsShell.java:287)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
        at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)

Here's the log from the DataNode, for that block:

grep 1078309397_4569046 hdfs_datanode.out 
  block  
=BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
newGs=0, bytesRcvd=[0, 0]
15/03/24 12:54:12 INFO datanode.DataNode: Receiving 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 src: 
/17.219.209.92:49184 dest: /IP.PARTLY.MASKED4.192:50010
15/03/24 12:54:12 DEBUG datanode.DataNode: BlockReceiver: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/24 12:54:12 DEBUG datanode.DataNode: writeTo metafile is 
/Volumes/SimcloudRuntime/simcloud/runtime/data/hdfs/datanode/current/BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555/current/rbw/blk_1078309397_4569046.meta
 of size 0
15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: 
PacketHeader with packetLen=516 header data: offsetInBlock: 0
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=107, 
lastPacketInBlock=false, offsetInBlock=512, 
ackEnqueueNanoTime=1427226852662586000, ackStatus=SUCCESS)
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 107 status: SUCCESS status: SUCCESS 
downstreamAckTimeNanos: 473000
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 107 status: SUCCESS status: 
SUCCESS status: SUCCESS downstreamAckTimeNanos: 932000
15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: 
PacketHeader with packetLen=65532 header data: offsetInBlock: 512
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=108, 
lastPacketInBlock=false, offsetInBlock=65536, 
ackEnqueueNanoTime=1427226852670673000, ackStatus=SUCCESS)
15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: 
PacketHeader with packetLen=65532 header data: offsetInBlock: 65536
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=109, 
lastPacketInBlock=false, offsetInBlock=130560, 
ackEnqueueNanoTime=1427226852672795000, ackStatus=SUCCESS)
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 108 status: SUCCESS status: SUCCESS 
downstreamAckTimeNanos: 1820000
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 108 status: SUCCESS status: 
SUCCESS status: SUCCESS downstreamAckTimeNanos: 3791000
15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: 
PacketHeader with packetLen=65532 header data: offsetInBlock: 130560
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=110, 
lastPacketInBlock=false, offsetInBlock=195584, 
ackEnqueueNanoTime=1427226852674869000, ackStatus=SUCCESS)
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 109 status: SUCCESS status: SUCCESS 
downstreamAckTimeNanos: 1105000
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 109 status: SUCCESS status: 
SUCCESS status: SUCCESS downstreamAckTimeNanos: 2186000
15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: 
PacketHeader with packetLen=65532 header data: offsetInBlock: 195584
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=111, 
lastPacketInBlock=false, offsetInBlock=260608, 
ackEnqueueNanoTime=1427226852676604000, ackStatus=SUCCESS)
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 110 status: SUCCESS status: SUCCESS 
downstreamAckTimeNanos: 1361000
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 110 status: SUCCESS status: 
SUCCESS status: SUCCESS downstreamAckTimeNanos: 2484000
15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: 
PacketHeader with packetLen=65532 header data: offsetInBlock: 260608
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=112, 
lastPacketInBlock=false, offsetInBlock=325632, 
ackEnqueueNanoTime=1427226852678501000, ackStatus=SUCCESS)
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 111 status: SUCCESS status: SUCCESS 
downstreamAckTimeNanos: 1136000
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 111 status: SUCCESS status: 
SUCCESS status: SUCCESS downstreamAckTimeNanos: 2120000
15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: 
PacketHeader with packetLen=65532 header data: offsetInBlock: 325632
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=113, 
lastPacketInBlock=false, offsetInBlock=390656, 
ackEnqueueNanoTime=1427226852679889000, ackStatus=SUCCESS)
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 112 status: SUCCESS status: SUCCESS 
downstreamAckTimeNanos: 1157000
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 112 status: SUCCESS status: 
SUCCESS status: SUCCESS downstreamAckTimeNanos: 2118000
15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: 
PacketHeader with packetLen=65532 header data: offsetInBlock: 390656
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=114, 
lastPacketInBlock=false, offsetInBlock=455680, 
ackEnqueueNanoTime=1427226852681499000, ackStatus=SUCCESS)
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 113 status: SUCCESS status: SUCCESS 
downstreamAckTimeNanos: 1146000
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 113 status: SUCCESS status: 
SUCCESS status: SUCCESS downstreamAckTimeNanos: 2219000
15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: 
PacketHeader with packetLen=65532 header data: offsetInBlock: 455680
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=115, 
lastPacketInBlock=false, offsetInBlock=520704, 
ackEnqueueNanoTime=1427226852683270000, ackStatus=SUCCESS)
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 114 status: SUCCESS status: SUCCESS 
downstreamAckTimeNanos: 1139000
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 114 status: SUCCESS status: 
SUCCESS status: SUCCESS downstreamAckTimeNanos: 2084000
15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: 
PacketHeader with packetLen=65532 header data: offsetInBlock: 520704
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=116, 
lastPacketInBlock=false, offsetInBlock=585728, 
ackEnqueueNanoTime=1427226852684984000, ackStatus=SUCCESS)
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 115 status: SUCCESS status: SUCCESS 
downstreamAckTimeNanos: 1248000
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 115 status: SUCCESS status: 
SUCCESS status: SUCCESS downstreamAckTimeNanos: 2329000
15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: 
PacketHeader with packetLen=65532 header data: offsetInBlock: 585728
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=117, 
lastPacketInBlock=false, offsetInBlock=650752, 
ackEnqueueNanoTime=1427226852686598000, ackStatus=SUCCESS)
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 116 status: SUCCESS status: SUCCESS 
downstreamAckTimeNanos: 1192000
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 116 status: SUCCESS status: 
SUCCESS status: SUCCESS downstreamAckTimeNanos: 2234000
15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: 
PacketHeader with packetLen=65532 header data: offsetInBlock: 650752
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=118, 
lastPacketInBlock=false, offsetInBlock=715776, 
ackEnqueueNanoTime=1427226852687973000, ackStatus=SUCCESS)
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 117 status: SUCCESS status: SUCCESS 
downstreamAckTimeNanos: 1168000
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 117 status: SUCCESS status: 
SUCCESS status: SUCCESS downstreamAckTimeNanos: 2150000
15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: 
PacketHeader with packetLen=65532 header data: offsetInBlock: 715776
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=119, 
lastPacketInBlock=false, offsetInBlock=780800, 
ackEnqueueNanoTime=1427226852689858000, ackStatus=SUCCESS)
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 118 status: SUCCESS status: SUCCESS 
downstreamAckTimeNanos: 1149000
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 118 status: SUCCESS status: 
SUCCESS status: SUCCESS downstreamAckTimeNanos: 2223000
15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: 
PacketHeader with packetLen=65532 header data: offsetInBlock: 780800
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=120, 
lastPacketInBlock=false, offsetInBlock=845824, 
ackEnqueueNanoTime=1427226852691199000, ackStatus=SUCCESS)
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 119 status: SUCCESS status: SUCCESS 
downstreamAckTimeNanos: 1197000
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 119 status: SUCCESS status: 
SUCCESS status: SUCCESS downstreamAckTimeNanos: 2194000
15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: 
PacketHeader with packetLen=65532 header data: offsetInBlock: 845824
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=121, 
lastPacketInBlock=false, offsetInBlock=910848, 
ackEnqueueNanoTime=1427226852693079000, ackStatus=SUCCESS)
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 120 status: SUCCESS status: SUCCESS 
downstreamAckTimeNanos: 1171000
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 120 status: SUCCESS status: 
SUCCESS status: SUCCESS downstreamAckTimeNanos: 2215000
15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: 
PacketHeader with packetLen=65532 header data: offsetInBlock: 910848
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=122, 
lastPacketInBlock=false, offsetInBlock=975872, 
ackEnqueueNanoTime=1427226852694657000, ackStatus=SUCCESS)
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 121 status: SUCCESS status: SUCCESS 
downstreamAckTimeNanos: 1190000
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 121 status: SUCCESS status: 
SUCCESS status: SUCCESS downstreamAckTimeNanos: 2189000
15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: 
PacketHeader with packetLen=65532 header data: offsetInBlock: 975872
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=123, 
lastPacketInBlock=false, offsetInBlock=1040896, 
ackEnqueueNanoTime=1427226852696164000, ackStatus=SUCCESS)
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 122 status: SUCCESS status: SUCCESS 
downstreamAckTimeNanos: 1223000
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 122 status: SUCCESS status: 
SUCCESS status: SUCCESS downstreamAckTimeNanos: 2318000
15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: 
PacketHeader with packetLen=65532 header data: offsetInBlock: 1040896
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=124, 
lastPacketInBlock=false, offsetInBlock=1105920, 
ackEnqueueNanoTime=1427226852697883000, ackStatus=SUCCESS)
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 123 status: SUCCESS status: SUCCESS 
downstreamAckTimeNanos: 1041000
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 123 status: SUCCESS status: 
SUCCESS status: SUCCESS downstreamAckTimeNanos: 2090000
15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: 
PacketHeader with packetLen=65532 header data: offsetInBlock: 1105920
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=125, 
lastPacketInBlock=false, offsetInBlock=1170944, 
ackEnqueueNanoTime=1427226852699500000, ackStatus=SUCCESS)
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 124 status: SUCCESS status: SUCCESS 
downstreamAckTimeNanos: 1341000
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 124 status: SUCCESS status: 
SUCCESS status: SUCCESS downstreamAckTimeNanos: 2489000
15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: 
PacketHeader with packetLen=65532 header data: offsetInBlock: 1170944
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=126, 
lastPacketInBlock=false, offsetInBlock=1235968, 
ackEnqueueNanoTime=1427226852701121000, ackStatus=SUCCESS)
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 125 status: SUCCESS status: SUCCESS 
downstreamAckTimeNanos: 1240000
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 125 status: SUCCESS status: 
SUCCESS status: SUCCESS downstreamAckTimeNanos: 2355000
15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: 
PacketHeader with packetLen=65532 header data: offsetInBlock: 1235968
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=127, 
lastPacketInBlock=false, offsetInBlock=1300992, 
ackEnqueueNanoTime=1427226852702358000, ackStatus=SUCCESS)
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 126 status: SUCCESS status: SUCCESS 
downstreamAckTimeNanos: 1171000
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 126 status: SUCCESS status: 
SUCCESS status: SUCCESS downstreamAckTimeNanos: 2209000
15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: 
PacketHeader with packetLen=65532 header data: offsetInBlock: 1300992
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=128, 
lastPacketInBlock=false, offsetInBlock=1366016, 
ackEnqueueNanoTime=1427226852704072000, ackStatus=SUCCESS)
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 127 status: SUCCESS status: SUCCESS 
downstreamAckTimeNanos: 1328000
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 127 status: SUCCESS status: 
SUCCESS status: SUCCESS downstreamAckTimeNanos: 2333000
15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: 
PacketHeader with packetLen=65532 header data: offsetInBlock: 1366016
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=129, 
lastPacketInBlock=false, offsetInBlock=1431040, 
ackEnqueueNanoTime=1427226852705571000, ackStatus=SUCCESS)
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 128 status: SUCCESS status: SUCCESS 
downstreamAckTimeNanos: 1217000
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 128 status: SUCCESS status: 
SUCCESS status: SUCCESS downstreamAckTimeNanos: 2230000
15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: 
PacketHeader with packetLen=65532 header data: offsetInBlock: 1431040
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=130, 
lastPacketInBlock=false, offsetInBlock=1496064, 
ackEnqueueNanoTime=1427226852707314000, ackStatus=SUCCESS)
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 129 status: SUCCESS status: SUCCESS 
downstreamAckTimeNanos: 1305000
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 129 status: SUCCESS status: 
SUCCESS status: SUCCESS downstreamAckTimeNanos: 2349000
15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: 
PacketHeader with packetLen=65532 header data: offsetInBlock: 1496064
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=131, 
lastPacketInBlock=false, offsetInBlock=1561088, 
ackEnqueueNanoTime=1427226852708544000, ackStatus=SUCCESS)
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 130 status: SUCCESS status: SUCCESS 
downstreamAckTimeNanos: 1276000
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 130 status: SUCCESS status: 
SUCCESS status: SUCCESS downstreamAckTimeNanos: 2294000
15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: 
PacketHeader with packetLen=65532 header data: offsetInBlock: 1561088
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=132, 
lastPacketInBlock=false, offsetInBlock=1626112, 
ackEnqueueNanoTime=1427226852710484000, ackStatus=SUCCESS)
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 131 status: SUCCESS status: SUCCESS 
downstreamAckTimeNanos: 1251000
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 131 status: SUCCESS status: 
SUCCESS status: SUCCESS downstreamAckTimeNanos: 2323000
15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: 
PacketHeader with packetLen=65532 header data: offsetInBlock: 1626112
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=133, 
lastPacketInBlock=false, offsetInBlock=1691136, 
ackEnqueueNanoTime=1427226852711771000, ackStatus=SUCCESS)
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 132 status: SUCCESS status: SUCCESS 
downstreamAckTimeNanos: 1223000
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 132 status: SUCCESS status: 
SUCCESS status: SUCCESS downstreamAckTimeNanos: 2278000
15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: 
PacketHeader with packetLen=65532 header data: offsetInBlock: 1691136
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=134, 
lastPacketInBlock=false, offsetInBlock=1756160, 
ackEnqueueNanoTime=1427226852713759000, ackStatus=SUCCESS)
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 133 status: SUCCESS status: SUCCESS 
downstreamAckTimeNanos: 1131000
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 133 status: SUCCESS status: 
SUCCESS status: SUCCESS downstreamAckTimeNanos: 2237000
15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: 
PacketHeader with packetLen=65532 header data: offsetInBlock: 1756160
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=135, 
lastPacketInBlock=false, offsetInBlock=1821184, 
ackEnqueueNanoTime=1427226852715109000, ackStatus=SUCCESS)
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 134 status: SUCCESS status: SUCCESS 
downstreamAckTimeNanos: 1170000
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 134 status: SUCCESS status: 
SUCCESS status: SUCCESS downstreamAckTimeNanos: 2187000
15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: 
PacketHeader with packetLen=65532 header data: offsetInBlock: 1821184
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=136, 
lastPacketInBlock=false, offsetInBlock=1886208, 
ackEnqueueNanoTime=1427226852716703000, ackStatus=SUCCESS)
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 135 status: SUCCESS status: SUCCESS 
downstreamAckTimeNanos: 1227000
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 135 status: SUCCESS status: 
SUCCESS status: SUCCESS downstreamAckTimeNanos: 2311000
15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: 
PacketHeader with packetLen=65532 header data: offsetInBlock: 1886208
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=137, 
lastPacketInBlock=false, offsetInBlock=1951232, 
ackEnqueueNanoTime=1427226852718074000, ackStatus=SUCCESS)
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 136 status: SUCCESS status: SUCCESS 
downstreamAckTimeNanos: 1153000
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 136 status: SUCCESS status: 
SUCCESS status: SUCCESS downstreamAckTimeNanos: 2204000
15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: 
PacketHeader with packetLen=65532 header data: offsetInBlock: 1951232
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=138, 
lastPacketInBlock=false, offsetInBlock=2016256, 
ackEnqueueNanoTime=1427226852719449000, ackStatus=SUCCESS)
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 137 status: SUCCESS status: SUCCESS 
downstreamAckTimeNanos: 1223000
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 137 status: SUCCESS status: 
SUCCESS status: SUCCESS downstreamAckTimeNanos: 2195000
15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: 
PacketHeader with packetLen=65532 header data: offsetInBlock: 2016256
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=139, 
lastPacketInBlock=false, offsetInBlock=2081280, 
ackEnqueueNanoTime=1427226852721161000, ackStatus=SUCCESS)
15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: 
PacketHeader with packetLen=15480 header data: offsetInBlock: 2081280
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=140, 
lastPacketInBlock=false, offsetInBlock=2096640, 
ackEnqueueNanoTime=1427226852721337000, ackStatus=SUCCESS)
15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: 
PacketHeader with packetLen=516 header data: offsetInBlock: 2096640
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=141, 
lastPacketInBlock=false, offsetInBlock=2097152, 
ackEnqueueNanoTime=1427226852721426000, ackStatus=SUCCESS)
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 138 status: SUCCESS status: SUCCESS 
downstreamAckTimeNanos: 1202000
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 138 status: SUCCESS status: 
SUCCESS status: SUCCESS downstreamAckTimeNanos: 2266000
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 139 status: SUCCESS status: SUCCESS 
downstreamAckTimeNanos: 1152000
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 139 status: SUCCESS status: 
SUCCESS status: SUCCESS downstreamAckTimeNanos: 2239000
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 140 status: SUCCESS status: SUCCESS 
downstreamAckTimeNanos: 1108000
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 140 status: SUCCESS status: 
SUCCESS status: SUCCESS downstreamAckTimeNanos: 2178000
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 141 status: SUCCESS status: SUCCESS 
downstreamAckTimeNanos: 1149000
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 141 status: SUCCESS status: 
SUCCESS status: SUCCESS downstreamAckTimeNanos: 2319000
15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving one packet for block 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046: 
PacketHeader with packetLen=0 header data: offsetInBlock: 2097152
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE: enqueue Packet(seqno=142, 
lastPacketInBlock=true, offsetInBlock=2097152, 
ackEnqueueNanoTime=1427226852725564000, ackStatus=SUCCESS)
15/03/24 12:54:12 DEBUG datanode.DataNode: Receiving an empty packet or the end 
of the block 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE got seqno: 142 status: SUCCESS status: SUCCESS 
downstreamAckTimeNanos: 894000
15/03/24 12:54:12 DEBUG impl.FsDatasetImpl: addBlock: Moved 
/Volumes/SimcloudRuntime/simcloud/runtime/data/hdfs/datanode/current/BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555/current/rbw/blk_1078309397_4569046.meta
 to 
/Volumes/SimcloudRuntime/simcloud/runtime/data/hdfs/datanode/current/BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555/current/finalized/subdir28/subdir45/blk_1078309397_4569046.meta
 and 
/Volumes/SimcloudRuntime/simcloud/runtime/data/hdfs/datanode/current/BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555/current/rbw/blk_1078309397
 to 
/Volumes/SimcloudRuntime/simcloud/runtime/data/hdfs/datanode/current/BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555/current/finalized/subdir28/subdir45/blk_1078309397
15/03/24 12:54:12 INFO DataNode.clienttrace: src: /17.219.209.92:49184, dest: 
/IP.PARTLY.MASKED4.192:50010, bytes: 2097152, op: HDFS_WRITE, cliID: 
DFSClient_NONMAPREDUCE_1880564922_1, offset: 0, srvID: 
ece6163a-705c-4373-ad4f-72e8ee993e99, blockid: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
duration: 67061000
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE, replyAck=seqno: 142 status: SUCCESS status: 
SUCCESS status: SUCCESS downstreamAckTimeNanos: 1903000
15/03/24 12:54:12 INFO datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE terminating
15/03/24 12:54:12 DEBUG datanode.DataNode: PacketResponder: 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
type=HAS_DOWNSTREAM_IN_PIPELINE: closing
15/03/24 12:54:17 DEBUG datanode.DataNode: 
block=BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
replica=FinalizedReplica, blk_1078309397_4569046, FINALIZED
15/03/24 12:54:17 DEBUG datanode.DataNode: replica=FinalizedReplica, 
blk_1078309397_4569046, FINALIZED
15/03/24 12:54:19 INFO datanode.BlockPoolSliceScanner: Verification succeeded 
for BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/24 17:01:25 INFO datanode.DataNode: 
DatanodeRegistration(IP.PARTLY.MASKED4.192, 
datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, 
ipcPort=50020, 
storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0)
 Starting thread to transfer 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to 
IP.PARTLY.MASKED5.251:50010 IP.PARTLY.MASKED6.216:50010 
IP.PARTLY.MASKED6.215:50010 
15/03/24 17:01:25 DEBUG datanode.DataNode: 
block=BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046, 
replica=FinalizedReplica, blk_1078309397_4569046, FINALIZED
15/03/24 17:01:25 DEBUG datanode.DataNode: replica=FinalizedReplica, 
blk_1078309397_4569046, FINALIZED
15/03/24 17:01:25 INFO datanode.DataNode: DataTransfer: Transmitted 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 
(numBytes=2097152) to /IP.PARTLY.MASKED5.251:50010
15/03/25 09:43:20 INFO impl.FsDatasetAsyncDiskService: Scheduling 
blk_1078309397_4569046 file 
/Volumes/SimcloudRuntime/simcloud/runtime/data/hdfs/datanode/current/BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555/current/finalized/subdir28/subdir45/blk_1078309397
 for deletion
15/03/25 09:43:20 INFO impl.FsDatasetAsyncDiskService: Deleted 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555 blk_1078309397_4569046 file 
/Volumes/SimcloudRuntime/simcloud/runtime/data/hdfs/datanode/current/BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555/current/finalized/subdir28/subdir45/blk_1078309397
15/03/25 09:43:57 INFO datanode.DataNode: opReadBlock 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 
received exception 
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:43:57 WARN datanode.DataNode: 
DatanodeRegistration(IP.PARTLY.MASKED4.192, 
datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, 
ipcPort=50020, 
storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got
 exception while serving 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to 
/IP.PARTLY.MASKED6.227:57872
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:44:36 INFO datanode.DataNode: opReadBlock 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 
received exception 
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:44:36 WARN datanode.DataNode: 
DatanodeRegistration(IP.PARTLY.MASKED4.192, 
datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, 
ipcPort=50020, 
storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got
 exception while serving 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to 
/IP.PARTLY.MASKED6.227:57905
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:45:15 INFO datanode.DataNode: opReadBlock 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 
received exception 
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:45:15 WARN datanode.DataNode: 
DatanodeRegistration(IP.PARTLY.MASKED4.192, 
datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, 
ipcPort=50020, 
storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got
 exception while serving 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to 
/IP.PARTLY.MASKED6.227:57924
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:45:54 INFO datanode.DataNode: opReadBlock 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 
received exception 
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:45:54 WARN datanode.DataNode: 
DatanodeRegistration(IP.PARTLY.MASKED4.192, 
datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, 
ipcPort=50020, 
storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got
 exception while serving 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to 
/IP.PARTLY.MASKED6.227:57966
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:45:57 INFO datanode.DataNode: opReadBlock 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 
received exception 
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:45:57 WARN datanode.DataNode: 
DatanodeRegistration(IP.PARTLY.MASKED4.192, 
datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, 
ipcPort=50020, 
storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got
 exception while serving 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to 
/IP.PARTLY.MASKED6.40:58165
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:45:58 INFO datanode.DataNode: opReadBlock 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 
received exception 
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:45:58 WARN datanode.DataNode: 
DatanodeRegistration(IP.PARTLY.MASKED4.192, 
datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, 
ipcPort=50020, 
storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got
 exception while serving 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to 
/IP.PARTLY.MASKED5.87:59400
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:45:58 INFO datanode.DataNode: opReadBlock 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 
received exception 
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:45:58 WARN datanode.DataNode: 
DatanodeRegistration(IP.PARTLY.MASKED4.192, 
datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, 
ipcPort=50020, 
storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got
 exception while serving 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to 
/IP.PARTLY.MASKED5.62:42339
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:45:58 INFO datanode.DataNode: opReadBlock 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 
received exception 
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:45:58 WARN datanode.DataNode: 
DatanodeRegistration(IP.PARTLY.MASKED4.192, 
datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, 
ipcPort=50020, 
storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got
 exception while serving 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to 
/IP.PARTLY.MASKED6.100:54148
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:45:59 INFO datanode.DataNode: opReadBlock 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 
received exception 
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:45:59 WARN datanode.DataNode: 
DatanodeRegistration(IP.PARTLY.MASKED4.192, 
datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, 
ipcPort=50020, 
storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got
 exception while serving 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to 
/IP.PARTLY.MASKED6.104:42736
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:00 INFO datanode.DataNode: opReadBlock 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 
received exception 
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:00 WARN datanode.DataNode: 
DatanodeRegistration(IP.PARTLY.MASKED4.192, 
datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, 
ipcPort=50020, 
storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got
 exception while serving 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to 
/IP.PARTLY.MASKED5.51:41096
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:01 INFO datanode.DataNode: opReadBlock 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 
received exception 
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:01 WARN datanode.DataNode: 
DatanodeRegistration(IP.PARTLY.MASKED4.192, 
datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, 
ipcPort=50020, 
storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got
 exception while serving 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to 
/IP.PARTLY.MASKED5.247:36436
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:01 INFO datanode.DataNode: opReadBlock 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 
received exception 
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:01 WARN datanode.DataNode: 
DatanodeRegistration(IP.PARTLY.MASKED4.192, 
datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, 
ipcPort=50020, 
storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got
 exception while serving 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to 
/IP.PARTLY.MASKED6.54:56894
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:02 INFO datanode.DataNode: opReadBlock 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 
received exception 
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:02 WARN datanode.DataNode: 
DatanodeRegistration(IP.PARTLY.MASKED4.192, 
datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, 
ipcPort=50020, 
storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got
 exception while serving 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to 
/IP.PARTLY.MASKED6.161:60278
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:31 INFO datanode.DataNode: opReadBlock 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 
received exception 
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:31 WARN datanode.DataNode: 
DatanodeRegistration(IP.PARTLY.MASKED4.192, 
datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, 
ipcPort=50020, 
storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got
 exception while serving 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to 
/IP.PARTLY.MASKED6.254:50599
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:31 INFO datanode.DataNode: opReadBlock 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 
received exception 
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:31 WARN datanode.DataNode: 
DatanodeRegistration(IP.PARTLY.MASKED4.192, 
datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, 
ipcPort=50020, 
storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got
 exception while serving 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to 
/IP.PARTLY.MASKED5.94:35042
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:31 INFO datanode.DataNode: opReadBlock 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 
received exception 
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:31 WARN datanode.DataNode: 
DatanodeRegistration(IP.PARTLY.MASKED4.192, 
datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, 
ipcPort=50020, 
storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got
 exception while serving 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to 
/IP.PARTLY.MASKED6.135:65058
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:32 INFO datanode.DataNode: opReadBlock 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 
received exception 
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:32 WARN datanode.DataNode: 
DatanodeRegistration(IP.PARTLY.MASKED4.192, 
datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, 
ipcPort=50020, 
storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got
 exception while serving 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to 
/IP.PARTLY.MASKED6.148:40964
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:33 INFO datanode.DataNode: opReadBlock 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 
received exception 
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:33 WARN datanode.DataNode: 
DatanodeRegistration(IP.PARTLY.MASKED4.192, 
datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, 
ipcPort=50020, 
storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got
 exception while serving 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to 
/IP.PARTLY.MASKED6.55:33231
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:33 INFO datanode.DataNode: opReadBlock 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 
received exception 
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:33 WARN datanode.DataNode: 
DatanodeRegistration(IP.PARTLY.MASKED4.192, 
datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, 
ipcPort=50020, 
storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got
 exception while serving 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to 
/IP.PARTLY.MASKED6.227:58010
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:33 INFO datanode.DataNode: opReadBlock 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 
received exception 
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:33 WARN datanode.DataNode: 
DatanodeRegistration(IP.PARTLY.MASKED4.192, 
datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, 
ipcPort=50020, 
storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got
 exception while serving 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to 
/IP.PARTLY.MASKED6.197:46999
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:34 INFO datanode.DataNode: opReadBlock 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 
received exception 
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:34 WARN datanode.DataNode: 
DatanodeRegistration(IP.PARTLY.MASKED4.192, 
datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, 
ipcPort=50020, 
storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got
 exception while serving 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to 
/IP.PARTLY.MASKED6.164:61581
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:34 INFO datanode.DataNode: opReadBlock 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 
received exception 
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:34 WARN datanode.DataNode: 
DatanodeRegistration(IP.PARTLY.MASKED4.192, 
datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, 
ipcPort=50020, 
storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got
 exception while serving 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to 
/IP.PARTLY.MASKED5.93:63000
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:34 INFO datanode.DataNode: opReadBlock 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 
received exception 
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:34 WARN datanode.DataNode: 
DatanodeRegistration(IP.PARTLY.MASKED4.192, 
datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, 
ipcPort=50020, 
storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got
 exception while serving 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to 
/IP.PARTLY.MASKED5.37:55288
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:35 INFO datanode.DataNode: opReadBlock 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 
received exception 
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:35 WARN datanode.DataNode: 
DatanodeRegistration(IP.PARTLY.MASKED4.192, 
datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, 
ipcPort=50020, 
storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got
 exception while serving 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to 
/IP.PARTLY.MASKED5.229:50147
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:35 INFO datanode.DataNode: opReadBlock 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 
received exception 
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:35 WARN datanode.DataNode: 
DatanodeRegistration(IP.PARTLY.MASKED4.192, 
datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, 
ipcPort=50020, 
storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got
 exception while serving 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to 
/IP.PARTLY.MASKED5.24:37956
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:35 INFO datanode.DataNode: opReadBlock 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 
received exception 
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:35 WARN datanode.DataNode: 
DatanodeRegistration(IP.PARTLY.MASKED4.192, 
datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, 
ipcPort=50020, 
storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got
 exception while serving 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to 
/IP.PARTLY.MASKED5.141:57225
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:35 INFO datanode.DataNode: opReadBlock 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 
received exception 
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:35 WARN datanode.DataNode: 
DatanodeRegistration(IP.PARTLY.MASKED4.192, 
datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, 
ipcPort=50020, 
storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got
 exception while serving 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to 
/IP.PARTLY.MASKED6.87:50226
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:35 INFO datanode.DataNode: opReadBlock 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 
received exception 
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:35 WARN datanode.DataNode: 
DatanodeRegistration(IP.PARTLY.MASKED4.192, 
datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, 
ipcPort=50020, 
storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got
 exception while serving 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to 
/IP.PARTLY.MASKED5.65:64743
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:35 INFO datanode.DataNode: opReadBlock 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 
received exception 
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:35 WARN datanode.DataNode: 
DatanodeRegistration(IP.PARTLY.MASKED4.192, 
datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, 
ipcPort=50020, 
storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got
 exception while serving 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to 
/IP.PARTLY.MASKED5.234:51010
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:35 INFO datanode.DataNode: opReadBlock 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 
received exception 
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:35 WARN datanode.DataNode: 
DatanodeRegistration(IP.PARTLY.MASKED4.192, 
datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, 
ipcPort=50020, 
storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got
 exception while serving 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to 
/IP.PARTLY.MASKED5.100:38300
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:36 INFO datanode.DataNode: opReadBlock 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 
received exception 
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:36 WARN datanode.DataNode: 
DatanodeRegistration(IP.PARTLY.MASKED4.192, 
datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, 
ipcPort=50020, 
storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got
 exception while serving 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to 
/IP.PARTLY.MASKED5.60:36632
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:36 INFO datanode.DataNode: opReadBlock 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 
received exception 
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:36 WARN datanode.DataNode: 
DatanodeRegistration(IP.PARTLY.MASKED4.192, 
datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, 
ipcPort=50020, 
storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got
 exception while serving 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to 
/IP.PARTLY.MASKED5.224:52901
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:36 INFO datanode.DataNode: opReadBlock 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 
received exception 
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:36 WARN datanode.DataNode: 
DatanodeRegistration(IP.PARTLY.MASKED4.192, 
datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, 
ipcPort=50020, 
storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got
 exception while serving 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to 
/IP.PARTLY.MASKED6.40:58191
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:37 INFO datanode.DataNode: opReadBlock 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 
received exception 
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:37 WARN datanode.DataNode: 
DatanodeRegistration(IP.PARTLY.MASKED4.192, 
datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, 
ipcPort=50020, 
storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got
 exception while serving 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to 
/IP.PARTLY.MASKED5.62:42366
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:37 INFO datanode.DataNode: opReadBlock 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 
received exception 
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:37 WARN datanode.DataNode: 
DatanodeRegistration(IP.PARTLY.MASKED4.192, 
datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, 
ipcPort=50020, 
storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got
 exception while serving 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to 
/IP.PARTLY.MASKED5.87:59427
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:37 INFO datanode.DataNode: opReadBlock 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 
received exception 
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:37 WARN datanode.DataNode: 
DatanodeRegistration(IP.PARTLY.MASKED4.192, 
datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, 
ipcPort=50020, 
storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got
 exception while serving 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to 
/IP.PARTLY.MASKED6.100:54175
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:38 INFO datanode.DataNode: opReadBlock 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 
received exception 
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:38 WARN datanode.DataNode: 
DatanodeRegistration(IP.PARTLY.MASKED4.192, 
datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, 
ipcPort=50020, 
storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got
 exception while serving 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to 
/IP.PARTLY.MASKED6.104:42766
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:39 INFO datanode.DataNode: opReadBlock 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 
received exception 
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:39 WARN datanode.DataNode: 
DatanodeRegistration(IP.PARTLY.MASKED4.192, 
datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, 
ipcPort=50020, 
storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got
 exception while serving 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to 
/IP.PARTLY.MASKED5.51:41153
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:40 INFO datanode.DataNode: opReadBlock 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 
received exception 
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:40 WARN datanode.DataNode: 
DatanodeRegistration(IP.PARTLY.MASKED4.192, 
datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, 
ipcPort=50020, 
storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got
 exception while serving 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to 
/IP.PARTLY.MASKED6.54:56921
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:41 INFO datanode.DataNode: opReadBlock 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 
received exception 
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:41 WARN datanode.DataNode: 
DatanodeRegistration(IP.PARTLY.MASKED4.192, 
datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, 
ipcPort=50020, 
storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got
 exception while serving 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to 
/IP.PARTLY.MASKED6.161:60304
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:41 INFO datanode.DataNode: opReadBlock 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 
received exception 
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
15/03/25 09:46:41 WARN datanode.DataNode: 
DatanodeRegistration(IP.PARTLY.MASKED4.192, 
datanodeUuid=ece6163a-705c-4373-ad4f-72e8ee993e99, infoPort=50075, 
ipcPort=50020, 
storageInfo=lv=-55;cid=CID-b99af807-d600-464e-84bc-f756943ea87e;nsid=224929830;c=0):Got
 exception while serving 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046 to 
/IP.PARTLY.MASKED5.247:36503
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for 
BP-2091375182-IP.PARTLY.MASKED7.219-1423771577555:blk_1078309397_4569046




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8208) Namenode reports stale block locations during decommissioning

Reply via email to