[
https://issues.apache.org/jira/browse/HDFS-4723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein resolved HDFS-4723.
---------------------------------
Release Note: I cannot reproduce the same stack trace. HDFS-15461
investigates addresses the same unit test with a different stack trace.
Resolution: Cannot Reproduce
> Occasional failure in TestDFSClientRetries#testGetFileChecksum because the
> number of available xcievers is set too low
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-4723
> URL: https://issues.apache.org/jira/browse/HDFS-4723
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: test
> Affects Versions: 2.0.4-alpha, 3.0.0-alpha1
> Reporter: Andrew Kyle Purtell
> Priority: Minor
> Attachments: 4723-branch-2.patch, 4723.patch
>
>
> Occasional failure in TestDFSClientRetries#testGetFileChecksum because the
> number of available xcievers is set too low.
> {noformat}
> 2013-04-21 18:48:28,273 WARN datanode.DataNode
> (DataXceiverServer.java:run(161)) - 127.0.0.1:37608:DataXceiverServer:
> java.io.IOException: Xceiver count 3 exceeds the limit of concurrent
> xcievers: 2
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:143)
> at java.lang.Thread.run(Thread.java:662)
> 2013-04-21 18:48:28,274 INFO datanode.DataNode
> (DataXceiver.java:writeBlock(453)) - Datanode 2 got response for connect ack
> from downstream datanode with firstbadlink as 127.0.0.1:37608
> 2013-04-21 18:48:28,276 INFO datanode.DataNode
> (DataXceiver.java:writeBlock(491)) - Datanode 2 forwarding connect ack to
> upstream firstbadlink is 127.0.0.1:37608
> 2013-04-21 18:48:28,276 ERROR datanode.DataNode
> (DataXceiver.java:writeBlock(477)) -
> DataNode{data=FSDataset{dirpath='[/home/ec2-user/jenkins/workspace/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data3/current,
>
> /home/ec2-user/jenkins/workspace/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data4/current]'},
> localName='127.0.0.1:33298',
> storageID='DS-1506063529-10.174.86.97-33298-1366570107286',
> xmitsInProgress=0}:Exception transfering block
> BP-2121022065-10.174.86.97-1366570107029:blk_6876843860808656778_1071 to
> mirror 127.0.0.1:37608: java.io.EOFException: Premature EOF: no length prefix
> available
> 2013-04-21 18:48:28,276 INFO hdfs.DFSClient
> (DFSOutputStream.java:createBlockOutputStream(1105)) - Exception in
> createBlockOutputStream
> java.io.IOException: Bad connect ack with firstBadLink as 127.0.0.1:37608
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1096)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1019)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:464)
> 2013-04-21 18:48:28,276 INFO datanode.DataNode
> (DataXceiver.java:writeBlock(537)) - opWriteBlock
> BP-2121022065-10.174.86.97-1366570107029:blk_6876843860808656778_1071
> received exception java.io.EOFException: Premature EOF: no length prefix
> available
> 2013-04-21 18:48:28,277 INFO datanode.DataNode
> (BlockReceiver.java:receiveBlock(674)) - Exception for
> BP-2121022065-10.174.86.97-1366570107029:blk_6876843860808656778_1071
> java.io.IOException: Premature EOF from inputStream
> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:194)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:414)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:644)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:506)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:98)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:65)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:219)
> at java.lang.Thread.run(Thread.java:662)
> 2013-04-21 18:48:28,277 INFO hdfs.DFSClient
> (DFSOutputStream.java:nextBlockOutputStream(1022)) - Abandoning
> BP-2121022065-10.174.86.97-1366570107029:blk_6876843860808656778_1071
> 2013-04-21 18:48:28,277 ERROR datanode.DataNode (DataXceiver.java:run(223)) -
> 127.0.0.1:33298:DataXceiver error processing WRITE_BLOCK operation src:
> /127.0.0.1:55182 dest: /127.0.0.1:33298
> java.io.EOFException: Premature EOF: no length prefix available
> at
> org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1340)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:448)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:98)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:65)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:219)
> at java.lang.Thread.run(Thread.java:662)
> 2013-04-21 18:48:28,277 INFO datanode.DataNode (BlockReceiver.java:run(950))
> - PacketResponder:
> BP-2121022065-10.174.86.97-1366570107029:blk_6876843860808656778_1071,
> type=HAS_DOWNSTREAM_IN_PIPELINE
> java.io.EOFException: Premature EOF: no length prefix available
> at
> org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1340)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:116)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:894)
> at java.lang.Thread.run(Thread.java:662)
> 2013-04-21 18:48:28,278 INFO datanode.DataNode (BlockReceiver.java:run(962))
> - PacketResponder:
> BP-2121022065-10.174.86.97-1366570107029:blk_6876843860808656778_1071,
> type=HAS_DOWNSTREAM_IN_PIPELINE: Thread is interrupted.
> 2013-04-21 18:48:28,278 INFO datanode.DataNode
> (BlockReceiver.java:run(1043)) - PacketResponder:
> BP-2121022065-10.174.86.97-1366570107029:blk_6876843860808656778_1071,
> type=HAS_DOWNSTREAM_IN_PIPELINE terminating
> 2013-04-21 18:48:28,278 INFO datanode.DataNode
> (DataXceiver.java:writeBlock(537)) - opWriteBlock
> BP-2121022065-10.174.86.97-1366570107029:blk_6876843860808656778_1071
> received exception java.io.IOException: Premature EOF from inputStream
> 2013-04-21 18:48:28,278 ERROR datanode.DataNode (DataXceiver.java:run(223)) -
> 127.0.0.1:58102:DataXceiver error processing WRITE_BLOCK operation src:
> /127.0.0.1:47124 dest: /127.0.0.1:58102
> java.io.IOException: Premature EOF from inputStream
> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:194)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:414)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:644)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:506)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:98)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:65)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:219)
> at java.lang.Thread.run(Thread.java:662)
> 2013-04-21 18:48:28,279 INFO hdfs.DFSClient
> (DFSOutputStream.java:nextBlockOutputStream(1025)) - Excluding datanode
> 127.0.0.1:37608
> {noformat}
> As a consequence of this failure one datanode has been excluded and from this
> point there are insufficient datanodes to place replicas:
> {noformat}
> 2013-04-21 18:48:54,288 WARN blockmanagement.BlockPlacementPolicy
> (BlockPlacementPolicyDefault.java:chooseTarget(232)) - Not able to place
> enough replicas, still in need of 1 to reach 3
> ...
> {noformat}
> and the test eventually times out.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
