Yongjun Zhang created HDFS-7070:
-----------------------------------
Summary: TestWebHdfsFileSystemContract fails occassionally
Key: HDFS-7070
URL: https://issues.apache.org/jira/browse/HDFS-7070
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Affects Versions: 2.6.0
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract.testResponseCode
and
org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract.testRenameDirToSelf
failed recently, maybe introduced by some latest code change due to file
descriptor leak (?)
E.g.
https://builds.apache.org/job/PreCommit-HDFS-Build/8026/testReport/org.apache.hadoop.hdfs.web/TestWebHdfsFileSystemContract/testResponseCode/.
{code}
2014-09-15 12:52:18,866 INFO datanode.DataNode
(DataXceiver.java:writeBlock(749)) - opWriteBlock
BP-23833599-67.195.81.147-1410785517350:blk_1073741827_1461 received exception
java.io.IOException: Cannot run program "stat": java.io.IOException: error=24,
Too many open files
2014-09-15 12:52:18,867 ERROR datanode.DataNode (DataXceiver.java:run(243)) -
127.0.0.1:47221:DataXceiver error processing WRITE_BLOCK operation src:
/127.0.0.1:38112 dst: /127.0.0.1:47221
java.io.IOException: Cannot run program "stat": java.io.IOException: error=24,
Too many open files
at java.lang.ProcessBuilder.start(ProcessBuilder.java:470)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:485)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
at org.apache.hadoop.fs.HardLink.getLinkCount(HardLink.java:495)
at
org.apache.hadoop.hdfs.server.datanode.ReplicaInfo.unlinkBlock(ReplicaInfo.java:288)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.append(FsDatasetImpl.java:702)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.append(FsDatasetImpl.java:680)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.append(FsDatasetImpl.java:101)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:193)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:604)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:126)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:72)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:225)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: java.io.IOException: error=24, Too many open
files
at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
... 14 more
2014-09-15 12:52:18,867 INFO hdfs.DFSClient
(DFSOutputStream.java:createBlockOutputStream(1400)) - Exception in
createBlockOutputStream
java.io.EOFException: Premature EOF: no length prefix available
at
org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2101)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1368)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1210)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:530)
2014-09-15 12:52:18,870 WARN hdfs.DFSClient (DFSOutputStream.java:run(883)) -
DFSOutputStream ResponseProcessor exception for block
BP-23833599-67.195.81.147-1410785517350:blk_1073741827_1461
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2099)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:798)
2014-09-15 12:52:18,870 WARN hdfs.DFSClient (DFSOutputStream.java:run(627)) -
DataStreamer Exception
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.DFSOutputStream$Packet.writeTo(DFSOutputStream.java:273)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:579)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
