failure closing block of file
-----------------------------

                 Key: HADOOP-643
                 URL: http://issues.apache.org/jira/browse/HADOOP-643
             Project: Hadoop
          Issue Type: Bug
          Components: dfs
    Affects Versions: 0.7.2
            Reporter: Johan Oskarson
            Priority: Critical


I've been getting "failure closing block of file" on random files.
Both datanode and tasktracker running on node7. No problems with pinging.
Guess it got stuck after the NPE in DataNode.

Job cannot start because of:

java.io.IOException: failure closing block of file 
/home/hadoop/mapred/system/submit_99u9cd/.job.jar.crc to node node7:50010
        at 
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.internalClose(DFSClient.java:1199)
        at 
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.endBlock(DFSClient.java:1163)
        at 
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClient.java:1241)
        at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
        at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
        at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
        at 
org.apache.hadoop.fs.FSDataOutputStream$Summer.close(FSDataOutputStream.java:96)
        at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
        at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
        at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
        at org.apache.hadoop.fs.FileUtil.copyContent(FileUtil.java:205)
        at org.apache.hadoop.fs.FileUtil.copyContent(FileUtil.java:190)
        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:77)
        at 
org.apache.hadoop.dfs.DistributedFileSystem.copyFromLocalFile(DistributedFileSystem.java:186)
        at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:289)
        at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:314)
        at 
org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:248)
        at 
org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:280)
        at java.lang.Thread.run(Thread.java:595)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
        at java.io.DataInputStream.readFully(DataInputStream.java:176)
        at java.io.DataInputStream.readLong(DataInputStream.java:380)
        at 
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.internalClose(DFSClient.java:1193)


Cxception in datanode.out on node7:
Exception in thread "[EMAIL PROTECTED]" java.lang.NullPointerException
        at 
org.apache.hadoop.dfs.FSDataset$FSDir.checkDirTree(FSDataset.java:162)
        at 
org.apache.hadoop.dfs.FSDataset$FSDir.checkDirTree(FSDataset.java:162)
        at 
org.apache.hadoop.dfs.FSDataset$FSVolume.checkDirs(FSDataset.java:238)
        at 
org.apache.hadoop.dfs.FSDataset$FSVolumeSet.checkDirs(FSDataset.java:326)
        at org.apache.hadoop.dfs.FSDataset.checkDataDir(FSDataset.java:522)
        at 
org.apache.hadoop.dfs.DataNode$DataXceiveServer.run(DataNode.java:480)
        at java.lang.Thread.run(Thread.java:595)


-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: 
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Reply via email to