I'm using nutch v.0.8 and have 3 computers. One of my tasktrakers always go down. This occurs during indexing (index crawl/indexes). On server with crashed tasktracker now available 53G of free disk space and used only 11G. How i can decide this problem? Why tasktarcker requires so much free space on HDD?
Piece of Log with error: 060613 151840 task_0083_r_000001_0 0.5% reduce > sort 060613 151841 task_0083_r_000001_0 0.5% reduce > sort 060613 151842 task_0083_r_000001_0 0.5% reduce > sort 060613 151843 task_0083_r_000001_0 0.5% reduce > sort 060613 151844 task_0083_r_000001_0 0.5% reduce > sort 060613 151845 task_0083_r_000001_0 0.5% reduce > sort 060613 151846 task_0083_r_000001_0 0.5% reduce > sort 060613 151847 task_0083_r_000001_0 0.5% reduce > sort 060613 151847 SEVERE FSError, exiting: java.io.IOException: No space left on device 060613 151847 task_0083_r_000001_0 SEVERE FSError from child 060613 151847 task_0083_r_000001_0 org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device 060613 151847 task_0083_r_000001_0 at org.apache.hadoop.fs.LocalFileSystem$LocalFSFileOutputStream.write(LocalFile Syst 060613 151847 task_0083_r_000001_0 at org.apache.hadoop.fs.FSDataOutputStream$Summer.write(FSDataOutputStream.java :69) 060613 151847 task_0083_r_000001_0 at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStre am.j 060613 151847 task_0083_r_000001_0 at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) 060613 151847 task_0083_r_000001_0 at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123) 060613 151847 task_0083_r_000001_0 at java.io.DataOutputStream.flush(DataOutputStream.java:106) 060613 151847 task_0083_r_000001_0 at java.io.FilterOutputStream.close(FilterOutputStream.java:140) 060613 151847 task_0083_r_000001_0 at org.apache.hadoop.io.SequenceFile$Sorter$SortPass.close(SequenceFile.java:59 8) 060613 151847 task_0083_r_000001_0 at org.apache.hadoop.io.SequenceFile$Sorter.sortPass(SequenceFile.java:533) 060613 151847 task_0083_r_000001_0 at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:519) 060613 151847 task_0083_r_000001_0 at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:316) 060613 151847 task_0083_r_000001_0 060613 151847 task_0083_r_000001_0 at org.apache.hadoop.mapred.TaskTracker$Chi 060613 151847 task_0083_r_000001_0 Caused by: java.io.IOException: No space left on device 060613 151847 task_0083_r_000001_0 at java.io.FileOutputStream.writeBytes(Native Method) 060613 151847 task_0083_r_000001_0 at java.io.FileOutputStream.write(FileOutputStream.java:260) 060613 151848 task_0083_r_000001_0 at org.apache.hadoop.fs.LocalFileSystem$LocalFSFileOutputStream.write(LocalFile Syst 060613 151848 task_0083_r_000001_0 ... 11 more 060613 151849 Server connection on port 50050 from 10.0.0.3: exiting 060613 151854 task_0083_m_000001_0 done; removing files. 060613 151855 task_0083_m_000003_0 done; removing files. _______________________________________________ Nutch-developers mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-developers
