[ http://issues.apache.org/jira/browse/NUTCH-159?page=comments#action_12361545 ]
byron miller commented on NUTCH-159: ------------------------------------ While it's from the mapred trunk, it is a non ndfs/local instance only. Mapred.temp.dir was left at it's defaults.. (which didn't exist) <property> <name>mapred.temp.dir</name> <value>/tmp/nutch/mapred/temp</value> <description>A shared directory for temporary files. </description> </property> I'm going to modify this and re-run my fetch and let you know how that works. > Specify temp/working directory for crawl > ---------------------------------------- > > Key: NUTCH-159 > URL: http://issues.apache.org/jira/browse/NUTCH-159 > Project: Nutch > Type: Bug > Components: fetcher, indexer > Versions: 0.8-dev > Environment: Linux/Debian > Reporter: byron miller > > I ran a crawl of 100k web pages and got: > org.apache.nutch.fs.FSError: java.io.IOException: No space left on device > at > org.apache.nutch.fs.LocalFileSystem$LocalNFSFileOutputStream.write(LocalFileSystem.java:149) > at org.apache.nutch.fs.FileUtil.copyContents(FileUtil.java:65) > at > org.apache.nutch.fs.LocalFileSystem.renameRaw(LocalFileSystem.java:178) > at > org.apache.nutch.fs.NutchFileSystem.rename(NutchFileSystem.java:224) > at > org.apache.nutch.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:80) > Caused by: java.io.IOException: No space left on device > at java.io.FileOutputStream.writeBytes(Native Method) > at java.io.FileOutputStream.write(FileOutputStream.java:260) > at > org.apache.nutch.fs.LocalFileSystem$LocalNFSFileOutputStream.write(LocalFileSystem.java:147) > ... 4 more > Exception in thread "main" java.io.IOException: Job failed! > at org.apache.nutch.mapred.JobClient.runJob(JobClient.java:308) > at org.apache.nutch.crawl.Fetcher.fetch(Fetcher.java:335) > at org.apache.nutch.crawl.Crawl.main(Crawl.java:107) > [EMAIL PROTECTED]:/data/nutch$ df -k > It appears crawl created a /tmp/nutch directory that filled up even though i > specified a db directory. > Need to add a parameter to the command line or make a globaly configurable > /tmp (work area) for the nutch instance so that crawls won't fail. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira ------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click _______________________________________________ Nutch-developers mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-developers
