Hi all. I have a problem when i do a crawl for few hour or days, im using nutch 1.5.1 and solr 3.6, but the crawl process fails and i dont know how to fix this problem, im intersted in make a crawl process without limit with 10 cicles or more but i have problem with space on hard disk, i have detected that /etc/tmp have 29 GB used and is not good for me, any body can help me or give some advices for configure nutch to make at least one crawl process without problems ?
here some features of my environment Ram 2 GB CPU:QuadCore(but im using only 2 cores) Hard Disk:40 GB Threads:50 db.fetch.interval.default=2 days this is a part of my log file when nutch fails: **************************************************************** 2013-02-06 18:45:25,961 INFO fetcher.Fetcher - fetching http://bibliodoc.uci.cu/TD/TD_03349_10.pdf 2013-02-06 18:45:25,964 INFO fetcher.Fetcher - fetching http://bibliodoc.uci.cu/TD/TD_0442_07.pdf 2013-02-06 18:45:25,977 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=49 2013-02-06 18:45:26,109 INFO fetcher.Fetcher - -activeThreads=49, spinWaiting=39, fetchQueues.totalSize=0 2013-02-06 18:45:26,180 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=48 2013-02-06 18:45:26,331 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=47 2013-02-06 18:45:26,331 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=46 2013-02-06 18:45:26,331 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=44 2013-02-06 18:45:26,331 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=45 2013-02-06 18:45:26,331 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=40 2013-02-06 18:45:26,331 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=39 2013-02-06 18:45:26,332 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=38 2013-02-06 18:45:26,332 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=37 2013-02-06 18:45:26,332 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=36 2013-02-06 18:45:26,332 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=35 2013-02-06 18:45:26,332 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=34 2013-02-06 18:45:26,333 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=33 2013-02-06 18:45:26,333 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=32 2013-02-06 18:45:26,333 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=31 2013-02-06 18:45:26,333 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=30 2013-02-06 18:45:26,333 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=29 2013-02-06 18:45:26,333 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=28 2013-02-06 18:45:26,334 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=27 2013-02-06 18:45:26,334 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=26 2013-02-06 18:45:26,334 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=25 2013-02-06 18:45:26,334 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=24 2013-02-06 18:45:26,334 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=23 2013-02-06 18:45:26,335 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=22 2013-02-06 18:45:26,335 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=21 2013-02-06 18:45:26,335 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=20 2013-02-06 18:45:26,335 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=19 2013-02-06 18:45:26,335 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=18 2013-02-06 18:45:26,331 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=41 2013-02-06 18:45:26,336 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=17 2013-02-06 18:45:26,336 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=15 2013-02-06 18:45:26,336 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=13 2013-02-06 18:45:26,336 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=12 2013-02-06 18:45:26,331 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=42 2013-02-06 18:45:26,331 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=43 2013-02-06 18:45:26,336 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=9 2013-02-06 18:45:26,336 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=10 2013-02-06 18:45:26,336 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=11 2013-02-06 18:45:26,336 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=14 2013-02-06 18:45:26,336 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=16 2013-02-06 18:45:26,404 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=8 2013-02-06 18:45:26,630 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=7 2013-02-06 18:45:27,069 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=6 2013-02-06 18:45:27,110 INFO fetcher.Fetcher - -activeThreads=6, spinWaiting=0, fetchQueues.totalSize=0 2013-02-06 18:45:27,129 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=5 2013-02-06 18:45:28,110 INFO fetcher.Fetcher - -activeThreads=5, spinWaiting=0, fetchQueues.totalSize=0 2013-02-06 18:45:28,502 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=4 2013-02-06 18:45:29,111 INFO fetcher.Fetcher - -activeThreads=4, spinWaiting=0, fetchQueues.totalSize=0 2013-02-06 18:45:30,123 INFO fetcher.Fetcher - -activeThreads=4, spinWaiting=0, fetchQueues.totalSize=0 2013-02-06 18:45:31,127 INFO fetcher.Fetcher - -activeThreads=4, spinWaiting=0, fetchQueues.totalSize=0 2013-02-06 18:45:31,187 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=3 2013-02-06 18:45:32,171 INFO fetcher.Fetcher - -activeThreads=3, spinWaiting=0, fetchQueues.totalSize=0 2013-02-06 18:45:32,206 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=2 2013-02-06 18:45:33,173 INFO fetcher.Fetcher - -activeThreads=2, spinWaiting=0, fetchQueues.totalSize=0 2013-02-06 18:45:34,173 INFO fetcher.Fetcher - -activeThreads=2, spinWaiting=0, fetchQueues.totalSize=0 2013-02-06 18:45:34,205 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=1 2013-02-06 18:45:34,457 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=0 2013-02-06 18:45:35,174 INFO fetcher.Fetcher - -activeThreads=0, spinWaiting=0, fetchQueues.totalSize=0 2013-02-06 18:45:35,174 INFO fetcher.Fetcher - -activeThreads=0 2013-02-06 18:45:35,742 WARN mapred.LocalJobRunner - job_local_0015 org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for output/file.out at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:381) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:146) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:127) at org.apache.hadoop.mapred.MapOutputFile.getOutputFileForWrite(MapOutputFile.java:69) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1640) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1323) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:437) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)
