The default heap size of 1G is just enough for a parsing fetcher with 10 threads. The only problem that may rise is too large and complicated PDF files or very large HTML files. If you generate fetch lists of a reasonable size there won't be a problem most of the time. And if you want to crawl a lot, then just generate more small segments.
If there is a bug it's most likely to be the parser eating memory and not releasing it. -----Original message----- > From:Tejas Patil <[email protected]> > Sent: Sun 03-Mar-2013 22:19 > To: [email protected] > Subject: Re: Nutch 1.6 : java.lang.OutOfMemoryError: unable to create new > native thread > > I agree with Sebastian. It was a crawl in local mode and not over a > cluster. The intended crawl volume is huge and if we dont override the > default heap size to some decent value, there is high possibility of facing > an OOM. > > > On Sun, Mar 3, 2013 at 1:04 PM, kiran chitturi > <[email protected]>wrote: > > > > If you find the time you should trace the process. > > > Seems to be either a misconfiguration or even a bug. > > > > > > I will try to track this down soon with the previous configuration. Right > > now, i am just trying to get data crawled by Monday. > > > > Kiran. > > > > > > > >> Luckily, you should be able to retry via "bin/nutch parse ..." > > > >> Then trace the system and the Java process to catch the reason. > > > >> > > > >> Sebastian > > > >> > > > >> On 03/02/2013 08:13 PM, kiran chitturi wrote: > > > >>> Sorry, i am looking to crawl 400k documents with the crawl. I said > > 400 > > > in > > > >>> my last message. > > > >>> > > > >>> > > > >>> On Sat, Mar 2, 2013 at 2:12 PM, kiran chitturi < > > > >> [email protected]>wrote: > > > >>> > > > >>>> Hi! > > > >>>> > > > >>>> I am running Nutch 1.6 on a 4 GB Mac OS desktop with Core i5 2.8GHz. > > > >>>> > > > >>>> Last night i started a crawl on local mode for 5 seeds with the > > config > > > >>>> given below. If the crawl goes well, it should fetch a total of 400 > > > >>>> documents. The crawling is done on a single host that we own. > > > >>>> > > > >>>> Config > > > >>>> --------------------- > > > >>>> > > > >>>> fetcher.threads.per.queue - 2 > > > >>>> fetcher.server.delay - 1 > > > >>>> fetcher.throughput.threshold.pages - -1 > > > >>>> > > > >>>> crawl script settings > > > >>>> ---------------------------- > > > >>>> timeLimitFetch- 30 > > > >>>> numThreads - 5 > > > >>>> topN - 10000 > > > >>>> mapred.child.java.opts=-Xmx1000m > > > >>>> > > > >>>> > > > >>>> I have noticed today that the crawl has stopped due to an error and > > i > > > >> have > > > >>>> found the below error in logs. > > > >>>> > > > >>>> 2013-03-01 21:45:03,767 INFO parse.ParseSegment - Parsed (0ms): > > > >>>>> http://scholar.lib.vt.edu/ejournals/JARS/v33n3/v33n3-letcher.htm > > > >>>>> 2013-03-01 21:45:03,790 WARN mapred.LocalJobRunner - > > job_local_0001 > > > >>>>> java.lang.OutOfMemoryError: unable to create new native thread > > > >>>>> at java.lang.Thread.start0(Native Method) > > > >>>>> at java.lang.Thread.start(Thread.java:658) > > > >>>>> at > > > >>>>> > > > >> > > > > > java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:681) > > > >>>>> at > > > >>>>> > > > >> > > > > > java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727) > > > >>>>> at > > > >>>>> > > > >> > > > > > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:655) > > > >>>>> at > > > >>>>> > > > >> > > > > > java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:92) > > > >>>>> at > > > >> org.apache.nutch.parse.ParseUtil.runParser(ParseUtil.java:159) > > > >>>>> at > > > org.apache.nutch.parse.ParseUtil.parse(PaifrseUtil.java:93) > > > >>>>> at > > > >> org.apache.nutch.parse.ParseSegment.map(ParseSegment.java:97) > > > >>>>> at > > > >> org.apache.nutch.parse.ParseSegment.map(ParseSegment.java:44) > > > >>>>> at > > org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) > > > >>>>> at > > > >> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436) > > > >>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) > > > >>>>> at > > > >>>>> > > > >> > > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212) > > > >>>>> (END) > > > >>>> > > > >>>> > > > >>>> > > > >>>> Did anyone run in to the same issue ? I am not sure why the new > > native > > > >>>> thread is not being created. The link here says [0] that it might > > due > > > to > > > >>>> the limitation of number of processes in my OS. Will increase them > > > solve > > > >>>> the issue ? > > > >>>> > > > >>>> > > > >>>> [0] - http://ww2.cs.fsu.edu/~czhang/errors.html > > > >>>> > > > >>>> Thanks! > > > >>>> > > > >>>> -- > > > >>>> Kiran Chitturi > > > >>>> > > > >>> > > > >>> > > > >>> > > > >> > > > >> > > > > > > > > > > > > > > > > > > > > -- > > Kiran Chitturi > > >
