Hi Feng, Thanks for your reply. I'm going to add these stuff into the OPTS: -XX:MaxPermSize, -XX:+CMSPermGenSweepingEnabled and -XX:+CMSClassUnloadingEnabled. I'm still not clear about something:
- Can I add -XX:MaxPermSize into JAVA_OPTS instead? - Does Garbage Collector clean the stuff in PermGen Space? If it doesn't clean the stuff in PermGen and Nutch keeps adding the stuff into the PermGen Space, it will have the PermGen issue again I guess (with the bigger size) ? Many thanks, Vu On Wed, Mar 20, 2013 at 1:20 PM, feng lu <[email protected]> wrote: > Hi Deals > > you can edit the bin/nutch script file and increase the Permanent > Generation Space to NUTCH_OPTS param. some code like this > > JAVA=$JAVA_HOME/bin/java > JAVA_HEAP_MAX=-Xmx1000m > NUTCH_OPTS="-XX:PermSize=128M -XX:MaxPermSize=256m" > > > On Wed, Mar 20, 2013 at 9:31 AM, Deals Collect <[email protected] > >wrote: > > > Hi all, > > > > I'm using Nutch 1.4 and Solr 3.6.1. The crawling is working well, it > crawls > > data, send to Solr perfectly. But the problem happens when the crawl is > > failed sometimes, I get the java.lang.OutOfMemoryError: PermGen space > right > > after that. Here is the log file: > > > > java.io.IOException: Job failed! > > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252) > > at org.apache.nutch.parse.ParseSegment.parse(ParseSegment.java:157) > > at org.apache.nutch.crawl.Crawl.run(Crawl.java:138) > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > > at jobs.CrawlerUtils.crawlJob(CrawlerUtils.java:15) > > at > > > > > jobs.cudo.GoldCoastCudoNutchCrawler.doJob(GoldCoastCudoNutchCrawler.java:23) > > at play.jobs.Job.doJobWithResult(Job.java:50) > > at play.jobs.Job.call(Job.java:146) > > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > > at > > > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98) > > at > > > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206) > > at > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > at > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:662) > > > > java.io.IOException: Job failed! > > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252) > > at org.apache.nutch.crawl.Injector.inject(Injector.java:217) > > at org.apache.nutch.crawl.Crawl.run(Crawl.java:127) > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > > at jobs.CrawlerUtils.crawlJob(CrawlerUtils.java:15) > > at > > jobs.cudo.HobartCudoNutchCrawler.doJob(HobartCudoNutchCrawler.java:23) > > at play.jobs.Job.doJobWithResult(Job.java:50) > > at play.jobs.Job.call(Job.java:146) > > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > > at > > > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98) > > at > > > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206) > > at > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > at > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:662) > > > > Every time I get the "Job Failed!", I will get the problem with memory > > right after that: > > > > Exception in thread "TP-Processor8" Exception in thread "TP-Processor1" > > Exception in thread "TP-Processor7" java.lang.OutOfMemoryError: PermGen > > space > > Exception in thread "TP-Processor6" java.lang.OutOfMemoryError: PermGen > > space > > Exception in thread "TP-Processor9" java.lang.OutOfMemoryError: PermGen > > space > > Exception in thread "TP-Processor3" java.lang.OutOfMemoryError: PermGen > > space > > Exception in thread "TP-Processor12" java.lang.OutOfMemoryError: PermGen > > space > > Exception in thread "TP-Processor5" java.lang.OutOfMemoryError: PermGen > > space > > Exception in thread "TP-Processor10" java.lang.OutOfMemoryError: PermGen > > space > > > > Anyone knows this issue? > > > > Many thanks, > > Vu Pham > > > > > > -- > Don't Grow Old, Grow Up... :-) >
