thanks, Bill and Otis. But both parameters are for JVM. Is there a way to control memory consumption inside nutch? The memory in my machine is less than 1G.
> Subject: RE: job exception> Date: Mon, 19 May 2008 11:57:25 -0400> From: > [EMAIL PROTECTED]> To: [email protected]> > Yes, but he is running > out of PermGen space, so you would use something> like: -XX:PermSize=128m > -XX:MaxPermSize=512m> > Bill.> > -----Original Message-----> From: [EMAIL > PROTECTED] [mailto:[EMAIL PROTECTED] > Sent: Monday, May 19, 2008 11:46 AM> > To: [email protected]> Subject: Re: job exception> > Yes, your > Nutch instance ran out of memory. Give it more if you can,> e.g. -Xmx1G or > -Xmx2G> > Otis> --> Sematext -- http://sematext.com/ -- Lucene - Solr - > Nutch> > > ----- Original Message ----> > From: Marcel T <[EMAIL PROTECTED]>> > > To: [email protected]> > Sent: Sunday, May 18, 2008 8:01:52 PM> > > Subject: job exception> > > > > > when I tested crawling a web deeply > (-depth 100 -topN 1000), the job> always > > failed with the exception below> > > > > Exception in thread "main" java.io.IOException: Job failed!> > at> > org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:894)> > at> > org.apache.nutch.crawl.Generator.generate(Generator.java:456)> > at> > org.apache.nutch.crawl.Generator.generate(Generator.java:393)> > at > org.apache.nutch.crawl.Crawl.main(Crawl.java:116)> > > > log shows the > following:> > java.lang.OutOfMemoryError: PermGen space> > at > java.lang.ClassLoader.defineClass1(Native Method)> > at > java.lang.ClassLoader.defineClass(ClassLoader.java:620)> > at > >> > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124)> > > at> java.net.URLClassLoader.defineClass(URLClassLoader.java:260)> > at > java.net.URLClassLoader.access$000(URLClassLoader.java:56)> > at > java.net.URLClassLoader$1.run(URLClassLoader.java:195)> > at > java.security.AccessController.doPrivileged(Native Method)> > at > java.net.URLClassLoader.findClass(URLClassLoader.java:188)> > at > java.lang.ClassLoader.loadClass(ClassLoader.java:306)> > at > java.lang.ClassLoader.loadClass(ClassLoader.java:251)> > at > >> > org.apache.nutch.plugin.Extension.getExtensionInstance(Extension.java:15> 6)> > > at > >> > org.apache.nutch.net.URLNormalizers.getURLNormalizers(URLNormalizers.jav> > a:170)> > at> org.apache.nutch.net.URLNormalizers.(URLNormalizers.java:128)> > > at > >> > org.apache.nutch.crawl.Generator$Selector.configure(Generator.java:109)> > at > > >> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)> > > at > >> > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:> > 82)> > at> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:250)> > at > > >> > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:164)> > > > > > > > > Is this is memory problem? I set the hadoop parameter> > > > > mapred.child.java.opts> > -Xmx512m> > > > > > Any idea why this keeps > happening? Many thanks!>
