Yes, but he is running out of PermGen space, so you would use something like: -XX:PermSize=128m -XX:MaxPermSize=512m
Bill. -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Sent: Monday, May 19, 2008 11:46 AM To: [email protected] Subject: Re: job exception Yes, your Nutch instance ran out of memory. Give it more if you can, e.g. -Xmx1G or -Xmx2G Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch ----- Original Message ---- > From: Marcel T <[EMAIL PROTECTED]> > To: [email protected] > Sent: Sunday, May 18, 2008 8:01:52 PM > Subject: job exception > > > when I tested crawling a web deeply (-depth 100 -topN 1000), the job always > failed with the exception below > > Exception in thread "main" java.io.IOException: Job failed! > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:894) > at org.apache.nutch.crawl.Generator.generate(Generator.java:456) > at org.apache.nutch.crawl.Generator.generate(Generator.java:393) > at org.apache.nutch.crawl.Crawl.main(Crawl.java:116) > > log shows the following: > java.lang.OutOfMemoryError: PermGen space > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:620) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:260) > at java.net.URLClassLoader.access$000(URLClassLoader.java:56) > at java.net.URLClassLoader$1.run(URLClassLoader.java:195) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:188) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at java.lang.ClassLoader.loadClass(ClassLoader.java:251) > at > org.apache.nutch.plugin.Extension.getExtensionInstance(Extension.java:15 6) > at > org.apache.nutch.net.URLNormalizers.getURLNormalizers(URLNormalizers.jav a:170) > at org.apache.nutch.net.URLNormalizers.(URLNormalizers.java:128) > at > org.apache.nutch.crawl.Generator$Selector.configure(Generator.java:109) > at > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58) > at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java: 82) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:250) > at > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:164) > > > > Is this is memory problem? I set the hadoop parameter > > mapred.child.java.opts > -Xmx512m > > > Any idea why this keeps happening? Many thanks!
