Yes, your Nutch instance ran out of memory. Give it more if you can, e.g. -Xmx1G or -Xmx2G
Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch ----- Original Message ---- > From: Marcel T <[EMAIL PROTECTED]> > To: [email protected] > Sent: Sunday, May 18, 2008 8:01:52 PM > Subject: job exception > > > when I tested crawling a web deeply (-depth 100 -topN 1000), the job always > failed with the exception below > > Exception in thread "main" java.io.IOException: Job failed! > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:894) > at org.apache.nutch.crawl.Generator.generate(Generator.java:456) > at org.apache.nutch.crawl.Generator.generate(Generator.java:393) > at org.apache.nutch.crawl.Crawl.main(Crawl.java:116) > > log shows the following: > java.lang.OutOfMemoryError: PermGen space > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:620) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:260) > at java.net.URLClassLoader.access$000(URLClassLoader.java:56) > at java.net.URLClassLoader$1.run(URLClassLoader.java:195) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:188) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at java.lang.ClassLoader.loadClass(ClassLoader.java:251) > at > org.apache.nutch.plugin.Extension.getExtensionInstance(Extension.java:156) > at > org.apache.nutch.net.URLNormalizers.getURLNormalizers(URLNormalizers.java:170) > at org.apache.nutch.net.URLNormalizers.(URLNormalizers.java:128) > at > org.apache.nutch.crawl.Generator$Selector.configure(Generator.java:109) > at > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58) > at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:250) > at > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:164) > > > > Is this is memory problem? I set the hadoop parameter > > mapred.child.java.opts > -Xmx512m > > > Any idea why this keeps happening? Many thanks!
