when I tested crawling a web deeply (-depth 100 -topN 1000), the job always
failed with the exception below
Exception in thread "main" java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:894)
at org.apache.nutch.crawl.Generator.generate(Generator.java:456)
at org.apache.nutch.crawl.Generator.generate(Generator.java:393)
at org.apache.nutch.crawl.Crawl.main(Crawl.java:116)
log shows the following:
java.lang.OutOfMemoryError: PermGen space
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:260)
at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at
org.apache.nutch.plugin.Extension.getExtensionInstance(Extension.java:156)
at
org.apache.nutch.net.URLNormalizers.getURLNormalizers(URLNormalizers.java:170)
at org.apache.nutch.net.URLNormalizers.(URLNormalizers.java:128)
at
org.apache.nutch.crawl.Generator$Selector.configure(Generator.java:109)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:250)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:164)
Is this is memory problem? I set the hadoop parameter
mapred.child.java.opts
-Xmx512m
Any idea why this keeps happening? Many thanks!
