Well, ERROR log is probably all we need to see, this is the application's main stdout which doesnt tell what is wrong.
M. -----Original message----- > From:shubham.gupta <[email protected]> > Sent: Friday 14th October 2016 12:53 > To: [email protected] > Subject: Re: Injector and Generator Job Failing > > Due to huge amount of hadoop logging I had only allowed the logging of > ERROR messages and above of both hadoop and nutch. Also, enabled the > periodic deletion of logs as lot of disk was being utilized. So, I am > kind of in the dark here. > > Thanks and Regards, > Shubham Gupta > > On Friday 14 October 2016 04:11 PM, Markus Jelsma wrote: > > Check the logs, this only tells you that i failed, not why. > > M. > > > > > > > > -----Original message----- > >> From:shubham.gupta <[email protected]> > >> Sent: Friday 14th October 2016 12:15 > >> To: [email protected] > >> Subject: Injector and Generator Job Failing > >> > >> Hey > >> > >> Whenever i run the nutch application, only the injector and generate job > >> fails. > >> The path of the plugin folders in conf/nutch-site.xml is correct. > >> > >> The following error occurs: > >> INFO mapreduce.Job: Job job_1476273924585_1272 failed with state FAILED > >> due to: Task failed task_1476273924585_1272_m_000000 > >> Job failed as tasks failed. failedMaps:1 failedReduces:0 > >> > >> INFO mapreduce.Job: Counters: 9 > >> Job Counters > >> Failed map tasks=4 > >> Launched map tasks=4 > >> Other local map tasks=3 > >> Data-local map tasks=1 > >> Total time spent by all maps in occupied slots (ms)=17594276 > >> Total time spent by all reduces in occupied slots (ms)=0 > >> Total time spent by all map tasks (ms)=2513468 > >> Total vcore-seconds taken by all map tasks=2513468 > >> Total megabyte-seconds taken by all map tasks=8978107696 > >> ERROR crawl.*InjectorJob*: InjectorJob: java.lang.RuntimeException: job > >> failed: name=inject jobid=job_1476273924585_1272 > >> at > >> org.apache.nutch.util.NutchJob.waitForCompletion(NutchJob.java:120) > >> at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:231) > >> at org.apache.nutch.crawl.InjectorJob.inject(InjectorJob.java:252) > >> at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:275) > >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > >> at org.apache.nutch.crawl.InjectorJob.main(InjectorJob.java:284) > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >> at > >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > >> at > >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >> at java.lang.reflect.Method.invoke(Method.java:606) > >> at org.apache.hadoop.util.RunJar.run(RunJar.java:221) > >> at org.apache.hadoop.util.RunJar.main(RunJar.java:136) > >> > >> mapreduce.Job: Job job_1476273924585_0973 failed with state FAILED due > >> to: Task failed task_1476273924585_0973_m_000000 > >> Job failed as tasks failed. failedMaps:1 failedReduces:0 > >> > >> INFO mapreduce.Job: Counters: 8 > >> Job Counters > >> Failed map tasks=4 > >> Launched map tasks=4 > >> Other local map tasks=4 > >> Total time spent by all maps in occupied slots > >> (ms)=17585463 > >> Total time spent by all reduces in occupied slots (ms)=0 > >> Total time spent by all map tasks (ms)=2512209 > >> Total vcore-seconds taken by all map tasks=2512209 > >> Total megabyte-seconds taken by all map tasks=8973610548 > >> 16/10/14 04:15:49 ERROR crawl.*GeneratorJob*: GeneratorJob: > >> java.lang.RuntimeException: job failed: name=generate: > >> 1476396190-679771226, jobid=job_1476273924585_0973 > >> at > >> org.apache.nutch.util.NutchJob.waitForCompletion(NutchJob.java:120) > >> at org.apache.nutch.crawl.GeneratorJob.run(GeneratorJob.java:227) > >> at > >> org.apache.nutch.crawl.GeneratorJob.generate(GeneratorJob.java:256) > >> at org.apache.nutch.crawl.GeneratorJob.run(GeneratorJob.java:322) > >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > >> at > >> org.apache.nutch.crawl.GeneratorJob.main(GeneratorJob.java:330) > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >> at > >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > >> at > >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >> at java.lang.reflect.Method.invoke(Method.java:606) > >> at org.apache.hadoop.util.RunJar.run(RunJar.java:221) > >> at org.apache.hadoop.util.RunJar.main(RunJar.java:136) > >> > >> Any suggestions. > >> > >> -- > >> Thanks and Regards, > >> Shubham Gupta > >> > >> > >
