Due to huge amount of hadoop logging I had only allowed the logging of
ERROR messages and above of both hadoop and nutch. Also, enabled the
periodic deletion of logs as lot of disk was being utilized. So, I am
kind of in the dark here.
Thanks and Regards,
Shubham Gupta
On Friday 14 October 2016 04:11 PM, Markus Jelsma wrote:
Check the logs, this only tells you that i failed, not why.
M.
-----Original message-----
From:shubham.gupta <[email protected]>
Sent: Friday 14th October 2016 12:15
To: [email protected]
Subject: Injector and Generator Job Failing
Hey
Whenever i run the nutch application, only the injector and generate job
fails.
The path of the plugin folders in conf/nutch-site.xml is correct.
The following error occurs:
INFO mapreduce.Job: Job job_1476273924585_1272 failed with state FAILED
due to: Task failed task_1476273924585_1272_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0
INFO mapreduce.Job: Counters: 9
Job Counters
Failed map tasks=4
Launched map tasks=4
Other local map tasks=3
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=17594276
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=2513468
Total vcore-seconds taken by all map tasks=2513468
Total megabyte-seconds taken by all map tasks=8978107696
ERROR crawl.*InjectorJob*: InjectorJob: java.lang.RuntimeException: job
failed: name=inject jobid=job_1476273924585_1272
at org.apache.nutch.util.NutchJob.waitForCompletion(NutchJob.java:120)
at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:231)
at org.apache.nutch.crawl.InjectorJob.inject(InjectorJob.java:252)
at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:275)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.nutch.crawl.InjectorJob.main(InjectorJob.java:284)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
mapreduce.Job: Job job_1476273924585_0973 failed with state FAILED due
to: Task failed task_1476273924585_0973_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0
INFO mapreduce.Job: Counters: 8
Job Counters
Failed map tasks=4
Launched map tasks=4
Other local map tasks=4
Total time spent by all maps in occupied slots
(ms)=17585463
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=2512209
Total vcore-seconds taken by all map tasks=2512209
Total megabyte-seconds taken by all map tasks=8973610548
16/10/14 04:15:49 ERROR crawl.*GeneratorJob*: GeneratorJob:
java.lang.RuntimeException: job failed: name=generate:
1476396190-679771226, jobid=job_1476273924585_0973
at
org.apache.nutch.util.NutchJob.waitForCompletion(NutchJob.java:120)
at org.apache.nutch.crawl.GeneratorJob.run(GeneratorJob.java:227)
at
org.apache.nutch.crawl.GeneratorJob.generate(GeneratorJob.java:256)
at org.apache.nutch.crawl.GeneratorJob.run(GeneratorJob.java:322)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.nutch.crawl.GeneratorJob.main(GeneratorJob.java:330)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Any suggestions.
--
Thanks and Regards,
Shubham Gupta
