RE: Injector and Generator Job Failing

2016-10-14 Thread Markus Jelsma 
Well, ERROR log is probably all we need to see, this is the application's main 
stdout which doesnt tell what is wrong.

M.

 
 
-Original message-
> From:shubham.gupta 
> Sent: Friday 14th October 2016 12:53
> To: user@nutch.apache.org
> Subject: Re: Injector and Generator Job Failing
> 
> Due to huge amount of hadoop logging I had only allowed the logging of 
> ERROR messages and above of both hadoop and nutch. Also, enabled the 
> periodic deletion of logs as lot of disk was being utilized. So, I am 
> kind of in the dark here.
> 
> Thanks and Regards,
> Shubham Gupta
> 
> On Friday 14 October 2016 04:11 PM, Markus Jelsma wrote:
> > Check the logs, this only tells you that i failed, not why.
> > M.
> >
> >   
> >   
> > -Original message-
> >> From:shubham.gupta 
> >> Sent: Friday 14th October 2016 12:15
> >> To: user@nutch.apache.org
> >> Subject: Injector and Generator Job Failing
> >>
> >> Hey
> >>
> >> Whenever i run the nutch application, only the injector and generate job
> >> fails.
> >> The path of the plugin folders in conf/nutch-site.xml is correct.
> >>
> >> The following error occurs:
> >> INFO mapreduce.Job: Job job_1476273924585_1272 failed with state FAILED
> >> due to: Task failed task_1476273924585_1272_m_00
> >> Job failed as tasks failed. failedMaps:1 failedReduces:0
> >>
> >> INFO mapreduce.Job: Counters: 9
> >>   Job Counters
> >>   Failed map tasks=4
> >>   Launched map tasks=4
> >>   Other local map tasks=3
> >>   Data-local map tasks=1
> >>   Total time spent by all maps in occupied slots (ms)=17594276
> >>   Total time spent by all reduces in occupied slots (ms)=0
> >>   Total time spent by all map tasks (ms)=2513468
> >>   Total vcore-seconds taken by all map tasks=2513468
> >>   Total megabyte-seconds taken by all map tasks=8978107696
> >> ERROR crawl.*InjectorJob*: InjectorJob: java.lang.RuntimeException: job
> >> failed: name=inject jobid=job_1476273924585_1272
> >>   at 
> >> org.apache.nutch.util.NutchJob.waitForCompletion(NutchJob.java:120)
> >>   at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:231)
> >>   at org.apache.nutch.crawl.InjectorJob.inject(InjectorJob.java:252)
> >>   at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:275)
> >>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> >>   at org.apache.nutch.crawl.InjectorJob.main(InjectorJob.java:284)
> >>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>   at
> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> >>   at
> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>   at java.lang.reflect.Method.invoke(Method.java:606)
> >>   at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> >>   at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> >>
> >> mapreduce.Job: Job job_1476273924585_0973 failed with state FAILED due
> >> to: Task failed task_1476273924585_0973_m_00
> >> Job failed as tasks failed. failedMaps:1 failedReduces:0
> >>
> >> INFO mapreduce.Job: Counters: 8
> >>   Job Counters
> >>   Failed map tasks=4
> >>   Launched map tasks=4
> >>   Other local map tasks=4
> >>   Total time spent by all maps in occupied slots
> >> (ms)=17585463
> >>   Total time spent by all reduces in occupied slots (ms)=0
> >>   Total time spent by all map tasks (ms)=2512209
> >>   Total vcore-seconds taken by all map tasks=2512209
> >>   Total megabyte-seconds taken by all map tasks=8973610548
> >> 16/10/14 04:15:49 ERROR crawl.*GeneratorJob*: GeneratorJob:
> >> java.lang.RuntimeException: job failed: name=generate:
> >> 1476396190-679771226, jobid=job_1476273924585_0973
> >>   at
> >> org.apache.nutch.util.NutchJob.waitForCompletion(NutchJob.java:120)
> >>   at org.apache.nutch.crawl.GeneratorJob.run(GeneratorJob.java:227)
> >>   at
> >> org.apache.nutch.crawl.GeneratorJob.generate(GeneratorJob.java:256)
> >>   at org.apache.nutch.crawl.GeneratorJob.run(GeneratorJob.java:322)
> >>

Re: Injector and Generator Job Failing

2016-10-14 Thread shubham.gupta 
Due to huge amount of hadoop logging I had only allowed the logging of 
ERROR messages and above of both hadoop and nutch. Also, enabled the 
periodic deletion of logs as lot of disk was being utilized. So, I am 
kind of in the dark here.


Thanks and Regards,
Shubham Gupta

On Friday 14 October 2016 04:11 PM, Markus Jelsma wrote:

Check the logs, this only tells you that i failed, not why.
M.

  
  
-Original message-

From:shubham.gupta 
Sent: Friday 14th October 2016 12:15
To: user@nutch.apache.org
Subject: Injector and Generator Job Failing

Hey

Whenever i run the nutch application, only the injector and generate job
fails.
The path of the plugin folders in conf/nutch-site.xml is correct.

The following error occurs:
INFO mapreduce.Job: Job job_1476273924585_1272 failed with state FAILED
due to: Task failed task_1476273924585_1272_m_00
Job failed as tasks failed. failedMaps:1 failedReduces:0

INFO mapreduce.Job: Counters: 9
  Job Counters
  Failed map tasks=4
  Launched map tasks=4
  Other local map tasks=3
  Data-local map tasks=1
  Total time spent by all maps in occupied slots (ms)=17594276
  Total time spent by all reduces in occupied slots (ms)=0
  Total time spent by all map tasks (ms)=2513468
  Total vcore-seconds taken by all map tasks=2513468
  Total megabyte-seconds taken by all map tasks=8978107696
ERROR crawl.*InjectorJob*: InjectorJob: java.lang.RuntimeException: job
failed: name=inject jobid=job_1476273924585_1272
  at org.apache.nutch.util.NutchJob.waitForCompletion(NutchJob.java:120)
  at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:231)
  at org.apache.nutch.crawl.InjectorJob.inject(InjectorJob.java:252)
  at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:275)
  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
  at org.apache.nutch.crawl.InjectorJob.main(InjectorJob.java:284)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:606)
  at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
  at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

mapreduce.Job: Job job_1476273924585_0973 failed with state FAILED due
to: Task failed task_1476273924585_0973_m_00
Job failed as tasks failed. failedMaps:1 failedReduces:0

INFO mapreduce.Job: Counters: 8
  Job Counters
  Failed map tasks=4
  Launched map tasks=4
  Other local map tasks=4
  Total time spent by all maps in occupied slots
(ms)=17585463
  Total time spent by all reduces in occupied slots (ms)=0
  Total time spent by all map tasks (ms)=2512209
  Total vcore-seconds taken by all map tasks=2512209
  Total megabyte-seconds taken by all map tasks=8973610548
16/10/14 04:15:49 ERROR crawl.*GeneratorJob*: GeneratorJob:
java.lang.RuntimeException: job failed: name=generate:
1476396190-679771226, jobid=job_1476273924585_0973
  at
org.apache.nutch.util.NutchJob.waitForCompletion(NutchJob.java:120)
  at org.apache.nutch.crawl.GeneratorJob.run(GeneratorJob.java:227)
  at
org.apache.nutch.crawl.GeneratorJob.generate(GeneratorJob.java:256)
  at org.apache.nutch.crawl.GeneratorJob.run(GeneratorJob.java:322)
  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
  at org.apache.nutch.crawl.GeneratorJob.main(GeneratorJob.java:330)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:606)
  at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
  at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Any suggestions.

--
Thanks and Regards,
Shubham Gupta

RE: Injector and Generator Job Failing

2016-10-14 Thread Markus Jelsma 
Check the logs, this only tells you that i failed, not why.
M.

 
 
-Original message-
> From:shubham.gupta 
> Sent: Friday 14th October 2016 12:15
> To: user@nutch.apache.org
> Subject: Injector and Generator Job Failing
> 
> Hey
> 
> Whenever i run the nutch application, only the injector and generate job 
> fails.
> The path of the plugin folders in conf/nutch-site.xml is correct.
> 
> The following error occurs:
> INFO mapreduce.Job: Job job_1476273924585_1272 failed with state FAILED 
> due to: Task failed task_1476273924585_1272_m_00
> Job failed as tasks failed. failedMaps:1 failedReduces:0
> 
> INFO mapreduce.Job: Counters: 9
>  Job Counters
>  Failed map tasks=4
>  Launched map tasks=4
>  Other local map tasks=3
>  Data-local map tasks=1
>  Total time spent by all maps in occupied slots (ms)=17594276
>  Total time spent by all reduces in occupied slots (ms)=0
>  Total time spent by all map tasks (ms)=2513468
>  Total vcore-seconds taken by all map tasks=2513468
>  Total megabyte-seconds taken by all map tasks=8978107696
> ERROR crawl.*InjectorJob*: InjectorJob: java.lang.RuntimeException: job 
> failed: name=inject jobid=job_1476273924585_1272
>  at org.apache.nutch.util.NutchJob.waitForCompletion(NutchJob.java:120)
>  at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:231)
>  at org.apache.nutch.crawl.InjectorJob.inject(InjectorJob.java:252)
>  at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:275)
>  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>  at org.apache.nutch.crawl.InjectorJob.main(InjectorJob.java:284)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>  at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:606)
>  at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
>  at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> 
> mapreduce.Job: Job job_1476273924585_0973 failed with state FAILED due 
> to: Task failed task_1476273924585_0973_m_00
> Job failed as tasks failed. failedMaps:1 failedReduces:0
> 
> INFO mapreduce.Job: Counters: 8
>  Job Counters
>  Failed map tasks=4
>  Launched map tasks=4
>  Other local map tasks=4
>  Total time spent by all maps in occupied slots 
> (ms)=17585463
>  Total time spent by all reduces in occupied slots (ms)=0
>  Total time spent by all map tasks (ms)=2512209
>  Total vcore-seconds taken by all map tasks=2512209
>  Total megabyte-seconds taken by all map tasks=8973610548
> 16/10/14 04:15:49 ERROR crawl.*GeneratorJob*: GeneratorJob: 
> java.lang.RuntimeException: job failed: name=generate: 
> 1476396190-679771226, jobid=job_1476273924585_0973
>  at 
> org.apache.nutch.util.NutchJob.waitForCompletion(NutchJob.java:120)
>  at org.apache.nutch.crawl.GeneratorJob.run(GeneratorJob.java:227)
>  at 
> org.apache.nutch.crawl.GeneratorJob.generate(GeneratorJob.java:256)
>  at org.apache.nutch.crawl.GeneratorJob.run(GeneratorJob.java:322)
>  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>  at org.apache.nutch.crawl.GeneratorJob.main(GeneratorJob.java:330)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>  at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:606)
>  at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
>  at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> 
> Any suggestions.
> 
> -- 
> Thanks and Regards,
> Shubham Gupta
> 
>