It seems like the job stops if I run it with hadoop jar but not if I run it with java -cp then again there could be something else im missing.
It would be great if it could be fixed as I don't like random errors poping up in my jobs and nor will our QA. I'll try patching crunch with the ticket you listed above and see if that takes care of the issue. On Thu, Aug 8, 2013 at 5:56 AM, Josh Wills <[email protected]> wrote: > Hey Matt, > > It looks like this is the problem: > https://issues.apache.org/jira/browse/MAPREDUCE-4442 > > Is it causing the job to fail? We could add a similar workaround to the one > described in the JIRA. > > J > > > On Wed, Aug 7, 2013 at 5:37 PM, Matt Christiansen <[email protected]> wrote: >> >> Hey guys, im getting this error while running a crunch job on our YARN >> cluster. It doesn't seem to be serious buts generating a lot of QA >> questions: >> >> >> ERROR exec.MRExecutor: Exception thrown fetching job counters for >> stage: com.rr.crunch.ads.KeyValuePairsAggregator: >> >> Avro(/QA/event_logs/2013_08_07/pixel_tracking)+S0+pre-distinct+GBK+post-distinct+S1+SeqFile(/tmp/crunch-1725623982/p1) >> java.io.IOException >> at >> org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:317) >> at >> org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:337) >> at >> org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:476) >> at org.apache.hadoop.mapreduce.Job$8.run(Job.java:757) >> at org.apache.hadoop.mapreduce.Job$8.run(Job.java:754) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAs(Subject.java:415) >> at >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) >> at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:754) >> at >> org.apache.crunch.impl.mr.exec.MRExecutor.execute(MRExecutor.java:69) >> at org.apache.crunch.impl.mr.MRPipeline.run(MRPipeline.java:153) >> at >> com.rr.crunch.ads.KeyValuePairsAggregator.process(KeyValuePairsAggregator.java:79) >> at >> com.rr.crunch.ads.KeyValuePairsAggregator.main(KeyValuePairsAggregator.java:63) >> >> This is with crunch 0.5.0 but I have tried 0.7.0 and get the same >> general error (no message about it being job counters but same strack >> trace and in the code its at the job.getJob().getCounters() part) I >> have tried both the hadoop-1 and hadoop-2 jars; I was wondering if any >> one had come acrossed this before or had any idea > > > > > -- > Director of Data Science > Cloudera > Twitter: @josh_wills
