[ https://issues.apache.org/jira/browse/OOZIE-3578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Denes Bodo updated OOZIE-3578: ------------------------------ Attachment: OOZIE-3578-003.patch > MapReduce counters cannot be used over 120 > ------------------------------------------ > > Key: OOZIE-3578 > URL: https://issues.apache.org/jira/browse/OOZIE-3578 > Project: Oozie > Issue Type: Bug > Components: core > Affects Versions: 5.1.0 > Reporter: Denes Bodo > Assignee: Denes Bodo > Priority: Critical > Attachments: OOZIE-3578-001.patch, OOZIE-3578-002.patch, > OOZIE-3578-003.patch > > > When we create a mapreduce action which then creates more than 120 counters > then the following exception is thrown: > {noformat} > org.apache.hadoop.mapreduce.counters.Limits.checkCounters(Limits.java:101) > org.apache.hadoop.mapreduce.counters.Limits.incrCounters(Limits.java:108) > org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounter(AbstractCounterGroup.java:78) > org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounterImpl(AbstractCounterGroup.java:95) > org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.findCounterImpl(AbstractCounterGroup.java:123) > org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:113) > org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:130) > org.apache.hadoop.mapreduce.counters.AbstractCounters.findCounter(AbstractCounters.java:155) > org.apache.hadoop.mapreduce.TypeConverter.fromYarn(TypeConverter.java:264) > org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:383) > org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:859) > org.apache.hadoop.mapreduce.Job$8.run(Job.java:820) > org.apache.hadoop.mapreduce.Job$8.run(Job.java:817) > java.security.AccessController.doPrivileged(Native Method) > javax.security.auth.Subject.doAs(Subject.java:422) > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875) > org.apache.hadoop.mapreduce.Job.getCounters(Job.java:817) > org.apache.hadoop.mapred.JobClient$NetworkedJob.getCounters(JobClient.java:379) > org.apache.oozie.action.hadoop.MapReduceActionExecutor.end(MapReduceActionExecutor.java:252) > org.apache.oozie.command.wf.ActionEndXCommand.execute(ActionEndXCommand.java:183) > org.apache.oozie.command.wf.ActionEndXCommand.execute(ActionEndXCommand.java:62) > org.apache.oozie.command.XCommand.call(XCommand.java:291) > org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:244) > org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:56) > org.apache.oozie.command.XCommand.call(XCommand.java:291) > java.util.concurrent.FutureTask.run(FutureTask.java:266) > org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:210) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748) > {noformat} > It turned out if we use Oozie with Hadoop 3 the MR class called {{Limits}} is > not initialised properly but with default values: > https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/Limits.java#L40 > If we set the "mapreduce.job.counters.max" to 500 in mapred-site.xml or in > core-site.xml has no positive effect. -- This message was sent by Atlassian Jira (v8.3.4#803005)