can you tell me which limit I should increase,I have increased mapreduce.job.counters.group.name.max=500, mapreduce.job.counters.counter.name.max=500 mapreduce.job.counters.groups.max=500 mapreduce.job.counters.max=500 but I get the same error .should I correct the MRJobConfig.java. like http://www.thecloudavenue.com/2011/12/limiting-usage-counters-in-hadoop.html ?
At 2015-04-28 01:27:53, "Shahab Yunus" <shahab.yu...@gmail.com> wrote: >Just to add one thing form my experience (and please feel free to correct >me if I am wrong) that even when you increase the limit, it can't go above >3000. > >Regards, >Shahab > >On Mon, Apr 27, 2015 at 1:21 PM, Daniel Dai <da...@hortonworks.com> wrote: > >> This is the known issue with rank implementation with counter limit. We >> fixed that in tez but not MR. You can either increase the limit in MR, or >> try Pig on Tez. >> >> Thanks, >> Daniel >> >> On 4/27/15, 3:07 AM, "李运田" <cumt...@163.com> wrote: >> >> >I use rank to add ID now ,but I always get error " >> > FATAL [AsyncDispatcher event handler] >> >org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread >> >org.apache.hadoop.mapreduce.counters.LimitExceededException: Too many >> >counters: 121 max=120 >> > at >> >org.apache.hadoop.mapreduce.counters.Limits.checkCounters(Limits.java:103) >> > at >> >org.apache.hadoop.mapreduce.counters.Limits.incrCounters(Limits.java:110) >> > at >> >org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounter(Abstr >> >actCounterGroup.java:78) >> > at >> >org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounterImpl(A >> >bstractCounterGroup.java:95) >> >" >> >I get many solution in website ,but it is not resolved . perhaps there is >> >something wrong with some sets like " mapreduce.job.counters.max >> > mapreduce.job.counters.group.name.max >> > mapreduce.job.counters.counter.name.max >> > mapreduce.job.counters.groups.max" >> >can you give me some advice? >> > >> > >> > >> > >> > >> > >> > >> > >> >At 2015-04-24 21:56:20, "Alex Nastetsky" <alex.nastet...@vervemobile.com> >> >wrote: >> >>Have you looked at the RANK function? >> >>https://pig.apache.org/docs/r0.11.0/basic.html#rank >> >> >> >>On Fri, Apr 24, 2015 at 5:15 AM, 李运田 <cumt...@163.com> wrote: >> >> >> >>> I have a big data about 10T ,I want to add column number to every >> >>>column >> >>> from 1 to COUNT(my data),,I use two functions in >> >>> >> >>> >> http://stackoverflow.com/questions/9288578/how-can-i-add-row-numbers-for >> >>>-rows-in-pig-or-hive >> >>> and http://www.aiuxian.com/article/p-139530.html >> >>> but I always get many files. every file column number is from 0 to >> >>> ...... perhaps there are many reduces. I dont know how to add column >> >>> number from 1 to COUNT(my data)? >> >>> thank you ,give me some advice or some one can give me some advice >> >>> websites? >> >>> thank you very much. >> >>> >> >>> >> >>
