I use rank to add ID now ,but I always get error "
FATAL [AsyncDispatcher event handler]
org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread
org.apache.hadoop.mapreduce.counters.LimitExceededException: Too many counters:
121 max=120
at
org.apache.hadoop.mapreduce.counters.Limits.checkCounters(Limits.java:103)
at
org.apache.hadoop.mapreduce.counters.Limits.incrCounters(Limits.java:110)
at
org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounter(AbstractCounterGroup.java:78)
at
org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounterImpl(AbstractCounterGroup.java:95)
"
I get many solution in website ,but it is not resolved . perhaps there is
something wrong with some sets like " mapreduce.job.counters.max
mapreduce.job.counters.group.name.max
mapreduce.job.counters.counter.name.max
mapreduce.job.counters.groups.max"
can you give me some advice?
At 2015-04-24 21:56:20, "Alex Nastetsky" <alex.nastet...@vervemobile.com> wrote:
>Have you looked at the RANK function?
>https://pig.apache.org/docs/r0.11.0/basic.html#rank
>
>On Fri, Apr 24, 2015 at 5:15 AM, 李运田 <cumt...@163.com> wrote:
>
>> I have a big data about 10T ,I want to add column number to every column
>> from 1 to COUNT(my data),,I use two functions in
>> http://stackoverflow.com/questions/9288578/how-can-i-add-row-numbers-for-rows-in-pig-or-hive
>> and http://www.aiuxian.com/article/p-139530.html
>> but I always get many files. every file column number is from 0 to
>> ...... perhaps there are many reduces. I dont know how to add column
>> number from 1 to COUNT(my data)?
>> thank you ,give me some advice or some one can give me some advice
>> websites?
>> thank you very much.
>>
>>
