there's one thing i want to clarify that you can use multi-reducers to sort
the data globally and then cat all the parts to get the top n records. The
data in all parts are globally in order.
Then you may find the problem is much easier.
在 2013-2-2 下午3:18，praveenesh kumar praveen...@gmail.com写道：
My actual problem is to rank all values and then run logic 1 to top n%
values and logic 2 to rest values.
1st - Ranking ? (need major suggestions here)
2nd - Find top n% out of them.
Then rest is covered.
Regards
Praveenesh
On Sat, Feb 2, 2013 at 1:42 PM, Lake Chang lakech...@gmail.com wrote:
My suggestion is to use secondary sort with a single reducer. That easy you
can easily extract the top N. If you want to get the top N% you'll need an
additional phase to determine how many records this N% really is.
--
Met vriendelijke groet,
Niels Basjes
(Verstuurd vanaf mobiel )
Op 2 feb.
Note that a one reducer isn't always the solution. If you know your
key space boundaries, consider using a total-order-partition to scale
the app/job and make use of nodes on the cluster.
On Sat, Feb 2, 2013 at 10:35 AM, praveenesh kumar praveen...@gmail.com wrote:
I am looking for a better
I am looking for a better solution for this.
1 way to do this would be to find top N values from each mappers and
then find out the top N out of them in 1 reducer. I am afraid that
this won't work effectively if my N is larger than number of values in
my inputsplit (or mapper input).
Otherway
Actually what I am trying to find to top n% of the whole data.
This n could be very large if my data is large.
Assuming I have uniform rows of equal size and if the total data size
is 10 GB, using the above mentioned approach, if I have to take top
10% of the whole data set, I need 10% of 10GB
Thanks for that Russell. Unfortunately I can't use Pig. Need to write
my own MR job. I was wondering how its usually done in the best way
possible.
Regards
Praveenesh
On Sat, Feb 2, 2013 at 1:00 PM, Russell Jurney russell.jur...@gmail.com wrote:
Pig. Datafu. 7 lines of code.
