Along with partitioner, try to plug in a combiner. It would provide significant performance gains. Not sure about the algo you use, but might have to tweak that a little to facilitate a combiner.
Thanks, Amogh -----Original Message----- From: Chandraprakash Bhagtani [mailto:[email protected]] Sent: Tuesday, September 29, 2009 12:25 PM To: [email protected] Cc: [email protected] Subject: Re: Best Idea to deal with following situation you can write your custom partitioner instead of hash partitioner On Sat, Sep 26, 2009 at 6:18 AM, Pankil Doshi <[email protected]> wrote: > Hello everyone, > > I have job whose result has only 5 keys but but each key has long list of > values like in 100000's . > What should be best way to deal with it. I feel few of my reducers get over > loaded as two or more keys go to same reduce and hence they have lots of > work to do. > > So what should be best way out with this situation? > > Pankil > -- Thanks & Regards, Chandra Prakash Bhagtani,
