Awesome thanks . Works great now
On Tuesday, November 20, 2012, Bejoy KS <[email protected]> wrote: > Hi Sasha > > By default the number or reducers are set to be 1. If you want more you need to specify it as > > hadoop jar myJar.jar myClass -D mapred.reduce.tasks=20 ... > > Regards > Bejoy KS > > Sent from handheld, please excuse typos. > ________________________________ > From: jamal sasha <[email protected]> > Date: Tue, 20 Nov 2012 14:38:54 -0500 > To: <[email protected]> > ReplyTo: [email protected] > Subject: number of reducers > > > Hi, > > I wrote a simple map reduce job in hadoop streaming. > > > > I am wondering if I am doing something wrong .. > > While number of mappers are projected to be around 1700.. reducers.. just 1? > > It’s couple of TB’s worth of data. > > What can I do to address this. > > Basically mapper looks like this > > > > For line in sys.stdin: > > Print line > > > > Reducer > > For line in sys.stdin: > > New_line = process_line(line) > > Print new_line > > > > > > Thanks > > >
