I specify mine inside mapred-site.xml
<property>
<name>mapred.reduce.tasks</name>
<value>20</value>
</property>
Rgds,
AK47
From: Bejoy KS [mailto:[email protected]]
Sent: Tuesday, November 20, 2012 3:10 PM
To: [email protected]
Subject: Re: number of reducers
Hi Sasha
By default the number or reducers are set to be 1. If you want more you need to
specify it as
hadoop jar myJar.jar myClass -D mapred.reduce.tasks=20 ...
Regards
Bejoy KS
Sent from handheld, please excuse typos.
________________________________
From: jamal sasha <[email protected]>
Date: Tue, 20 Nov 2012 14:38:54 -0500
To: <[email protected]>
ReplyTo: [email protected]
Subject: number of reducers
Hi,
I wrote a simple map reduce job in hadoop streaming.
I am wondering if I am doing something wrong ..
While number of mappers are projected to be around 1700.. reducers.. just 1?
It's couple of TB's worth of data.
What can I do to address this.
Basically mapper looks like this
For line in sys.stdin:
Print line
Reducer
For line in sys.stdin:
New_line = process_line(line)
Print new_line
Thanks
NOTICE: This e-mail message and any attachments are confidential, subject to
copyright and may be privileged. Any unauthorized use, copying or disclosure is
prohibited. If you are not the intended recipient, please delete and contact
the sender immediately. Please consider the environment before printing this
e-mail. AVIS : le pr?sent courriel et toute pi?ce jointe qui l'accompagne sont
confidentiels, prot?g?s par le droit d'auteur et peuvent ?tre couverts par le
secret professionnel. Toute utilisation, copie ou divulgation non autoris?e est
interdite. Si vous n'?tes pas le destinataire pr?vu de ce courriel,
supprimez-le et contactez imm?diatement l'exp?diteur. Veuillez penser ?
l'environnement avant d'imprimer le pr?sent courriel
