What is the relationship between number of reducers and cpu cores in your setup? I read somewhere that it must be .5 of number of cpu cores.
Thanks. Alex. -----Original Message----- From: Kartashov, Andy <[email protected]> To: user <[email protected]>; bejoy.hadoop <[email protected]> Sent: Tue, Nov 20, 2012 1:51 pm Subject: RE: number of reducers I specify mine inside mapred-site.xml <property> <name>mapred.reduce.tasks</name> <value>20</value> </property> Rgds, AK47 From: Bejoy KS [mailto:[email protected]] Sent: Tuesday, November 20, 2012 3:10 PM To: [email protected] Subject: Re: number of reducers Hi Sasha By default the number or reducers are set to be 1. If you want more you need to specify it as hadoop jar myJar.jar myClass -D mapred.reduce.tasks=20 ... Regards Bejoy KS Sent from handheld, please excuse typos. From: jamal sasha <[email protected]> Date: Tue, 20 Nov 2012 14:38:54 -0500 To: <[email protected]> ReplyTo: [email protected] Subject: number of reducers Hi, I wrote a simple map reduce job in hadoop streaming. I am wondering if I am doing something wrong .. While number of mappers are projected to be around 1700.. reducers.. just 1? It’s couple of TB’s worth of data. What can I do to address this. Basically mapper looks like this For line in sys.stdin: Print line Reducer For line in sys.stdin: New_line = process_line(line) Print new_line Thanks NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel
