Saket, Do note that while FileInputFormats and most of its derivatives would respect the "mapred.map.tasks" as a property to get a desired number of splits [1], other formats may not consider it at all (i.e. you need to check sources to make sure they can).
[1] - The getSplits(…) interface of InputFormats accept a number of splits to create, but that's about the level of enforcement there, and returned splits' sizes are not checked against the required amount by the framework. On 06-Jul-2011, at 12:32 AM, Saket Joshi wrote: > Praveen, > > You also need to set the property “mapred.map.tasks” to the value derived > from the following formula ( mapred.tasktracker.map.tasks.maximum * Number of > tasktracker Nodes in your cluster) . I am using Hadoop 0.20.2 and I am able > to increase the number of maps using these two mentoined properties. Similar > settings apply for the reducer . > > -Saket > > From: praveen.pe...@nokia.com [mailto:praveen.pe...@nokia.com] > Sent: Friday, July 01, 2011 1:03 PM > To: mapreduce-user@hadoop.apache.org > Subject: mapred.tasktracker.map.tasks.maximum is not taking into effect > > Hi all, > I am using hadoop 0.20.2. I am setting the property > mapred.tasktracker.map.tasks.maximum = 4 (same for reduce also) on my job > conf but I am still seeing max of only 2 map and reduce tasks on each node. I > know my machine can run 4 maps and 4 reduce tasks in parallel. Is this a bug > in 0.20.2 or am I doing something wrong? > > Configuration conf = new Configuration(); > conf.set("mapred.tasktracker.map.tasks.maximum", "4"); > conf.set("mapred.tasktracker.reduce.tasks.maximum", "4"); > > Thanks > Praveen > > This communication, along with any attachments, may include confidential > TouchCommerce Inc. information. Any distribution or copying of this material > is strictly prohibited. If you are not the intended recipient of this > transmission, please contact the sender immediately.
