Someone might correct me if I am wrong, but isn't the number of mappers determined by your input size and hdfs block size?
Example your input per system is 1Mb, setting your block size to 512k should result in only 2 mappers to execute on that system? -Eric From: Marco Zühlke [mailto:[email protected]] Sent: Tuesday, October 30, 2012 8:49 AM To: [email protected] Subject: Memory based scheduling Hi, on our cluster our jobs usually satisfied with less than 2 GB of heap space. so we have on our 8 GB computers 3 maps maximum and on our 16 GB computers 4 maps maximum (we only have quad core CPUs and to have memory left for reducers). This works very well. But now we have a new kind of jobs. Each mapper requires at lest 4 GB of heap space. Is it possible to limit the number of tasks (mapper) per computer to 1 or 2 for these kinds of jobs ? Regards, Marco
