On Wed, Aug 12, 2009 at 2:14 PM, Mayuran Yogarajah < [email protected]> wrote:
> Hello, > > Amandeep Khurana wrote: > >> So you are running 16 map tasks per node? Plus 2 reducers? >> >> > Thats correct. > >> I think that's high. With 6gb RAM, you should be looking at around 2 >> map tasks plus 1 reducer... >> I have 9 nodes with quad core + 8gb RAM and I run 2M+1R on each node.. >> >> >> > I thought the number of maps should be set to 1/2 - 2 * number of cpus, > thats why > we set it so high. Right now I've set: > mapred.tasktracker.map.tasks.maximum = 16 > mapred.tasktracker.reduce.tasks.maximum = 16 > Its 2*number of nodes Moreover, its not only the CPU's, but also the RAM that matters.. Plus I/O.. Now, I'm not sure if you are I/O bound on this job or not, but thats also a consideration. Reduce the number to 2+1 and see how it goes. Once things work stably, increase the mapper by 2 and see.. You'll have to try a few times before you'll find out the optimal number for your setup. > > So the max mappers/reducers is 96/96. > > How much heap size have you given your hadoop instance? >> >> Also, is there a lot of processing going on in the mappers and reducers? >> >> >> > Yes these are pretty intensive jobs. > > thanks, > M >
