MR job launching is slower

2012-03-20 Thread praveenesh kumar
I have 10 node cluster ( around 24 CPUs, 48 GB RAM, 1 TB HDD, 10 GB ethernet connection) After triggering any MR job, its taking like 3-5 seconds to launch ( I mean the time when I can see any MR job completion % on the screen). I know internally its trying to launch the job,intialize mappers,

Re: MR job launching is slower

2012-03-20 Thread Michael Segel
Hi, First, it sounds like you have 2 6 core CPUs giving you 12 cores not 24. Even though the OS reports 24 cores that's the hyper threading and not real cores. This becomes an issue with respect to tuning. To answer your question ... You have a single 1TB HD. That's going to be a major