I have a question that I think I already know the answer to but I would like verification. I have a demo cluster comprised of two master nodes and eight slaves (all 1x1.2 Ghz cpu / 1 Gig Ram / 1x250 Gig Sata 7200 rpm hard drives). I'm running small MR processes, about 100-200Gigs of total data that take about 1-2 hours to process. These small processes seem to work fine. However, I'm starting to run larger processes on them (5-8 hour processes with 200-300Gigs of data) and the hard drives keep dieing. I know I'm not running out of space, the hard drives really are crashing under the load. I don't think it's over heating because the server room temp is a constant 68-72 degrees. I'm running under the default configuration, 2 maps + 2 reduces per node. I suspect that since I only have one hard drive per node it is almost continuously writing to at least four different files (on the same HD) and its just thrashing the read/write head and the motor. I assumed my first batch of HD's were just bad but I've just had 4 more brand new drives fail within a week so I think I'm pushing it too hard.

If this is the problem, do I need to be sure and place my dfs data and mapred data directories onto separate physical drives? If I upgrade to some dell servers with dual core and 4-5HD's per node will hadoop take care of balancing the load between hard drives (dfs/mapred/scratch storage)?

Regards,
Tim Nelson

Reply via email to