I have a question that I think I already know the answer to but I would
like verification. I have a demo cluster comprised of two master nodes
and eight slaves (all 1x1.2 Ghz cpu / 1 Gig Ram / 1x250 Gig Sata 7200
rpm hard drives). I'm running small MR processes, about 100-200Gigs of
total data that take about 1-2 hours to process. These small processes
seem to work fine. However, I'm starting to run larger processes on them
(5-8 hour processes with 200-300Gigs of data) and the hard drives keep
dieing. I know I'm not running out of space, the hard drives really are
crashing under the load. I don't think it's over heating because the
server room temp is a constant 68-72 degrees. I'm running under the
default configuration, 2 maps + 2 reduces per node. I suspect that
since I only have one hard drive per node it is almost continuously
writing to at least four different files (on the same HD) and its just
thrashing the read/write head and the motor. I assumed my first batch of
HD's were just bad but I've just had 4 more brand new drives fail within
a week so I think I'm pushing it too hard.
If this is the problem, do I need to be sure and place my dfs data and
mapred data directories onto separate physical drives? If I upgrade to
some dell servers with dual core and 4-5HD's per node will hadoop take
care of balancing the load between hard drives (dfs/mapred/scratch storage)?
Regards,
Tim Nelson
- Hard drives dropping like flies Tim Nelson
-