Also I should add, disks degrade and continue to work with bad
performance for a long time, not just before beginning to fail. I was
looking at some benchmarks and could isolate some of those machines. See
HADOOP-1649.
If DFS schedules less writes on such nodes, then it helps the whole
cluster and would make speculative execution of tasks in map/reduce more
effective.
Raghu.
Raghu Angadi wrote:
I did not watch large hadoop clusters closely but from my experience of
other large clusters that have heavy disk loads (seek dominated), the
behavior you see seems consistent. Some disks do become very slow and if
they are on some raid, whole raid runs at the speed of the slowest disk.
iostat -x also helps confirm this.
Also comparing ext2 and ext3, ext3 did not have noticeable slow down.
Many times application access patterns tend dictate most of the disk
performance than the native filesystem implementation itself. Filesystem
would probably matter more when we are dealing with lot of small files.
Dennis Kubes wrote:
Can anyone who is running large clusters (50+) tell me what you are
seeing with hard disk failure rates. Something that we are seeing is
that certain machines will consistently have double or triple the load
of other machines with the same tasks. I believe that it is due to
some hard disks beginning to fail, just wanted to know if anyone else
is seeing similar behavior?
Dennis Kubes
