Thorsten Schuett wrote:
On Wednesday 22 August 2007, Doug Cutting wrote:
Thorsten Schuett wrote:
In my case, it looks as if the loopback device is the bottleneck. So
increasing the number of tasks won't help.
Hmm. I have trouble believing that the loopback device is actually the
bottleneck. What makes you think that it is?
During the copy phase of reduce, the cpu load was very low and vmstat showed
constant reads from the disk at ~15MB/s and bursty writes. At the same time,
data was sent over the loopback device at ~15MB/s. I don't see what else
could limit the performance here. The disk can certainly provide the data at
higher speeds.
Disks can not always provide this rate of reads and writes. Could you
run iostat with -x to see how busy the the disks are? You could post a
small portion of 'iostat -x 3' here.
Raghu.
I'll be happy to repeat my experiments with the MiniMR Code. But I need a
pointer how to proceed/where to start.
Thorsten
