On Jan 10, 2007, at 1:05 PM, Scott Atchley wrote:
Hi all,
I used MPE to measure send, sendunexpected and recv on the client
during pvfs2-cp (from the client to the server) of a (just less
than) 256 MB file. This is using older 2.4 GHz Xeons with MX-2G (2
Gb/s line rate). I am using FlowBufferSizeBytes of 1 MB. I am using
the default FlowBuffersPerFlow. The server did not generate a log
and I am looking into why.
Hi Scott,
Ignore yesterday's email, I was thinking that the events included
both the client and server, but you're only using MPE on the client.
This first image shows the entire copy of the file from client to
server. Recv are blue, send are red and sendunexpected are orange.
There 25 transfer of 10 MB each and the 26th transfer is ~5.5 MB.
Note that there are distinct gaps between transfers in the range of
17-18 msecs.
<pvfs2-cp-3.png>
As I said, I was confused yesterday. These gaps are entirely on the
client. The end of one blue column marks the receive of the write
ack completing, and the start of the next column marks the beginning
of the next request. The only thing that pvfs2-cp is doing in the
meantime, is reading the next 10MB of the local file. I guess you're
getting about 55 MB/s from that local read.
The second image shows one of the 25 bulk IO transfers. There is a
11 msec delay from the last send completing to the recv completing
(the large blue box). On the very far left is the initial
sendunexpected that starts this operation.
<pvfs2-cp-2.png>
So this 11 msec delay could be caused by writing to disk, but if you
don't have data sync enabled (which I don't think you do), then 11
msec for 1 MB is about 90MB/s to write data to buffer cache and
return the write ack response. We might be delayed here by having to
wait for flow buffers being free'd up from one of the first 8 1MB
messages. Could you increase the FlowBuffersPerFlow option in the
fs.conf to 16 or 20 and see what happens?
-sam
The third image is of the start of the operation and includes the
first two 1 MB transfers. At the far left, you see the client post
a recv, then post the sendunexpected that contacts the server to
start the operation. Once the first recv completes (the server has
replied), the client sends the first 1 MB. Jumpshot did not draw
this in a way that is intuitive. The small red column on the left
actually completes just before the large red box on the right.
It is interesting that pvfs2-cp posts the first 1 MB send before
the recv. In actuality, it does not matter because the recv is
posted before the last IO transfer per operation.
The individual 1 MB transfers take about 4.5 msec, which works out
to about 1.8 Gb/s (out of the possible 2 Gb/s). The overall
throughput that pvfs2-cp reports is 1.0 Gb/s.
<pvfs2-cp-1.png>
I wonder if the gaps between operations is due to reading in the
file from disk or something else. Are the delays at the end of the
operation (after the last send completes until the recv completes)
due to the server processing the received data before replying to
the client?
I will try this next on perf.
Scott_______________________________________________
Pvfs2-developers mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
_______________________________________________
Pvfs2-developers mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
