That last patch I sent to the list was messed up. This one should apply cleanly.
-sam
async-pvfs2-cp.patch
Description: Binary data
On Jan 12, 2007, at 5:20 PM, Sam Lang wrote:
On Jan 11, 2007, at 1:14 PM, Scott Atchley wrote:On Jan 11, 2007, at 11:31 AM, Sam Lang wrote:On Jan 10, 2007, at 1:05 PM, Scott Atchley wrote:Hi all,I used MPE to measure send, sendunexpected and recv on the client during pvfs2-cp (from the client to the server) of a (just less than) 256 MB file. This is using older 2.4 GHz Xeons with MX-2G (2 Gb/s line rate). I am using FlowBufferSizeBytes of 1 MB. I am using the default FlowBuffersPerFlow. The server did not generate a log and I am looking into why.Hi Scott,Ignore yesterday's email, I was thinking that the events included both the client and server, but you're only using MPE on the client.No problem. I still do not know why I cannot get a log from the server.This first image shows the entire copy of the file from client to server. Recv are blue, send are red and sendunexpected are orange. There 25 transfer of 10 MB each and the 26th transfer is ~5.5 MB. Note that there are distinct gaps between transfers in the range of 17-18 msecs.<pvfs2-cp-3.png>As I said, I was confused yesterday. These gaps are entirely on the client. The end of one blue column marks the receive of the write ack completing, and the start of the next column marks the beginning of the next request. The only thing that pvfs2-cp is doing in the meantime, is reading the next 10MB of the local file. I guess you're getting about 55 MB/s from that local read.That sounds reasonable (these are older, slower machines). I would have assumed that pvfs2-cp would have two buffers so that it can read into one while sending the other.Hi Scott,Attached patch modifies pvfs2-cp to use two buffers. This should remove the 20 msec gaps you're seeing between requests. Initial tests I ran look like they also make the timings with pvfs2-cp more consistent between runs.You'll need to run ./config.status after patching. -sam <async-pvfs2-cp.patch>The second image shows one of the 25 bulk IO transfers. There is a 11 msec delay from the last send completing to the recv completing (the large blue box). On the very far left is the initial sendunexpected that starts this operation.<pvfs2-cp-2.png>So this 11 msec delay could be caused by writing to disk, but if you don't have data sync enabled (which I don't think you do), then 11 msec for 1 MB is about 90MB/s to write data to buffer cache and return the write ack response. We might be delayed here by having to wait for flow buffers being free'd up from one of the first 8 1MB messages. Could you increase the FlowBuffersPerFlow option in the fs.conf to 16 or 20 and see what happens?-samI will try this.I know that the default for FlowBuffersPerFlow is 8 per the webpage, but I count 10 per flow in the log. I also find that when using the -b flag, that it performs best with multiples of 10 buffers.Scott_______________________________________________ Pvfs2-developers mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
_______________________________________________ Pvfs2-developers mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
