On Jan 11, 2007, at 2:28 PM, Scott Atchley wrote:
On Jan 11, 2007, at 2:47 PM, Sam Lang wrote:
So this 11 msec delay could be caused by writing to disk, but if
you don't have data sync enabled (which I don't think you do),
then 11 msec for 1 MB is about 90MB/s to write data to buffer
cache and return the write ack response. We might be delayed
here by having to wait for flow buffers being free'd up from one
of the first 8 1MB messages. Could you increase the
FlowBuffersPerFlow option in the fs.conf to 16 or 20 and see
what happens?
-sam
I will try this.
I know that the default for FlowBuffersPerFlow is 8 per the
webpage, but I count 10 per flow in the log. I also find that
when using the -b flag, that it performs best with multiples of
10 buffers.
You're doing 10MB requests with 1MB flow buffers. I think the 10
flows per request are from that, so the number of flow buffers is
still 8, it just has to reuse 2 of them.
-b specifies the request size doesn't it? Using multiples of 10
for that value improves performance? I really have no idea, but
would have guessed that multiples of the flow buffer size (in your
case 1MB) would have been better.
-sam
With FlowBuffersPerFlow set to 20, I am roughly the same gaps as
before (~11 msec between last send completion and the recv
completion and ~18 msec between transfers).
Hmm..yeah changing that value never seems to do anything. <shrug>
As Julian points out we should probably try to debug this delay on
the server better without getting the disk involved. Unfortunately,
we don't have TAS in place in our current release. You might try
putting the server storage on tmpfs, Pete did this a month or so ago
and was getting repeatable results. He was seeing context switches
costing 10s of microseconds, so that doesn't explain an 11 msec lag.
http://www.beowulf-underground.org/pipermail/pvfs2-developers/2006-
December/002972.html
-sam
Scott
_______________________________________________
Pvfs2-developers mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
