On Oct 13, 2006, at 10:00 PM, Murali Vilayannur wrote:
Hi Sam,
Dean and I are looking at trying to push the efficiency of
requests from the kernel module up through the device to client-
core. I added the --threaded option to the client to allow the
client-core to run with multiple threads (one each for bmi, dev,
and main -- and also a remount thread, but lets ignore that for
now), so the device thread should be able to keep pulling requests
of the device without having to wait for bmi operations to complete.
Cool!
This could address some of the performance problems that Phil also
had pointed a while back where multiple outstanding requests were
slower than a single outstanding request.
Well it doesn't seem to make a difference, at least with the
workloads that we were trying.
PINT_dev_test_unexpected takes an incount of 5, so its only going
to read at most 5 requests off the device for each call. Once it
returns, each of the unexpected requests is added to the completed
jobs array and then we signal the jobs completed condition
variable _for each request_. It seems like this will be 5x the
number of context switches between the device thread and the main
thread that we need.
Also, we poll every time before reading another request off the
device. What about trying to read a number of requests off the
device at once with one read (or possibly a readv so we can keep
separate buffers per request).
Hmm.. both of these are good points. I had dabbled with doing a
readv a while back. It might make a difference although I suspect
this might be in the noise region since
if there are requests to be serviced, poll() will only take the
time of a syscall which should be pretty fast these days.. but
worth a shot.
Also, it looks like we do a malloc for each new request buffer,
and then a free once we're done with it, and a memset of the info
struct. It seems like we could manage the buffers on the stack
instead of the heap, and save on a few system calls there.
Now we are definitely in the noise region.. :) just kidding.
glibc's malloc implementation should typically amortize overheads
in invoking system calls (sbrk etc).
Dean was seeing memset at the top of list while running oprofile on
pvfs2-client-core. malloc and free were also up there.
For both threaded and nonthreaded, with the workload that Dean is
using, he found that the PINT_dev_test_unexpected always returned
5 requests in the outcount. So it looks like there are always
requests sitting on the device, waiting to be read by client-
core. Are we just not able to process requests fast enough
through BMI and the state machines, or is the cost of polling and
signaling every time we read a request off the device slowing us
down? In other words, does it make sense to rework the code a
little bit or will we just get bottlenecked elsewhere?
It is definitely interesting to try all this out, but I am not sure
if the bottlenecks are here or elsewhere.
What does this workload do by the way?
If I understand it correctly, there are a number of threads doing
simultaneous reads or writes (64K block sizes) on the same file.
-sam
thanks,
Murali
_______________________________________________
Pvfs2-developers mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
