[EMAIL PROTECTED] wrote on Wed, 18 Oct 2006 13:15 -0400:
> Yes, this is exactly what we are seeing. I have never tried a single
> buffer, but looking a bit into the kernel code I can see some areas that
> are a little inefficient.
> 1) In wait_for_a_slot <ident?v=pvfs2;i=wait_for_a_slot>, while holding a
> spinlock, a thread must linearly search through all buffers to find an
> available buffer. If one is found, then fine. If all buffers are full
> (which is probably the common case if doing large I/O), it sleeps until
> woken up, at which point it starts all over again, rescanning the entire
> list for a buffer. Technically, we could have starvation for a thread,
> who continually gets unlucky about finding an empty buffer. My guess
> would be that this code is small should happen really fast no matter
> what, but who knows.....
There's 5 4 MB buffers passed in from userspace. Doing 5
comparisons in a loop on slargs->slot_array[i] is not going to take
any time at all. The use of add_wait_queue_exclusive() makes sure
that there is no starvation, although you might still wait for a
good number of previous threads to do their thing.
If you decide that you want 64 small buffers instead, you'd probably
want to switch over to a pair of linked lists instead, being sure to
put back free ones at the head to keep them hot.
There could be bugs though.
I can't think of why it gets slow as numbufs goes up. Maybe try
oprofile sometime.
> 2) With multiple buffers, the threads will be fighting over using kmap
> to copy the data to the mmapped buffer. From my understanding of the
> kernel (which may be outdated), there are very few kmap spinlocks
> available, effectively serializing the process of copying data into the
> mmapped buffers. As we increase the number of buffers, this contention
> will increase and the time to copy the data for any single buffer will
> increase.
Just some comments to this. kmap() only does something on highmem
pages on 32-bit machines. On x86-64 and the like, kmap() is a noop.
So the problem eventually goes away. :) On 32-bit machines, kmap()
is a function that might sleep. There are a finite number of kernel
page table entries for kmapped pages, and you might end up waiting
for one. But in practice, there are 512 or 1024 on x86, so probably
no waiting. There is a single lock protecting kmap(), though, for a
bit of contention.
Still, may want to do all this on opteron just to get rid of one
possible source of slowness.
-- Pete
_______________________________________________
Pvfs2-developers mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
