On Nov 30, 2006, at 12:13 PM, Sam Lang wrote:
On Nov 30, 2006, at 9:27 AM, Scott Atchley wrote:
Hi all,
No one replied the the original post. The one I am most curious
about is (3). Is FlowBufferSizeBytes available to the BMI
implementations?
Hi Scott,
Sorry for not responding. I've included responses to your
questions inline.
No problem. I have been side-tracked on a couple things as well.
I am thinking about how to handle BMI_mx_memalloc(). I might be
able to assist the reg cache in MX if I do things a certain way.
I mention this below, on the server we always call BMI_memalloc to
allocate buffers and post them to BMI_post_send/BMI_post_recv. On
the client that's not always the case, the user buffer is usually
passed directly to BMI_post_send/BMI_post_recv. Would it help you
to know in advance of the post (or all posts) what the maximum size
of a buffer for BMI_post_send/BMI_post_recv is going to be? We
might be able to add a BMI info option that other methods could
ignore. We do limit it, but its all above the BMI interface that
this is done.
Knowing the max size in advance helps iff the buffers are allocated
using BMI_memalloc(). If not, it does not matter.
1. Is there an upper bound on how many transfer operations
between a pair of hosts at any one time (i.e. between a client
and host)?
There is a max of unexpected requests (UnexpectedRequests option in
the config file, defaults to 50). This limits the number of
unexpected messages that can be handled at once by the server, but
doesn't limit the number of IO operations (large expected messages).
I read this as the server will only service up to 50 unexpected
request, but that the client (or clients) can send more. Is that
correct?
2. If there is a limit in the above and it is greater than 1, is
that value the same for small (unexpected and some expected)
messages and large (expected) messages? Or are large messages
restricted to a lower value?
So I don't think there's a limit at present for expected messages.
The unexpected limit is the one I mentioned above.
Ok.
3. In the thread entitled "libpvfs2 usage", SamL mentioned the
tunable called FlowBufferSizeBytes and that it is set to 256KB by
default. Is this value the upper limit on large messages (i.e.
8KB < large messages <= FlowBufferSizeBytes assuming 8 KB is the
maximum unexpected size)?
Yes, on the server BMI_post_recv and BMI_post_send won't get called
with values larger than that. On the client, there's no strict
limit on the size of the messages posted.
Below, you seem to imply that client messages are broken into
FlowBufferSizeBytes chunks. That would make sense if a server never
allocated more than FlowBufferSizeBytes.
Would the BMI method have any knowledge of what this value is
(i.e. can it use FlowBufferSizeBytes to allocate large buffers)?
On the server it does. BMI_memalloc is called from the flow with
exactly the 256K (FlowBufferSizeBytes) value. The buffer returned
is the one passed to BMI_post_recv or BMI_post_send (although, the
size of the request may be less than 256K). On the client, the
user buffer is used for calls to BMI_post_recv and BMI_post_send,
so you won't see BMI_memalloc calls first. The client does break
up the entire client request into chunks of FlowBufferSizeBytes for
the BMI posts, but you won't know what that value is in advance of
the post operations.
-sam
Ok. Given that the client does not use BMI_memalloc(), I will not try
to do anything for BMI_memalloc() for now other than call malloc().
Do not worry about exposing the FlowBufferSizeBytes value for now.
Thanks,
Scott
_______________________________________________
Pvfs2-developers mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
