Hello Sean, I see your point. However I am also writing a middleware and I would like to avoid 'memcpy's as much as possible. Therefore I wanted to avoid the FI_MULTI_RECV solution, even though it looks indeed elegant.
I was thinking to do a single malloc only when I know the size of the incoming buffer, then pass it to libfabric to receive the message and then move it to the user. But this matches with your 'inefficient' approach, which I totally agree. But in my case, the messages can be as big as 50Mb, which translates to big memcpy cost. My next approach will be to ask the user to provide me with a 'maximum' size of the expected message, and allocate buffers of that size during the reception... indeed it doesn't sound the best, but I can avoid copying data around. If I understand correctly, in order to receive any kind of data data with libfabric I will need to: 1) Post the receive buffers (with fi_recv*) 2) Wait for a receive CQ event However I was wondering... is there any way to wait for a 'data pending' event, containing the size of them? Or perhaps another kind of channel through I can send some lightweight (64-bit long) messages? Cheers, Ioannis On 21/01/16 16:30, Hefty, Sean wrote: >> Therefore I wanted to ask, if there is there any way to: 1) Wait >> for CQ when the message arrives 2) Peek the first few bytes >> (header) 3) Then read the entire buffer > > Libfabric is message based, not stream based, and buffering is the > responsibility of the app. The behavior that you're asking for > would result in receive side data copies and is not supported. The > receive side buffering must be provided before the send is > initiated, or the incoming send will be dropped/truncated. > > There are a couple ways to handle this. A less efficient way is to > post a bunch of buffers large enough to receive any incoming > message. A more efficient way is to use FI_MULTI_RECV to post a > single large buffer that will be used to pack received messages > into the larger buffer. > > There shouldn't be a need to carry the message size in an app > header, as the completion will report the size of the message that > was received. But if you really need to implement the above > behavior, it would need to be done as an abstraction over some > method like one mentioned. > > - Sean > _______________________________________________ ofiwg mailing list [email protected] http://lists.openfabrics.org/mailman/listinfo/ofiwg
