> I think this may boil down a question of tastefulness. > > In the case where A does a GET at B, the nature of our transport > dictates that B must figure out what he's going to do to satisfy the > GET, get it staged up, then send something back to A saying "here's > the thing you're receiving from". Both the get request and the > response describing where A should pull from will go over the > short-message channel.
Alternatively A can send over the RDMA descriptor for its sink buffer and B can just launch the transfer after it has called lnet_parse() and had its recv() callback called to "receive" the GET. The elan and openib(gen1)/cisco LNDs do this. Where it gets a bit yeuchy is that the last arg of lnet_parse() is 1 (TRUE) in this case, meaning that the LND wants to do RDMA using whatever other info (e.g. RDMA descriptors) it just received, so lnet should call its recv() callback passing the matched buffers (which the recv() will actually send from!!!). If it passed 0 (FALSE), the recv() callback would occur, but just to allow the LND to repost its buffer - it could be some time later that the LND's send() callback would be invoked with the matched buffers and it hugely complicates the LND code to make it hang on to the receive buffer (including peer sink buffer RDMA descriptor) until then. > Looking at existing LNDs that do similar things, it looks like the > response message is often handled more "under the hood". But our > transport is asymmetrical in that way, ie when a receiver starts a > pull, the signalling is handled by the hardware, but when a > transmitter wants to push, he must send a short message to tell the > receiver to pull. > > So the question is, is it a "Bad Thing" (tm) to just invent a new > opcode analogous to LND_GET_REQUEST, and let it percolate through > the same flow of control as everything else? Or do you think I'm > better off having basically two layers of message, one for these out > of band ones, and the other for the usual short-message stuff? I used RDMA READ on the first RDMA capable networks, but stopped using it for iiblnd, (and other LNDs just followed suit) because I was worried about additional resources required in the underlying network to support it. If RDMA READ is a natural choice on Scicortex, it's a no-brainer to eliminate yet another network latency. However, note the restrictions on when you can use it to implement GET/REPLY. When LNET routers are involved, you cannot optimize the GET in this - the LNET GET request must make it all the way to the destination before the source buffers can be matched and the LNET REPLY is forwarded back just like PUTs. -- Cheers, Eric --------------------------------------------------- |Eric Barton Barton Software | |9 York Gardens Tel: +44 (117) 330 1575 | |Clifton Mobile: +44 (7909) 680 356 | |Bristol BS8 4LL Fax: call first | |United Kingdom E-Mail: [EMAIL PROTECTED]| --------------------------------------------------- _______________________________________________ Lustre-devel mailing list Lustre-devel@clusterfs.com https://mail.clusterfs.com/mailman/listinfo/lustre-devel