>>> Hmm. Can you put a number on how much better RDMA write with >>> immediate is on current HCA hardware? How does using the underlying >>> OpenIB verbs ability to post a list of work requests compare (ie >>> posting an RDMA write followed by a send in one verbs call)? >>> Maybe "post multiple" is a better direction for DAT. >>> >>> >> With post multiple, unlike immediate data, you don't have the >> ability to distinguish between a normal receive and a rdma >> write completion indication on the other end. This is the >> uniqueness of the service that cannot be provided by the post >> multiple. Yes, post multiple would be a nice option for DAT >> it is just a different service. It would also be required to >> conform to the semantics rules of the bundled operations so >> you could not do any optimization tricks under the covers >> with an IB rdma_write_immediate operation. >> > >A post_multiple also requires defining a single "DTO" data >structure. If the post multiple is atomic (meaning all make >it or none do) then it requires an intermediate data structure >to have been created. If it is not atomic there really isn't >reason for it to not just be a utility function layered >above DAT.
That is very good point. And since the emulated immediate data service can't make the atomic guarantee it is the killer argument for just making the service plain - a potentially more efficient write/send. > >What I'm not seeing with the immediate is this urgent need >by the application to be able to use the same 32-bit value >for both an immediate and a 4 byte message that requires >an entire additional API just to support it. Why can't >the application just add a bool to the send message? >Or encode the 32-bits so that they come from disjoint >domains? Some applications can do as you suggest. Some applications can make good use of unambiguous indications where the buffer size, content, or arrival timing is not constrained. Some don't need write notification at all. What's your point? > >There seems to be agreement that a consolidated write-and-send >call would enable the application to get the benefits of >rdma write with immediate whenever the application could >distinguish the two. Well, I think there is agreement that *some* applications can use write-and-send in a beneficial way. But then again, nothing prevents them from doing that now. They do not need an additional API. But again, I don't have an issue with defining a helper function. I do have an issue with defining an API and semantic that says the target side needs to be coded in a way to always deal with both "true" immediate data and emulation. Just define a write/send helper API and the UPL can be coded in a consistent manner if that is a beneficial service. If a true unambiguous indication service is more beneficial or required, it can use the extension and accept the extra complexity. To demand extra complexity in applications that obviously don't need the true immediate data semantic is just wrong in my option. > >I cannot see why doing this is almost free for virtually >all applications, and trivial for the remainder. Adding >and documenting an extra call to deal with such an >extreme corner case that is being presented only in >the abstract is just not justified. This extra capability >has to have enough functionality for enough applications >to justify keeping it on the books, writing test cases >for it, etc. All we're asking is that a write/send combined API not be called immediate data unless it fits the semantics of immediate data. I am puzzled at the resistance this is getting. There is a standards body specification for immediate data. If it is not followed, don't call it immediate data. It's that simple. For those transports that can provide the service, the UPL may be able to gain access to it through an extension. Roy _______________________________________________ openib-general mailing list [email protected] http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
