On Wed, 2008-02-06 at 15:22 -0600, Tom Tucker wrote: > Fab: > > This seems broken to me. The app should not have to know whether the remote > peer Linux or Windows.
The rule should be what Fab stated in a previous email: big endian (network order) on the IB "wire". Isn't that sufficient ? -- Hal > Tom > > > On 2/6/08 3:11 PM, "Fab Tillier" <[EMAIL PROTECTED]> wrote: > > > Right, the same code won't work properly. You have to swap on one of the > > machines, it doesn't really matter which unless you care about how the data > > is > > on the wire. If you care about what it looks like on the wire, you should > > swap on the Linux side so that the RKey is always in network order when > > going > > over the network: > > > > - If the Windows machine is exposing a buffer for RDMA access by a Linux > > machine, send the RKey as returned by IBAL (it's already in network order). > > On the Linux machine, swap the received RKey into host order to use it in an > > RDMA operation. The Linux HCA driver will swap the RKey back to network > > order > > when putting it on the wire (in the RDMA header). > > > > - If the Linux machine is exposing a buffer for RDMA access by a Windows > > machine, swap the RKey returned by the OFED stack into network byte order > > before sending it to the other application. On the Windows machine, use the > > RKey value as provided. The HCA driver won't swap it, and put it on the > > wire > > as provided (in the RDMA header). > > > > In IBAL, values that are in network order are identified by their type > > (net32 > > or ib_net32, vs. uint32 for host order). There's no type safety, but the > > API > > declarations document it. > > > > Cheers, > > -Fab > > > > -----Original Message----- > > From: frank zago [mailto:[EMAIL PROTECTED] > > Sent: Wednesday, February 06, 2008 11:31 AM > > To: Fab Tillier > > Cc: [email protected] > > Subject: possible rkey byteswap (was Re: [ofw] what's up with the OFA > > Windows > > project?) > > > > Hello Fab, > > > > Fab Tillier wrote: > >> Hi Frank, > >> > >> The fact that the Windows and Linux stacks treat the rkey differently in > >> the > >> SW should not affect wire interop. In the Windows stack, any values that > >> end > >> up on the wire (like the RKEY in the RDMA header) are always expressed in > >> network order. They're tokens anyhow and swapping them isn't optimal - you > >> end up with the HCA driver swapping it when returning it to the user, just > >> so > >> the user can swap it to put it on the wire, so that the recipient can swap > >> it > >> back to host order, just to have the HCA driver swap it back to network > >> order > >> when performing the RDMA operation. > >> > >> So on the Linux side, applications have to swap the rkey and treat it as a > >> host-order value. On the Windows side they don't. Can you elaborate on > >> what > >> prevents clean RDMA interop between Linux and Windows? > >> > >> -Fab > >> > > I have an application that runs on windows and linux. The code is the > > same. If I do linux<->linux, or windows<->windows, it works. However > > when I do linux<->windows, I have to swap the rkey for rdma operations. > > To me it seems that either linux or windows is byteswapping the rkey > > that's returned to userspace. The current fix is for my app to byteswap > > the rkey when the memory is registered and byteswap it again when > > posting the rdma work request, but only on one of the OS. > > > > Unfortunateley I didn't have the time to fully investigate with an cable > > analyzer to see who is misbehaving or if my analysis is actually correct. > > > > Frank. > > > > _______________________________________________ > > ofw mailing list > > [email protected] > > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw > > > _______________________________________________ > ofw mailing list > [email protected] > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw _______________________________________________ ofw mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw
