On Tue, 2008-05-27 at 10:33 -0500, Tom Tucker wrote: > On Mon, 2008-05-26 at 16:02 -0700, Roland Dreier wrote: > > > The "invalidate local stag" part of a read is just a local sink side > > > operation (ie no wire protocol change from a read). It's not like > > > processing an ingress send-with-inv. It is really functionally like a > > > read followed immediately by a fenced invalidate-local, but it doesn't > > > stall the pipe. So the device has to remember the read is a "with inv > > > local stag" and invalidate the stag after the read response is placed > > > and before the WCE is reaped by the application. > > > > Yes, understood. My point was just that in IB, at least in theory, one > > could just use an L_Key that doesn't have any remote permissions in the > > scatter list of an RDMA read, while in iWARP, the STag used to place an > > RDMA read response has to have remote write permission. So RDMA read > > with invalidate makes sense for iWARP, because it gives a race-free way > > to allow an STag to be invalidated immediately after an RDMA read > > response is placed, while in IB it's simpler just to never give remote > > access at all. > > > > So I think from an NFSRDMA coding perspective it's a wash... > > When creating the local data sink, We need to check the transport type. > > If it's IB --> only local access, > if it's iWARP --> local + remote access. > > When posting the WR, We check the fastreg capabilities bit + transport type > bit: > If fastreg is true --> > Post FastReg > If iWARP (or with a cap bit read-with-inv-flag) > post rdma read w/ invalidate > else /* IB */ > post rdma read
Steve pointed out a good optimization here. Instead of fencing the RDMA READ here in advance of the INVALIDATE, we should post the INVALIDATE when the READ WR completes. This will avoid stalling the SQ. Since IB doesn't put the LKEY on the wire, there's no security issue to close. We need to keep a bunch of fastreg MR around anyway for concurrent RPC. Thoughts? Tom > post invalidate > fi > else > ... today's logic > fi > > I make the observation, however, that the transport type is now overloaded > with a set of required verbs. For iWARP's case, this means rdma-read-w-inv, > plus rdma-send-w-inv, etc... This also means that new transport types will > inherit one or the other set of verbs (IB or iWARP). > > Tom > > > > - R. > > _______________________________________________ > > general mailing list > > general@lists.openfabrics.org > > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > > > To unsubscribe, please visit > > http://openib.org/mailman/listinfo/openib-general > > _______________________________________________ > general mailing list > general@lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general _______________________________________________ general mailing list general@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general