An ex post facto notification of a PTE change would enable the RDMA Device driver to know when a Memory Region had been invalidated so that it could probably declare an access violation and tear all the connections using it down.
But if the intent is to allow it to migrate the memory region to the new mapping it would need a more synchronized notice. It needs to be told of a *pending* change, so that it can indicate when it has completed any data movements based on the old data. It can then use the new data. This has generally been discussed as a two part interface: suspend (to request that the old mapping no longer be used) and resume (to resume usage of the mapping with the new values), and it is generally done at a Memory Region scope rather than on a per PTE basis. RDMA has strict ordering requirements. In particular, completing a receive work request represents a guarnatee to the consumer that the prior writes have been updated in its buffer. With an unsynchronized notice that "PTE entry X has been changed" I don't see how it can fulfill those semantics. It cannot know if portions of an RDMA Write were placed to the old physical location, and therefore it cannot know that the entire RDMA Write payload will be in user memory at the anticipated locations when it generates the work completion. If it cannot make that guarantee it is obligated to terminate the connection. On 5/3/05, David Addison <[EMAIL PROTECTED]> wrote: > Ronald G. Minnich wrote: > > > > On Fri, 29 Apr 2005, Greg Lindahl wrote: > > > >>It doesn't imply that there's an MMU, either. I know that Myricom uses a > >>little lookup routine in software on their nic, which most people > >>wouldn't call an MMU. I don't know what Mellanox does for this, they > >>don't talk much about what's hardware and what's software on their nic. > >>I think Quadrics actually uses the TLB of their risc cpu on their nic > >>for this lookup, but that's just a guess. > > > > but only quadrics rewrites the mm layer code .. > > > > > Hi Ron, > as our recent IOPROC patch on lkml shows, it's not that invasive. There > are just 24 hooks added to the Linux VM code paths - which we have been able > to > maintain outside the mainline tree for many years now. > As these hooks only need to synchronise the Elan's MMU state with that of the > CPU, the device drivers calls don't change the Linux MM behaviour. > > We believe the IOPROC patch is generic and powerful and would allow other > RDMA NICs to solve the page registration problems in a different manner. > For NICs which require page registration, new VM hooks can be used to avoid > pages being unloaded whilst DMAs are active. Our latest cut of the IOPROC > patch > has such a hook. > > Cheers > Addy. > _______________________________________________ > openib-general mailing list > [email protected] > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > _______________________________________________ openib-general mailing list [email protected] http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
