On 3/20/06, Dror Goldenberg <[EMAIL PROTECTED]> wrote: > > > From: Talpey, Thomas [mailto:[EMAIL PROTECTED] > > Sent: Monday, March 20, 2006 10:20 PM > > > > At 02:58 PM 3/20/2006, Roland Dreier wrote: > > >If you want to invalidate each FMR individually, then there's not > > >much > > >point in using FMRs at all. Just register and unregister memory > > >regions. > > I'd say +/- here, because map() speeds up the registration (no > need for cmdif), and unmpap() may slow down the deregistration. > > > > > Really? The idea of FMR, I thought, was to preallocate the > > TPT entry up front (alloc_fmr), and then populate it with the > > lightweight fmr api (map_phys_fmr).
You pre-alloc the MPT entry, but not the MTT entries. You then populate the MTT by doing posted writes to the HCA memory (or host memory for memfree HCAs). > > If I use ib_reg_phys_mr(), then I incur all the preallocation > > overhead each time. Sure the benefit of an FMR isn't merely > > that the hard work is deferred, opening a vulnerability while > > it's pending?? I don't know if allocating MTT entries is really expensive. What costs is the fact that you need to do command interface transactions to write the MTT entries, while FMRs support posted writes. That is, just like with alloc_fmr, you need to reserve and format an MPT for regular memory registrations, which is a command interface transaction. For memory registration, one or more commands precede this to write to the MTT. Thus, a memory registration is at a minimum a 2 command interface transaction operation, potentially more depending on the size of the registration. Deregistration and freeing (not unmapping) an FMR should be equivalent, I would think. I'd spoken with Dror about changing the implementation of memory registration to always use posted writes, and we'd come to the conclusion that this would work, though doing so was not the intended usage and thus not something that was garanteed to work going forward. One of Dror's main concerns was that a future change in firmware could break this. Such a change would allow memory registration to require only a single command interface transaction (and thus only a single wait operation while that command completes). I'd think that was beneficial, but haven't had a chance to poke around to quatify the gains. I'd still be interested in seeing regular registration calls improved, as it's clear that an application that is sensitive about its security must either restrict itself to send/recv, buffer the data (data copy overhead), or register/unregister for each I/O. As to using FMRs to create virtually contiguous regions, the last data I saw about this related to SRP (not on OpenIB), and resulted in a gain of ~25% in throughput when using FMRs vs the "full frontal" DMA MR. So there is definitely something to be gained by creating virutally contiguous regions, especially if you're doing a lot of RDMA reads for which there's a fairly low limit to how many can be in flight (4 comes to mind). - Fab _______________________________________________ openib-general mailing list [email protected] http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
