[ sorry for the dupe, mis-edited Roland's address ] On Tue, 2011-01-18 at 21:31 -0800, Roland Dreier wrote: > Hi Dave, > > > Now that at least one vendor is implementing full support for the SRP > > indirect memory descriptor tables, we can safely expand the sg_tablesize, > > and realize some performance gains, in many cases quite large. I don't > > have vendor code that implements the full support needed for safety, but > > the rareness of FMR mapping failures allows the mapping code to function, > > at a risk, with existing targets. > > Have you considered using memory registration through a send queue (from > the base memory management extensions)? mlx4 at least has support for > this operation, which would let you pre-allocate everything and avoid > the possibility of failure (I think).
I'm looking at this now, and it may not be too bad; I think I can re-use the mapping machinery pretty easily without it getting too ugly. I do have a few questions, though. It looks like I can only have up to 256 keys for each fast page registration MR, so that seems to limit the mappings in flight -- I can see cases where we may run out and have to push the request back to the mid-layer. We'd also be cycling through the key space pretty quickly, negating some of the advantages of being able to invalidate the mappings quickly. I'm thinking about just allocating an MR per request -- this seems like a simple way to avoid the restrictions, but would of course use more MRs. mlx4 seems to allow just shy of 512 K of them, so that's probably not a large concern. What do you think? Is that overkill? Do I misunderstand the BMM extensions? -- Dave Dillow National Center for Computational Science Oak Ridge National Laboratory (865) 241-6602 office -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
