[EMAIL PROTECTED] wrote: > At 06:00 PM 3/20/2006, Sean Hefty wrote: >> Can you provide more details on this statement? When are you fencing >> the send queue when using memory windows? > > Infiniband 101, and VI before it. Memory windows fence later > operations on the send queue until the bind completes. It's a > misguided attempt to make upper layers' job "easier" because > they can post a bind and then immediately post a send > carrying the rkey. In reality, it introduces bubbles in the > send pipeline and reduces op rates dramatically. > > I argued against them in iWARP verbs, and lost. If Linux > could introduce a way to make the fencing behavior optional, I would > lead the parade. I fear most hardware is implemented otherwise. > > Yes, I know about binding on a separate queue. That doesn't > work, because windows are semantically not fungible (for security > reasons). > > Tom. >
If Memory Windows are efficient enough they don't stall the pipeline. In any event it is far more of a stall to have to set up MRs, MWs and FMRs outside the scope of the pipeline. Having Linux specify hardware is not a good idea. It has enough on its hands understanding the hardware that is already developed. Having Linux developers take part in industry forums where things like Verbs are developed on an industry-wide basis would be an excellent idea in future rounds. The rationale for narrow memory windows is that it makes per QP caching of state far more efficient, which makes providing security easier. Providing deterministic cessation of the use of an invalidated RKey/STag is much easier when the scope of potential use of the RKey/STag is confined to the single QP. Since this corresponds to 90% or more of the usage, optimizing for this case makes sense. The major oversight in the RDMAC verbs (and IB 1.2) is that all of this rationale for narrow memory windows is fully applicable (or more so) to FMRs, which are unfortunately defined as wide (although the terms aren't used, the narrow/ wide terminology came after the fact from IT-API). _______________________________________________ openib-general mailing list [email protected] http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
