At 06:00 PM 3/20/2006, Sean Hefty wrote:
>Can you provide more details on this statement? When are you fencing the send
>queue when using memory windows?
Infiniband 101, and VI before it. Memory windows fence later operations
on the send queue until the bind completes. It's a misguided attempt to
make upper layers' job "easier" because they can post a bind and then
immediately post a send carrying the rkey. In reality, it introduces bubbles
in the send pipeline and reduces op rates dramatically.
The requirement / semantics were derived from the ULP being used to construct the technology. The combination of a bind-n-send operation was to reduce the software interactions with the device by consolidating this into a combo operation. I do not follow your logic that this creates a bubble in the send pipeline as there were also ordering and correctness issues w.r.t. subsequent operations to the send. The bind-n-send is a single operation and its fence semantics were required to allow the bind to complete before informing the remote of the subsequent information in order to avoid race conditions.
I argued against them in iWARP verbs, and lost. If Linux could introduce
a way to make the fencing behavior optional, I would lead the parade.
I fear most hardware is implemented otherwise.
Hardware generally implements operations in the order they are posted to a given QP, i.e. it is a serial execution flow that allows pipelined operations to be posted and executed by the hardware. Scaling is achieved by executing across a set of QP and thus a set of resources. The ordering domain requirements are kept simple to allow low-cost hardware implementations. This does not preclude software from executing across a set of QP in any order that it desires.
Yes, I know about binding on a separate queue. That doesn't work, because windows are semantically not fungible (for security reasons).
You could always simply allow a region to be accessible across multiple operations but then again storage argued that it must only be accessible for a single op thus things like FMR, bind-n-send, etc. were all created. To say that storage was not listened to or their needs were not met or balanced against what is practical to implement in either the creation of IB or iWARP is simply incorrect.
Mike
Tom.
_______________________________________________
openib-general mailing list
[email protected]
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
_______________________________________________ openib-general mailing list [email protected] http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
