Comments in-line below...
Kanevsky, Arkady wrote:
Group,
below is proposal on how to resolve peer-to-peer iWARP CM issue
discovered at interop event.
The main issue is that MPA spec (relevant portion of IETF RFC 5044
is below) require that
connection initiator send first message over the established connection.
Multiple MPI implementations and several other apps use peer-to-peer
model.
So rather then forcing all of them to do it on their own, which will
not help with
interop between different implementations, the goal is to extend
lower layers to provide it.
Our first idea was to leave MPA protocol untouched and try to solve
this problem
in iw_cm. But there are too many complications to it. First, in
order to adhere to RFC5044
initiator must send first FPDU and responder process it. But since
the connection is already
established processing FPDU involves ULP on whose behalf the
connection is created.
So either initiator sends a message which generates completion on
responder CQ, thus visible
to ULP, or not.
In the later case, the only op which can do it is
RDMA one, which means
that responder somehow provided initiator S-tag which it can use.
So, this is an extension
to MPA, probably using private data. And that responder upon
receiving it destroy this S-tag.
In any case this is an extension of MPA.
This stag exchange isn't needed if this RDMA op is a 0B READ. The
responder waits for that 0B read and only indicates the rdma connection
is established to its ULP when it replies to the 0B read. In this
scenario, the responder/server side doesn't consume any CQ resources.
But it would require an IRD of at least 1 to be configured on the QP.
The initiator still requires an SQ entry, and possibly a CQ entry, for
initiating the 0B read and handling completion. But its perhaps a
little less painful than doing a SEND/RECV exchange. The read wr could
be unsignaled so that it won't generate a CQE. But it still consumes an
SQ WR slot so the SQ would have to be sized to allow this extra WR. And
I guess the CQ would also need to be sized accordingly in case the read
failed.
In the former, Send is used but this requires a buffer to be posted
to CQ. But since
the same CQ (or SharedCQ) can be used by other connections at the
same time it can cause
the responder CM posted buffer to be consumed by other connection.
This is not acceptable.
So new we consider extension to MPA protocol.
The goal is to be completely backwards compatible to existing version 1.
In a nutshell, use a "flag" in the MPA request message which
indicates that
"ready to receive" message will be send by requestor upon receiving
MPA response message with connection acceptance.
here are the changes to IETF RFC5044
1. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8
9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 0
| | + Key (16 bytes containing "MPA ID Req Frame") + 4 | (4D 50 41
20 49 44 20 52 65 71 20 46 72 61 6D 65) | + Or (16 bytes containing
"MPA ID Rep Frame") + 8 | (4D 50 41 20 49 44 20 52 65 70 20 46 72 61
6D 65) | + Or (16 bytes containing "MPA ID Rtr Frame") + 12 | (4D 50
41 20 49 44 20 52 74 52 20 46 72 61 6D 65) | +
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 16
|M|C|R|S| Res | Rev | PD_Length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
| ~ ~ ~ Private Data ~ | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
2. S: indicator in the Req frame whether or not Requestor will send
Rtr frame.
In Req frame, if set to 1 then Rtr frame will be sent if responder
sends Rep frame with accept bit set. 0 indicate that Rtr frame
will not be sent.
In Rep frame, 0 means that Responder cannot support Rtr frame,
while 1 that it is and is waiting for it.
(While my preference is to handle this as MPA protocol version
matching rules,
proposed method will provide complete backwards compatibility)
Unused by Rtr frame. That is set to 0 in Rtr frame and ignored
by responder.
All other bits M,C,R and remainder of Res treated as in MPA ver 1.
Rtr frame adhere to C bit as specified in Rep frame
First, the RTR frame _must_ be an FPDU for this to work. Thus it
violates the DDP/RDMAP specs because it is an known DDP/RDMAP opcode.
Second, assuming the RTR frame is sent as an FPDU, then this won't work
with existing RNIC HW. The HW will post an async error because the
incoming DDP/RDMAP opcode is unknown.
The only way I see that we can fix this for the existing rnic HW is to
come up with some way to send a valid RDMAP message from the initiator
to the responder under the covers -and- have the responder only indicate
that the connection is established when that FPDU is received.
Chelsio cannot support this hack via a 0B write, but the could support a
0B read or send/recv exchange. But as you indicate, this is very
painful and perhaps impossible to do without impacting the ULP and
breaking verbs semantics.
(that's why we punted on this a year ago :)
Steve.
_______________________________________________
general mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
