Michael S. Tsirkin wrote: > Quoting r. Pradipta Kumar Banerjee <[EMAIL PROTECTED]>: >> Subject: Re: [openib-general] [PATCH 1/2] perftest: enhancement to rdma_bw >> to allow use of RDMA CM >> >> Michael S. Tsirkin wrote: >>> Quoting r. Pradipta Kumar Banerjee <[EMAIL PROTECTED]>: >>>> IMO using rcq seems to be a generic and better solution. >>> Hmm, I see. Need to document the message format then. >>> We are only pasing the vaddr there, right? >>> >> Michael, >> Actually 'rcq' is being used for handling the 'start' and 'done' messages. >> As for the lid, qpn, psn, rkey and vaddr, these gets exchanged as part of >> the >> rdma_listen/rdma_connect calls. See pp_server_connect and pp_client_connect. >> OTH I tried testing rdma_bw on Ammasso iWARP without exchanging the 'start' >> and >> 'done' messages and it worked. I am not sure if this is the right thing to >> do. >> Maybe Steve can throw more light on this. > > This makes sense. But why do we need the start message then? > Hi Michael, As per my understanding of the mpa/iwarp spec, here is the reason for the 'start' message .
The receipt of RDMA_CM_EVENT_ESTABLISHED event by the user mode application, as part of the rdma_listen/rdma_accept/rdma_connect sequence, is an indication that the MPA startup sequence has successfully completed. Taking the example of the cxgb3 driver - the driver generates the IW_CM_EVENT_ESTABLISHED event and goes to the FPDU mode only after successful completion of the MPA startup sequence. This event eventually gets passed to the user as RDMA_CM_EVENT_ESTABLISHED. (function cm_conn_est_handler in core/iwcm.c, function cma_iw_handler in core/cma.c, function established_upcall in hw/cxgb3/iwch_cm.c) Now, as per the MPA spec, the complete startup negotation is over only when the initiator sends the first FPDU frame and the responder receives it. [ http://www.ietf.org/internet-drafts/draft-ietf-rddp-mpa-05.txt 7.1.2 Connection Startup Rules 4. MPA Responder mode implementations MUST receive and validate at least one FPDU before sending any FPDUs or Markers. Note: this requirement is present to allow the Initiator time to get its receiver into Full Operation before an FPDU arrives, avoiding potential race conditions at the Initiator. This was also subject to some debate in the work group before rough consensus was reached. Eliminating this requirement would allow faster startup in some types of applications. However, that would also make certain implementations (particularly "dual stack") much harder. ] So the 'pp_send_start/pp_wait_for_start' actually takes care of the above requirement, initiator (client) sending a 'start message' as FPDU and the responder (server) receiving the same correctly. ____Client____ ____Server____ Waits for MPA Request Frame Sends MPA Request Frame Waits for incoming MPA Reply Frame Receives MPA Request Frame Enables FPDU decoding (doesn't send any FPDUs) Receives MPA Reply Frame < The above communication sequence is taken care by the rdma_listen - rdma_accept/rdma_connect calls and subsequent generation of RDMA_CM_EVENT_ESTABLISHED event > Send first FPDU Receive first FPDU < The above communication sequence is taken care by the pp_wait_for_start/pp_send_start calls > Hope I have made myself clear without generating further confusions :-) Thanks, Pradipta Kumar. _______________________________________________ openib-general mailing list [email protected] http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
