[openib-general] RE: [RFC] DAT 2.0 immediate data proposal

[Kanevsky, Arkady]

Arlin, comments inline.   Arkady Kanevsky                       email: [EMAIL PROTECTED] Network Appliance Inc.               phone: 781-768-5395 1601 Trapelo Rd. - Suite 16.        Fax: 781-895-1195 Waltham, MA 02451                   central phone: 781-768-5300   From: Davis, Arlin R [mailto:[EMAIL PROTECTED] Sent: Monday, January 23, 2006 7:15 PM To: Kanevsky, Arkady; Lentini, James Cc: openib-general@openib.org; [EMAIL PROTECTED] Subject: RE: [RFC] DAT 2.0 immediate data proposal Arkady,   Response inline…   From: Kanevsky, Arkady [mailto:[EMAIL PROTECTED] Sent: Tuesday, January 17, 2006 7:16 AM To: Davis, Arlin R; Lentini, James Cc: [EMAIL PROTECTED]; openib-general@openib.org Subject: RE: [RFC] DAT 2.0 immediate data proposal   Arlin, a few things need to be addressed.   1. correlation with local and remote invalidate This potentially effects both DAT_DTOs and post operations     How does this differ from normal sends or writes? [AK] We had added a new Send_with_Invalidate. The completion also states whether RMR was invalidated and which one. But the text for interaction is added through out the completion and post operations. See the latest draft of uDAPL and kDAPL 1.3 specs on the DAT reflector.     2. Need a precise defintion for CONFIRM_FLAG definition in a transport independent fashion. What guarantees DAT Provider "provides" on successful local completion? Remote end guarantee?   My understanding what you are trying to do is create 2 models one IB and one for iWARP. So for IB Consumers will use CONFIRM_FLAG and for iWARP IMMED_FLAG. Provider will indicate in Provider_attr which model it supports.   The issue I have with it is that I do not see a model that Consumer can use to create a transport independent code. It looks like Immed_flag can be made transport independent. But with "sender" specifying the behavior a protocol extension is needed for IB. IB will always deliver Immediate data in the header not a payload and remote Provider can control how it is delivered to a Consumer. But this means that there is no need for DTO_flags for Send side. Instead it can be used for Recv side or controlled purely by Provider.   Maybe we need to just go back to one model and always deliver via the event? With the post_recv_immed requirements, other transports have a mechanism to emulate and create the necessary resources on the recv side to place idata and copy to event when operation is completed. Would this work for iWARP?   Two different models for receiving idata should be avoided if at all possible. [AK] Caitlin already responded to this.    3. Need to define error behavior. for new operations, async errors, EP behavior.   I will work on updating the draft. post_send_immed will look much like post_send and post_rdma_write_immed will look a lot like post_rdma_write with some additional errors based on the post receive buffer requirement.   [AK] Also consider if you want to add remote invalidate to the new operation.    4. Need to define DAT_Provider attributes for immediate data and dto_flags behavior     5. Does Solicited_wait completion_flag value now applicable for RDMA_write for immediate data?   yes, applicable to  send, send_immed, and write_immed   6. Is dto_completion_data xfer_length include immediate_data size or not?   no [AK] It can work both ways. Either we include 4 extra bytes for immediate data or not. Consumer just have to know. The real data always starts at 4 byte boundary into the buffer is immediate data is returned inline. We need to state how immediate data is positioned if it is smaller than 4 bytes.    7. what memory privilages needed for a recv buffer for immediate data?   Based on the operation… write_immed would require write privileges and send_immed would require recv privileges.    8. SRQ interaction?   Good question. all post_recv_immed or all post_recv?   [AK] Will this work for the user model? Not supporting handling immediate recv and regular recv with potential immediate data on one SRQ.    9. What happens of buffer for recv operation NOT recv_immed is matched for incomming recv/rdma_write op?   The rules should be: Can receive a send, send_immed, or write_immed with recv_immed. Cannot receive send_immed or write_immed on a recv.   However, I am not sure how you would enforce this on IB (DTO error on the receiving side?) since the idata is delivered via CQ and does not require a special receive post descriptor.       [AK] We can make this Provider attribute. Or we can state that if immed data is return in event then there is no error for recv.    10. Change dat_ep_post_write_immed to dat_ep_post_rdma_write_immed to be consistent with current terminology.   Ok   11. Need to cleanup operation description to make it clear that Send|RDMA_write and immediate data part is a single atomic operation. The current "followed by" language is misleading. Make it explicit that there is a single local DTO completion and single remote DTO completion.   Ok, I will clean that up   12. Is your intension that post_recv_immed can ONLY except immediate data and is not capable to recv any message?   No, the intention is to extend the post_recv to handle 32bit idata which may arrive with or without other send or rdma_write data.   Does it make more sense to add a dto_flags to the existing post_recv?   13. size should be num_segments for dat_ep_post_recv_immed()   ok

_______________________________________________
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general