Re: [ofa-general] [RFP] support for iWARP requirement - activeconnectside MUST send first FPDU
Felix Marti wrote: -Original Message- From: Tom Tucker [mailto:[EMAIL PROTECTED] Sent: Tuesday, October 23, 2007 9:32 PM To: Felix Marti Cc: Kanevsky, Arkady; Glenn Grundstrom; Sean Hefty; Steve Wise; Roland Dreier; [EMAIL PROTECTED]; OpenFabrics General Subject: Re: [ofa-general] [RFP] support for iWARP requirement - activeconnectside MUST send first FPDU Felix Marti wrote: -Original Message- From: [EMAIL PROTECTED] [mailto:general- [EMAIL PROTECTED] On Behalf Of Kanevsky, Arkady Sent: Tuesday, October 23, 2007 6:26 PM To: Glenn Grundstrom; Sean Hefty; Steve Wise Cc: Roland Dreier; [EMAIL PROTECTED]; OpenFabrics General Subject: RE: [ofa-general] [RFP] support for iWARP requirement - activeconnectside MUST send first FPDU This is still a protocol and should be defined by IETF not OFA. But if we get agreement from all iWARP vendors this will be a good step. [felix] This will not work with a Chelsio RNIC which follows the IETF specification by a) not issuing a 0B RDMA Write to the wire and b) silently consuming an incoming 0B write. Therefore 0B RDMA Writes cannot be 'abused' for such a synchronization mechanism. I believe that the mentioned apps adhering to the iWarp requirement do a 'send' from the active side and only have the passive side issue RDMA ops once the incoming send has been received. I would guess that following a similar model is the best way to go and supported by all iWarp vendors implementing the IETF spec. IMO, the iWARP vendors _must_ get together and work on MPA '2'. Standardizing FPDU 'abuse' might be a good place to start, but it needs to be fixed to support peer-to-peer going forward. In the mean-time, imperfectly hiding the issue in the Firmware, uDAPL, the iWARP CM or anywhere else except the application seems to me to be the only customer friendly solution. [felix] While I'm not against trying to hide the connection migration details somewhere below the ULP, I'm not convinced that the issue is as severe as you make it to be and I would not press to have the issue resolved in a matter that requires a new MPA version. In fact, the different rdma transports (and maybe even different versions of the same transport (in the case of IB)) provide different features and I would assume that ULPs will eventually code to these features and must thus be aware of the underlying transport protocol. In that bigger picture, the connection migration issue at hand seems fairly trivial to solve even if it requires an ULP change... I didn't make an argument about severity. Qualifying the severity is in the customer's purview. I'm simply pointing out the following: a) the perspective that the restriction is trivial is how we got here, b) making the app change is putting a decision in the customer's hands that IMO an iWARP vendor would rather they didn't have to make Do I or don't I support iWARP?, and c) you have the power to hide this behavior for most cases. Finally, I believe RFC means Request for Comment. Well here's one last comment -- Add an FPDU message at the end of MPA exchange and fix the problem in the protocol. If we can not get agreement on it on reflector lets do it at SC'07 OFA dev. conference. Arkady Kanevsky email: [EMAIL PROTECTED] Network Appliance Inc. phone: 781-768-5395 1601 Trapelo Rd. - Suite 16.Fax: 781-895-1195 Waltham, MA 02451 central phone: 781-768-5300 -Original Message- From: Glenn Grundstrom [mailto:[EMAIL PROTECTED] Sent: Tuesday, October 23, 2007 9:02 PM To: Sean Hefty; Steve Wise Cc: Roland Dreier; [EMAIL PROTECTED]; OpenFabrics General Subject: RE: [ofa-general] [RFP] support for iWARP requirement - activeconnect side MUST send first FPDU That is what I've been trying to push. Both MVAPICH2 and OMPI have been open to adjusting their transports to adhere to this requirement. I wouldn't mind implementing something to enforce this in the IWCM or the iWARP drivers IF there was a clean way to do it. So far there hasn't been a clean way proposed. Why can't either uDAPL or iW CM always do a send from the active to passive side that gets stripped off? From the active side, the first send is always posted before any user sends, and if necessary, a user send can be queued by software to avoid a QP/CQ overrun. The completion can simply be eaten by software. On the passive side, you have a similar process for receiving the data. This is similar to an option in the NetEffect driver. A zero byte RDMA write is sent from the active side and accounted for on the passive side. This can
RE: [ofa-general] [RFP] support for iWARP requirement- activeconnectside MUST send first FPDU
The bottom line we need to single solution which works for all vendors. This issue cause interoperability problems. So Customers will stay on the sideline until these type of issues are resolved. Hiding behind protocol holes is not going to help adoption. Will sending 0-size send message from initiator side work? Can IWCM on responder side squeeze 0-size buffer to recv this message and swallow it. Hope that there is no check that need to be done on all comletions? Will work for both interrupt and polling mode? I still believe that it will be simplier to add it to MPA protocol. Thanks, Arkady Kanevsky email: [EMAIL PROTECTED] Network Appliance Inc. phone: 781-768-5395 1601 Trapelo Rd. - Suite 16.Fax: 781-895-1195 Waltham, MA 02451 central phone: 781-768-5300 -Original Message- From: Tom Tucker [mailto:[EMAIL PROTECTED] Sent: Wednesday, October 24, 2007 10:40 AM To: Felix Marti Cc: Kanevsky, Arkady; Roland Dreier; Glenn Grundstrom; OpenFabrics General; [EMAIL PROTECTED] Subject: Re: [ofa-general] [RFP] support for iWARP requirement- activeconnectside MUST send first FPDU Felix Marti wrote: -Original Message- From: Tom Tucker [mailto:[EMAIL PROTECTED] Sent: Tuesday, October 23, 2007 9:32 PM To: Felix Marti Cc: Kanevsky, Arkady; Glenn Grundstrom; Sean Hefty; Steve Wise; Roland Dreier; [EMAIL PROTECTED]; OpenFabrics General Subject: Re: [ofa-general] [RFP] support for iWARP requirement - activeconnectside MUST send first FPDU Felix Marti wrote: -Original Message- From: [EMAIL PROTECTED] [mailto:general- [EMAIL PROTECTED] On Behalf Of Kanevsky, Arkady Sent: Tuesday, October 23, 2007 6:26 PM To: Glenn Grundstrom; Sean Hefty; Steve Wise Cc: Roland Dreier; [EMAIL PROTECTED]; OpenFabrics General Subject: RE: [ofa-general] [RFP] support for iWARP requirement - activeconnectside MUST send first FPDU This is still a protocol and should be defined by IETF not OFA. But if we get agreement from all iWARP vendors this will be a good step. [felix] This will not work with a Chelsio RNIC which follows the IETF specification by a) not issuing a 0B RDMA Write to the wire and b) silently consuming an incoming 0B write. Therefore 0B RDMA Writes cannot be 'abused' for such a synchronization mechanism. I believe that the mentioned apps adhering to the iWarp requirement do a 'send' from the active side and only have the passive side issue RDMA ops once the incoming send has been received. I would guess that following a similar model is the best way to go and supported by all iWarp vendors implementing the IETF spec. IMO, the iWARP vendors _must_ get together and work on MPA '2'. Standardizing FPDU 'abuse' might be a good place to start, but it needs to be fixed to support peer-to-peer going forward. In the mean-time, imperfectly hiding the issue in the Firmware, uDAPL, the iWARP CM or anywhere else except the application seems to me to be the only customer friendly solution. [felix] While I'm not against trying to hide the connection migration details somewhere below the ULP, I'm not convinced that the issue is as severe as you make it to be and I would not press to have the issue resolved in a matter that requires a new MPA version. In fact, the different rdma transports (and maybe even different versions of the same transport (in the case of IB)) provide different features and I would assume that ULPs will eventually code to these features and must thus be aware of the underlying transport protocol. In that bigger picture, the connection migration issue at hand seems fairly trivial to solve even if it requires an ULP change... I didn't make an argument about severity. Qualifying the severity is in the customer's purview. I'm simply pointing out the following: a) the perspective that the restriction is trivial is how we got here, b) making the app change is putting a decision in the customer's hands that IMO an iWARP vendor would rather they didn't have to make Do I or don't I support iWARP?, and c) you have the power to hide this behavior for most cases. Finally, I believe RFC means Request for Comment. Well here's one last comment -- Add an FPDU message at the end of MPA exchange and fix the problem in the protocol. If we can not get agreement on it on reflector lets do it at SC'07 OFA dev. conference. Arkady Kanevsky email: [EMAIL PROTECTED] Network Appliance Inc. phone: 781-768-5395 1601 Trapelo Rd. - Suite 16.Fax: 781-895-1195 Waltham, MA 02451 central phone: 781-768-5300 -Original Message- From: Glenn
RE: [ofa-general] [RFP] support for iWARP requirement - activeconnectside MUST send first FPDU
-Original Message- From: [EMAIL PROTECTED] [mailto:general- [EMAIL PROTECTED] On Behalf Of Kanevsky, Arkady Sent: Tuesday, October 23, 2007 6:26 PM To: Glenn Grundstrom; Sean Hefty; Steve Wise Cc: Roland Dreier; [EMAIL PROTECTED]; OpenFabrics General Subject: RE: [ofa-general] [RFP] support for iWARP requirement - activeconnectside MUST send first FPDU This is still a protocol and should be defined by IETF not OFA. But if we get agreement from all iWARP vendors this will be a good step. [felix] This will not work with a Chelsio RNIC which follows the IETF specification by a) not issuing a 0B RDMA Write to the wire and b) silently consuming an incoming 0B write. Therefore 0B RDMA Writes cannot be 'abused' for such a synchronization mechanism. I believe that the mentioned apps adhering to the iWarp requirement do a 'send' from the active side and only have the passive side issue RDMA ops once the incoming send has been received. I would guess that following a similar model is the best way to go and supported by all iWarp vendors implementing the IETF spec. If we can not get agreement on it on reflector lets do it at SC'07 OFA dev. conference. Arkady Kanevsky email: [EMAIL PROTECTED] Network Appliance Inc. phone: 781-768-5395 1601 Trapelo Rd. - Suite 16.Fax: 781-895-1195 Waltham, MA 02451 central phone: 781-768-5300 -Original Message- From: Glenn Grundstrom [mailto:[EMAIL PROTECTED] Sent: Tuesday, October 23, 2007 9:02 PM To: Sean Hefty; Steve Wise Cc: Roland Dreier; [EMAIL PROTECTED]; OpenFabrics General Subject: RE: [ofa-general] [RFP] support for iWARP requirement - activeconnect side MUST send first FPDU That is what I've been trying to push. Both MVAPICH2 and OMPI have been open to adjusting their transports to adhere to this requirement. I wouldn't mind implementing something to enforce this in the IWCM or the iWARP drivers IF there was a clean way to do it. So far there hasn't been a clean way proposed. Why can't either uDAPL or iW CM always do a send from the active to passive side that gets stripped off? From the active side, the first send is always posted before any user sends, and if necessary, a user send can be queued by software to avoid a QP/CQ overrun. The completion can simply be eaten by software. On the passive side, you have a similar process for receiving the data. This is similar to an option in the NetEffect driver. A zero byte RDMA write is sent from the active side and accounted for on the passive side. This can be turned on and off by compile and module options for compatibility. I second Sean's question - why can't uDAPL or the iw_cm do this? (Yes this adds wire protocol, which requires both sides to support it.) - Sean ___ general mailing list general@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ general mailing list general@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib- general ___ general mailing list general@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [ofa-general] [RFP] support for iWARP requirement - activeconnectside MUST send first FPDU
Felix Marti wrote: -Original Message- From: [EMAIL PROTECTED] [mailto:general- [EMAIL PROTECTED] On Behalf Of Kanevsky, Arkady Sent: Tuesday, October 23, 2007 6:26 PM To: Glenn Grundstrom; Sean Hefty; Steve Wise Cc: Roland Dreier; [EMAIL PROTECTED]; OpenFabrics General Subject: RE: [ofa-general] [RFP] support for iWARP requirement - activeconnectside MUST send first FPDU This is still a protocol and should be defined by IETF not OFA. But if we get agreement from all iWARP vendors this will be a good step. [felix] This will not work with a Chelsio RNIC which follows the IETF specification by a) not issuing a 0B RDMA Write to the wire and b) silently consuming an incoming 0B write. Therefore 0B RDMA Writes cannot be 'abused' for such a synchronization mechanism. I believe that the mentioned apps adhering to the iWarp requirement do a 'send' from the active side and only have the passive side issue RDMA ops once the incoming send has been received. I would guess that following a similar model is the best way to go and supported by all iWarp vendors implementing the IETF spec. IMO, the iWARP vendors _must_ get together and work on MPA '2'. Standardizing FPDU 'abuse' might be a good place to start, but it needs to be fixed to support peer-to-peer going forward. In the mean-time, imperfectly hiding the issue in the Firmware, uDAPL, the iWARP CM or anywhere else except the application seems to me to be the only customer friendly solution. If we can not get agreement on it on reflector lets do it at SC'07 OFA dev. conference. Arkady Kanevsky email: [EMAIL PROTECTED] Network Appliance Inc. phone: 781-768-5395 1601 Trapelo Rd. - Suite 16.Fax: 781-895-1195 Waltham, MA 02451 central phone: 781-768-5300 -Original Message- From: Glenn Grundstrom [mailto:[EMAIL PROTECTED] Sent: Tuesday, October 23, 2007 9:02 PM To: Sean Hefty; Steve Wise Cc: Roland Dreier; [EMAIL PROTECTED]; OpenFabrics General Subject: RE: [ofa-general] [RFP] support for iWARP requirement - activeconnect side MUST send first FPDU That is what I've been trying to push. Both MVAPICH2 and OMPI have been open to adjusting their transports to adhere to this requirement. I wouldn't mind implementing something to enforce this in the IWCM or the iWARP drivers IF there was a clean way to do it. So far there hasn't been a clean way proposed. Why can't either uDAPL or iW CM always do a send from the active to passive side that gets stripped off? From the active side, the first send is always posted before any user sends, and if necessary, a user send can be queued by software to avoid a QP/CQ overrun. The completion can simply be eaten by software. On the passive side, you have a similar process for receiving the data. This is similar to an option in the NetEffect driver. A zero byte RDMA write is sent from the active side and accounted for on the passive side. This can be turned on and off by compile and module options for compatibility. I second Sean's question - why can't uDAPL or the iw_cm do this? (Yes this adds wire protocol, which requires both sides to support it.) - Sean ___ general mailing list general@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ general mailing list general@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib- general ___ general mailing list general@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ general mailing list general@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [ofa-general] [RFP] support for iWARP requirement - activeconnectside MUST send first FPDU
-Original Message- From: Tom Tucker [mailto:[EMAIL PROTECTED] Sent: Tuesday, October 23, 2007 9:32 PM To: Felix Marti Cc: Kanevsky, Arkady; Glenn Grundstrom; Sean Hefty; Steve Wise; Roland Dreier; [EMAIL PROTECTED]; OpenFabrics General Subject: Re: [ofa-general] [RFP] support for iWARP requirement - activeconnectside MUST send first FPDU Felix Marti wrote: -Original Message- From: [EMAIL PROTECTED] [mailto:general- [EMAIL PROTECTED] On Behalf Of Kanevsky, Arkady Sent: Tuesday, October 23, 2007 6:26 PM To: Glenn Grundstrom; Sean Hefty; Steve Wise Cc: Roland Dreier; [EMAIL PROTECTED]; OpenFabrics General Subject: RE: [ofa-general] [RFP] support for iWARP requirement - activeconnectside MUST send first FPDU This is still a protocol and should be defined by IETF not OFA. But if we get agreement from all iWARP vendors this will be a good step. [felix] This will not work with a Chelsio RNIC which follows the IETF specification by a) not issuing a 0B RDMA Write to the wire and b) silently consuming an incoming 0B write. Therefore 0B RDMA Writes cannot be 'abused' for such a synchronization mechanism. I believe that the mentioned apps adhering to the iWarp requirement do a 'send' from the active side and only have the passive side issue RDMA ops once the incoming send has been received. I would guess that following a similar model is the best way to go and supported by all iWarp vendors implementing the IETF spec. IMO, the iWARP vendors _must_ get together and work on MPA '2'. Standardizing FPDU 'abuse' might be a good place to start, but it needs to be fixed to support peer-to-peer going forward. In the mean-time, imperfectly hiding the issue in the Firmware, uDAPL, the iWARP CM or anywhere else except the application seems to me to be the only customer friendly solution. [felix] While I'm not against trying to hide the connection migration details somewhere below the ULP, I'm not convinced that the issue is as severe as you make it to be and I would not press to have the issue resolved in a matter that requires a new MPA version. In fact, the different rdma transports (and maybe even different versions of the same transport (in the case of IB)) provide different features and I would assume that ULPs will eventually code to these features and must thus be aware of the underlying transport protocol. In that bigger picture, the connection migration issue at hand seems fairly trivial to solve even if it requires an ULP change... If we can not get agreement on it on reflector lets do it at SC'07 OFA dev. conference. Arkady Kanevsky email: [EMAIL PROTECTED] Network Appliance Inc. phone: 781-768-5395 1601 Trapelo Rd. - Suite 16.Fax: 781-895-1195 Waltham, MA 02451 central phone: 781-768-5300 -Original Message- From: Glenn Grundstrom [mailto:[EMAIL PROTECTED] Sent: Tuesday, October 23, 2007 9:02 PM To: Sean Hefty; Steve Wise Cc: Roland Dreier; [EMAIL PROTECTED]; OpenFabrics General Subject: RE: [ofa-general] [RFP] support for iWARP requirement - activeconnect side MUST send first FPDU That is what I've been trying to push. Both MVAPICH2 and OMPI have been open to adjusting their transports to adhere to this requirement. I wouldn't mind implementing something to enforce this in the IWCM or the iWARP drivers IF there was a clean way to do it. So far there hasn't been a clean way proposed. Why can't either uDAPL or iW CM always do a send from the active to passive side that gets stripped off? From the active side, the first send is always posted before any user sends, and if necessary, a user send can be queued by software to avoid a QP/CQ overrun. The completion can simply be eaten by software. On the passive side, you have a similar process for receiving the data. This is similar to an option in the NetEffect driver. A zero byte RDMA write is sent from the active side and accounted for on the passive side. This can be turned on and off by compile and module options for compatibility. I second Sean's question - why can't uDAPL or the iw_cm do this? (Yes this adds wire protocol, which requires both sides to support it.) - Sean ___ general mailing list general@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ general mailing list general@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib- general ___ general mailing list
