Re: [ewg] RE: [ofa-general] how do I use uDAPL with iWARP?

Thu, 03 Apr 2008 10:48:18 -0700 

Guys,
I think this is the same iWARP issue that has been biting me for a while: The client must send the first RDMA message. The dtest app is a peer-2-peer (p2p) application where both sides send immediately after setting up the connection. So dtest doesn't adhere to the iWARP specification (I know: the iWARP spec is broken :). 


News:  I have some prototype FW from chelsio that supports p2p setup and 
with that FW and my associated iw_cxgb3 driver/library changes, then 
dtest seems to work fine.  These changes will be published upstream soon 
in order to support Open MPI and other p2p applications for chelsio.



For this initial release of p2p support over chelsio, the functionality 
will be 100% handled in the iw_cxgb3 driver and fw.  This is similar to 
what iw_nes does today with its send_first module option to send a 0B 
write from the client and defer connection establishment on the server 
until the 0B write is received.  Chelsio will have a similar module 
option called peer2peer (or I could make it the same option name: 
send_first) that will use a 0B read to force the client to send first 
(chelsio cannot use a 0B write for this).  The chelsio FW will defer the 
ESTABLISHED event until the 0B read is received and responded to.



The final proper device-independent solution to this will be done in the 
rdma-cma, the iwarp core and iwarp devices for upstream inclusion as 
well as for ofed-1.4.  Its a much bigger change and will affect the ABI 
for the rdma_cm probably (app can request p2p behavior).  There was a 
thread a while back driven by Arkady at NetApp with details on how we 
will implement this (using a small protocol in mpa start req/rep to 
negotiate this p2p mode).  Stay tuned for more on this.




Steve.


Steve Wise wrote:

I can reproduce this.  Lemme dig into it...

Steve.


Scott Weitzenkamp (sweitzen) wrote:

Steve,

Thanks, that gets further, but dtest still fails.

Client side:

[EMAIL PROTECTED] ~]$ DAPL_MAX_INLINE=64 dtest -h 192.168.0.198
13926 Running as client - OpenIB-cma
13926 Server Name: 192.168.0.198
13926 Server Net Address: 192.168.0.198
13926 Waiting for connect response
13926 Error unexpected conn event : DAT_CONNECTION_EVENT_UNREACHABLE
13926 Error connect_ep: DAT_ABORT

13926: DAPL Test Complete.

13926: Message RTT: Total=      0.00 usec, 10 bursts, itime=      0.00
usec, pc=
0
13926: RDMA write:  Total=      0.00 usec, 10 bursts, itime=      0.00
usec, pc=
0
13926: RDMA read:   Total=      0.00 usec,   4 bursts, itime=      0.00
usec, pc
=0
13926: RDMA read:   Total=      0.00 usec,   4 bursts, itime=      0.00
usec, pc
=0
13926: RDMA read:   Total=      0.00 usec,   4 bursts, itime=      0.00
usec, pc
=0
13926: RDMA read:   Total=      0.00 usec,   4 bursts, itime=      0.00
usec, pc
=0
13926: open:        36619.19 usec
13926: close:       32500.98 usec
13926: PZ create:       7.87 usec
13926: PZ free:         4.05 usec
13926: LMR create:     58.89 usec
13926: LMR free:       11.92 usec
13926: EVD create:      9.78 usec
13926: EVD free:       14.07 usec
13926: EP create:      78.92 usec
13926: EP free:        26.23 usec
13926: TOTAL:         199.79 usec

Server side:

[EMAIL PROTECTED] ~]$ DAPL_MAX_INLINE=64 dtest
11461 Running as server - OpenIB-cma
11461 Server waiting for connect request..
11461 Waiting for connect response

11461 CONNECTED!

11461 Send RMR to remote: snd_msg:
r_key_ctx=bff,pad=0,va=146db580,len=0x40
11461 Waiting for remote to send RMR data
11461 Error waiting on h_dto_rcv_evd: DAT_TIMEOUT_EXPIRED
11461 Error connect_ep: DAT_TIMEOUT_EXPIRED

11461: DAPL Test Complete.

11461: Message RTT: Total=      0.00 usec, 10 bursts, itime=      0.00
usec, pc=
0
11461: RDMA write:  Total=      0.00 usec, 10 bursts, itime=      0.00
usec, pc=
0
11461: RDMA read:   Total=      0.00 usec,   4 bursts, itime=      0.00
usec, pc
=0
11461: RDMA read:   Total=      0.00 usec,   4 bursts, itime=      0.00
usec, pc
=0
11461: RDMA read:   Total=      0.00 usec,   4 bursts, itime=      0.00
usec, pc
=0
11461: RDMA read:   Total=      0.00 usec,   4 bursts, itime=      0.00
usec, pc
=0
11461: open:       900676.01 usec
11461: close:       31543.97 usec
11461: PZ create:       7.87 usec
11461: PZ free:         5.01 usec
11461: LMR create:     51.98 usec
11461: LMR free:       12.16 usec
11461: EVD create:     10.97 usec
11461: EVD free:       12.87 usec
11461: EP create:      77.01 usec
11461: EP free:        30.04 usec
11461: TOTAL:         195.03 usec

Scott


 


-----Original Message-----

From: Steve Wise [mailto:[EMAIL PROTECTED] Sent: Thursday, 
April 03, 2008 9:19 AM

To: Scott Weitzenkamp (sweitzen)
Cc: Joshua Bernstein; OpenFabrics EWG; [ofa_general]
Subject: Re: [ewg] RE: [ofa-general] how do I use uDAPL with iWARP?



Scott Weitzenkamp (sweitzen) wrote:

I tried that, and it didn't work:

[EMAIL PROTECTED] ~]# grep eth /etc/dat.conf

OpenIB-cma u1.2 nonthreadsafe default libdaplcma.so.1 

dapl.1.2 "eth2 0"

""
[EMAIL PROTECTED] ~]# dtest
10194 Running as server - OpenIB-cma
10194 Error dat_ep_create: DAT_INVALID_HANDLE
10194 Error freeing EP: DAT_INVALID_HANDLE DAT_INVALID_HANDLE_EP


try setting DAPL_MAX_INLINE=64



_______________________________________________
ewg mailing list
ewg@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg

_______________________________________________
ewg mailing list
ewg@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg























					Reply via email to