Or, Are these results after you have applied Eli's fix ? Sagi
-----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Or Gerlitz Sent: Tuesday, July 01, 2008 9:42 AM To: Oren Meron Cc: Eli Cohen; [email protected] Subject: [ofa-general] Re: performance drop for datagram mode with the newconnectx FW > Using the new connectx FW (2.5), I see performance drop to almost zero > with ipoib datagram mode. The code that runs on these systems is ofed > 1.3 and not mainline kernel, details below. > > Running netperf With connected mode (64k MTU) I get about 950MB/s > where with datagram mode (2k MTU) I get only 20-40MB/s. I used to see > about 650MB/s and above with FW 2.3 and datagram mode. Not that it > could explain the drop, but the NIC reports to the OS stateless > offload support - /sys/class/net/ib1/features is 0x11423 I think to manage narrowing this a little further, the issue seems most notable under long messages sent over datagram mode, could be an issue in the LSO engine ?! see in this table, ==================================================== mode mtu size bw note ==================================================== datagram 2044 64000 30 <---------- problem datagram 2044 2000 430 datagram 2044 2000 300 TCP_NODELAY ----------------------------------------------------- connected 2044 64000 450 connected 2044 2000 450 connected 2044 2000 300 TCP_NODELAY --------------------------------------------------- connected 64000 64000 930 connected 64000 2000 930 connected 64000 2000 470 TCP_NODELAY ==================================================== notes: - in all cases, I have set the mtu manually - verbs tests of bidirectional bandwidth shows that the HCA UD and RC engines work very well, and that the cables/switch etc are operating fine as the SDR limit is easily reached. # qperf -li mlx4_0:2 -ri mlx4_0:1 172.25.5.77 -m 2000 -t 10 rc_bi_bw rc_bi_bw: bw = 1.91 GB/sec # qperf -li mlx4_0:2 -ri mlx4_0:1 172.25.5.77 -m 2000 -t 10 ud_bi_bw ud_bi_bw: send_bw = 1.95 GB/sec recv_bw = 1.95 GB/sec datagram mode mtu 2044 # netperf -H 10.10.0.90 -fM -l 600 -D 1, -- -m 64000 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90 (10.10.0.90) port 0 AF_INET : demo Interim result: 33.00 MBytes/s over 10.31 seconds Interim result: 22.27 MBytes/s over 1.48 seconds Interim result: 34.22 MBytes/s over 2.51 seconds Interim result: 34.05 MBytes/s over 1.01 seconds Interim result: 22.88 MBytes/s over 1.49 seconds Interim result: 30.03 MBytes/s over 1.00 seconds Interim result: 28.26 MBytes/s over 1.01 seconds Interim result: 28.21 MBytes/s over 1.00 seconds Interim result: 14.26 MBytes/s over 1.98 seconds # netperf -H 10.10.0.90 -fM -l 600 -D 1, -- -m 2000 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90 (10.10.0.90) port 0 AF_INET : demo Interim result: 429.54 MBytes/s over 1.00 seconds Interim result: 429.22 MBytes/s over 1.00 seconds Interim result: 429.50 MBytes/s over 1.00 seconds Interim result: 429.47 MBytes/s over 1.00 seconds Interim result: 429.35 MBytes/s over 1.00 seconds Interim result: 429.10 MBytes/s over 1.00 seconds Interim result: 429.16 MBytes/s over 1.00 seconds # netperf -H 10.10.0.90 -fM -l 600 -D 1, -- -m 2000 -D L,R TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90 (10.10.0.90) port 0 AF_INET : nodelay : demo Interim result: 294.15 MBytes/s over 1.32 seconds Interim result: 294.27 MBytes/s over 1.00 seconds Interim result: 294.98 MBytes/s over 1.00 seconds Interim result: 294.74 MBytes/s over 1.00 seconds Interim result: 294.45 MBytes/s over 1.00 seconds Interim result: 294.53 MBytes/s over 1.00 seconds Interim result: 294.35 MBytes/s over 1.00 seconds Interim result: 294.43 MBytes/s over 1.00 seconds connected mode mtu 2044 # netperf -H 10.10.0.90 -fM -l 600 -D 1, -- -m 64000 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90 (10.10.0.90) port 0 AF_INET : demo Interim result: 161.05 MBytes/s over 1.71 seconds Interim result: 231.38 MBytes/s over 1.00 seconds Interim result: 231.16 MBytes/s over 1.00 seconds Interim result: 232.40 MBytes/s over 1.00 seconds Interim result: 322.28 MBytes/s over 1.00 seconds Interim result: 448.25 MBytes/s over 1.00 seconds Interim result: 448.54 MBytes/s over 1.00 seconds Interim result: 448.36 MBytes/s over 1.00 seconds Interim result: 448.52 MBytes/s over 1.00 seconds Interim result: 448.18 MBytes/s over 1.00 seconds Interim result: 448.06 MBytes/s over 1.00 seconds Interim result: 448.26 MBytes/s over 1.00 seconds Interim result: 448.41 MBytes/s over 1.00 seconds # netperf -H 10.10.0.90 -fM -l 600 -D 1, -- -m 2000 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90 (10.10.0.90) port 0 AF_INET : demo Interim result: 454.45 MBytes/s over 1.00 seconds Interim result: 455.10 MBytes/s over 1.00 seconds Interim result: 454.87 MBytes/s over 1.00 seconds Interim result: 454.64 MBytes/s over 1.00 seconds Interim result: 454.30 MBytes/s over 1.00 seconds Interim result: 454.11 MBytes/s over 1.00 seconds Interim result: 454.08 MBytes/s over 1.00 seconds Interim result: 454.07 MBytes/s over 1.00 seconds Interim result: 454.06 MBytes/s over 1.00 seconds Interim result: 454.19 MBytes/s over 1.00 seconds Interim result: 454.42 MBytes/s over 1.00 seconds Interim result: 454.31 MBytes/s over 1.00 seconds # netperf -H 10.10.0.90 -fM -l 600 -D 1, -- -m 2000 -D L,R TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90 (10.10.0.90) port 0 AF_INET : nodelay : demo Interim result: 305.93 MBytes/s over 1.00 seconds Interim result: 306.84 MBytes/s over 1.00 seconds Interim result: 306.88 MBytes/s over 1.00 seconds Interim result: 306.80 MBytes/s over 1.00 seconds Interim result: 306.69 MBytes/s over 1.00 seconds Interim result: 306.41 MBytes/s over 1.00 seconds Interim result: 307.25 MBytes/s over 1.00 seconds Interim result: 307.56 MBytes/s over 1.00 seconds Interim result: 307.51 MBytes/s over 1.00 seconds Interim result: 307.58 MBytes/s over 1.00 seconds connected mode mtu 64000 # netperf -H 10.10.0.90 -fM -l 600 -D 1, -- -m 64000 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90 (10.10.0.90) port 0 AF_INET : demo Interim result: 932.87 MBytes/s over 1.21 seconds Interim result: 931.33 MBytes/s over 1.00 seconds Interim result: 930.20 MBytes/s over 1.00 seconds Interim result: 929.64 MBytes/s over 1.00 seconds Interim result: 931.16 MBytes/s over 1.00 seconds Interim result: 929.96 MBytes/s over 1.00 seconds # netperf -H 10.10.0.90 -fM -l 600 -D 1, -- -m 2000 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90 (10.10.0.90) port 0 AF_INET : demo Interim result: 932.82 MBytes/s over 1.46 seconds Interim result: 930.69 MBytes/s over 1.00 seconds Interim result: 931.36 MBytes/s over 1.00 seconds Interim result: 930.05 MBytes/s over 1.00 seconds Interim result: 931.40 MBytes/s over 1.00 seconds Interim result: 930.07 MBytes/s over 1.00 seconds # netperf -H 10.10.0.90 -fM -l 600 -D 1, -- -m 2000 -D L,R TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90 (10.10.0.90) port 0 AF_INET : nodelay : demo Interim result: 475.01 MBytes/s over 1.58 seconds Interim result: 474.22 MBytes/s over 1.00 seconds Interim result: 474.25 MBytes/s over 1.00 seconds Interim result: 473.14 MBytes/s over 1.00 seconds Interim result: 473.62 MBytes/s over 1.00 seconds Interim result: 474.37 MBytes/s over 1.00 seconds Interim result: 473.81 MBytes/s over 1.00 seconds _______________________________________________ general mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general _______________________________________________ general mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
