Jim, Like I've said before, I don't see any change in throughput with SDP zcopy, plus the throughput bounces around. When you run netperf with -D, do you see variations in throughput?
Here's example from a dual socket Xeon 5355 quad core RHEL5 x86_64 system with ConnectX 2.3.0 firmware, the interim results throughput bounces around between 4-7 Gbps. [EMAIL PROTECTED] ~]$ cat /sys/module/ib_sdp/parameters/sdp_zcopy_thresh 16384 [EMAIL PROTECTED] ~]$ LD_PRELOAD=libsdp.so netperf241 -C -c -P 0 -t TCP_STREAM -H 192.168.1.127 -D -l 60 -- -m 1000000 Interim result: 6676.50 10^6bits/s over 1.00 seconds Interim result: 6674.47 10^6bits/s over 1.00 seconds Interim result: 6687.89 10^6bits/s over 1.00 seconds Interim result: 7075.40 10^6bits/s over 1.00 seconds Interim result: 7065.08 10^6bits/s over 1.00 seconds Interim result: 7074.69 10^6bits/s over 1.00 seconds Interim result: 6667.10 10^6bits/s over 1.06 seconds Interim result: 4492.29 10^6bits/s over 1.48 seconds Interim result: 4503.65 10^6bits/s over 1.00 seconds Interim result: 4481.25 10^6bits/s over 1.01 seconds Interim result: 4495.91 10^6bits/s over 1.00 seconds Interim result: 4521.51 10^6bits/s over 1.00 seconds Interim result: 4466.58 10^6bits/s over 1.01 seconds Interim result: 4482.09 10^6bits/s over 1.00 seconds Interim result: 4480.21 10^6bits/s over 1.00 seconds Interim result: 4490.07 10^6bits/s over 1.00 seconds Interim result: 4479.47 10^6bits/s over 1.00 seconds Interim result: 4480.30 10^6bits/s over 1.00 seconds Interim result: 4489.14 10^6bits/s over 1.00 seconds Interim result: 4484.38 10^6bits/s over 1.00 seconds Interim result: 4473.64 10^6bits/s over 1.00 seconds Interim result: 4479.71 10^6bits/s over 1.00 seconds Interim result: 4486.54 10^6bits/s over 1.00 seconds Interim result: 4456.65 10^6bits/s over 1.01 seconds Interim result: 4483.70 10^6bits/s over 1.00 seconds Interim result: 4486.41 10^6bits/s over 1.00 seconds Interim result: 4489.58 10^6bits/s over 1.00 seconds Interim result: 4478.15 10^6bits/s over 1.00 seconds Interim result: 4476.67 10^6bits/s over 1.00 seconds Interim result: 4496.49 10^6bits/s over 1.00 seconds Interim result: 4489.26 10^6bits/s over 1.00 seconds Interim result: 4479.86 10^6bits/s over 1.00 seconds Interim result: 4500.97 10^6bits/s over 1.00 seconds Interim result: 4473.96 10^6bits/s over 1.00 seconds Interim result: 7346.56 10^6bits/s over 1.00 seconds Interim result: 7524.94 10^6bits/s over 1.00 seconds Interim result: 7540.16 10^6bits/s over 1.00 seconds Interim result: 7553.53 10^6bits/s over 1.00 seconds Interim result: 7552.08 10^6bits/s over 1.00 seconds Interim result: 7550.08 10^6bits/s over 1.00 seconds Interim result: 7554.35 10^6bits/s over 1.00 seconds Interim result: 7550.85 10^6bits/s over 1.00 seconds Interim result: 7557.27 10^6bits/s over 1.00 seconds Interim result: 7568.28 10^6bits/s over 1.00 seconds Interim result: 7497.24 10^6bits/s over 1.01 seconds Interim result: 7436.44 10^6bits/s over 1.01 seconds Interim result: 6098.26 10^6bits/s over 1.22 seconds Interim result: 5644.82 10^6bits/s over 1.08 seconds Interim result: 5639.07 10^6bits/s over 1.00 seconds Interim result: 5636.32 10^6bits/s over 1.00 seconds Interim result: 5640.45 10^6bits/s over 1.00 seconds Interim result: 6319.06 10^6bits/s over 1.00 seconds Interim result: 7324.10 10^6bits/s over 1.00 seconds Interim result: 7323.53 10^6bits/s over 1.00 seconds Interim result: 7333.88 10^6bits/s over 1.00 seconds Interim result: 7172.70 10^6bits/s over 1.02 seconds Interim result: 4488.97 10^6bits/s over 1.60 seconds Interim result: 4492.37 10^6bits/s over 1.00 seconds 87380 16384 1000000 60.00 5701.15 17.41 16.26 2.001 1.870 Scott Weitzenkamp SQA and Release Manager Server Virtualization Business Unit Cisco Systems > -----Original Message----- > From: Jim Mott [mailto:[EMAIL PROTECTED] > Sent: Thursday, January 24, 2008 9:47 AM > To: Scott Weitzenkamp (sweitzen); Weikuan Yu > Cc: [email protected] > Subject: RE: [ofa-general] RE: [ewg] Not seeing any SDP > performance changes inOFED 1.3 beta, and I get Oops when > enabling sdp_zcopy_thresh > > I am really puzzled. The majority of my testing has been between > Rhat4U4 and Rhat5. Using netperf command lines of the form: > netperf -C -c -P 0 -t TCP_RR -H 193.168.10.143 -l 60 ---r 64 > netperf -C -c -P 0 -t TCP_STREAM -H 193.168.10.143 -l 60 > ---r 1000000 > and a process of: > - set sdp_zcopy_thresh=0, run bandwidth test > - set sdp_zcopy_thresh=size, run bandwidth test > I repeatedly get results that look like this: > size SDP Bzcopy > 65536 7375.00 7515.98 > 131072 7465.70 8105.58 > 1000000 6541.87 9948.76 > > These numbers are from high end (2-socket, quad-core) machines. When > you > use smaller machines, like the AMD dual-core shown below, the > differences > between SDP with and without bzcopy are more striking. > > The process to start the netserver is: > export LD_LIBRARY_PATH=/usr/local/ofed/lib64:/usr/local/ofed/lib > export LD_PRELOAD=libsdp.so > export LIBSDP_CONFIG_FILE=/etc/infiniband/libsdp.conf > netserver > > The process to start the netperf is similar: > export LD_LIBRARY_PATH=/usr/local/ofed/lib64:/usr/local/ofed/lib > export LD_PRELOAD=libsdp.so > export LIBSDP_CONFIG_FILE=/etc/infiniband/libsdp.conf > netperf -C -c -P 0 -t TCP_STREAM -H 193.168.10.143 -l 60 > ---r 1000000 > > You and unload and reload ib_sdp between tests, but I just echo 0 and > echo size into sdp_zcopy_thresh on the sending side. Note that it is > in a different place on Rhat4u4 and Rhat5. > > My libsdp.conf is the default that ships with OFED. Stripping the > comments (grep -v), it is just: > log min-level 9 destination file libsdp.log > use both server * *:* > use both client * *:* > Note that if you build locally: > cd /tmp/openib_gen2/xxxx/ofa_1_3_dev_kernel > make install > the libsdp.conf file seems to get lost. You must restore it by > hand. > > I have a shell script that automates this testing for a > wide range of message sizes: > 64 128 512 1024 2048 4096 8192 16000 32768 65536 131072 1000000 > on multiple transports: > IP both "echo datagram > /sys/class/net/ib0/mode" > IP-CM both "echo connected > /sys/class/net/ib0/mode" > SDP both > Bzcopy TCP_STREAM > Where both is TCP_RR and TCP_STREAM testing. > > The variance in SDP bandwidth results can be 10%-15% between > runs. The > difference between Bzcopy and non-Bzcopy is always very > visible for 128K > and up tests though. > > Could some other people please try to run some of these > tests? If only > help me know if I am crazy? > > Thanks, > JIm > > Jim Mott > Mellanox Technologies Ltd. > mail: [EMAIL PROTECTED] > Phone: 512-294-5481 > > > -----Original Message----- > From: Scott Weitzenkamp (sweitzen) [mailto:[EMAIL PROTECTED] > Sent: Thursday, January 24, 2008 11:17 AM > To: Jim Mott; Weikuan Yu > Cc: [EMAIL PROTECTED]; [email protected] > Subject: RE: [ofa-general] RE: [ewg] Not seeing any SDP performance > changes inOFED 1.3 beta, and I get Oops when enabling sdp_zcopy_thresh > > I've tested on RHEL4 and RHEL5, and see no sdp_zcopy_thresh > improvement > for any message size, as measured with netperf, for any Arbel or > ConnectX HCA. > > Scott > > > > -----Original Message----- > > From: Jim Mott [mailto:[EMAIL PROTECTED] > > Sent: Thursday, January 24, 2008 7:57 AM > > To: Weikuan Yu; Scott Weitzenkamp (sweitzen) > > Cc: [EMAIL PROTECTED]; [email protected] > > Subject: RE: [ofa-general] RE: [ewg] Not seeing any SDP > > performance changes inOFED 1.3 beta, and I get Oops when > > enabling sdp_zcopy_thresh > > > > Hi, > > 64K is borderline for seeing bzcopy effect. Using an AMD > > 6000+ (3 Ghz > > dual core) in Asus M2A-VM motherboard with ConnectX running > > 2.3 firmware > > and OFED 1.3-rc3 stack running on 2.6.23.8 kernel.org kernel, > > I ran the > > test for 128K: > > 5546 sdp_zcopy_thresh=0 (off) > > 8709 sdp_zcopy_thresh=65536 > > > > For these tests, I just have LD_PRELOAD set in my environment. > > > > ======================= > > > > I see that TCP_MAXSEG is not being handled by libsdp and will > > look into > > it. > > > > > > [EMAIL PROTECTED] ~]# modprobe ib_sdp > > [EMAIL PROTECTED] ~]# netperf -v2 -4 -H 193.168.10.198 -l 30 -t > TCP_STREAM -c > > -C -- -m 128K > > TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to > > 193.168.10.198 > > (193.168.10.198) port 0 AF_INET > > netperf: get_tcp_info: getsockopt TCP_MAXSEG: errno 92 > > Recv Send Send Utilization > Service > > Demand > > Socket Socket Message Elapsed Send Recv Send > > Recv > > Size Size Size Time Throughput local remote local > > remote > > bytes bytes bytes secs. 10^6bits/s % S % S us/KB > > us/KB > > > > 87380 16384 131072 30.01 5545.69 51.47 14.43 1.521 > > 1.706 > > > > Alignment Offset Bytes Bytes Sends Bytes > > Recvs > > Local Remote Local Remote Xfered Per Per > > Send Recv Send Recv Send (avg) > Recv (avg) > > 8 8 0 0 2.08e+10 131072.00 158690 > 33135.60 > > 627718 > > > > Maximum > > Segment > > Size (bytes) > > -1 > > [EMAIL PROTECTED] ~]# echo 65536 > > >/sys/module/ib_sdp/parameters/sdp_zcopy_thresh > > [EMAIL PROTECTED] ~]# netperf -v2 -4 -H 193.168.10.198 -l 30 -t > TCP_STREAM -c > > -C -- -m 128K > > TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to > > 193.168.10.198 > > (193.168.10.198) port 0 AF_INET > > netperf: get_tcp_info: getsockopt TCP_MAXSEG: errno 92 > > Recv Send Send Utilization > Service > > Demand > > Socket Socket Message Elapsed Send Recv Send > > Recv > > Size Size Size Time Throughput local remote local > > remote > > bytes bytes bytes secs. 10^6bits/s % S % S us/KB > > us/KB > > > > 87380 16384 131072 30.01 8708.58 50.63 14.55 0.953 > > 1.095 > > > > Alignment Offset Bytes Bytes Sends Bytes > > Recvs > > Local Remote Local Remote Xfered Per Per > > Send Recv Send Recv Send (avg) > Recv (avg) > > 8 8 0 0 3.267e+10 131072.00 249228 > 26348.30 > > 1239807 > > > > Maximum > > Segment > > Size (bytes) > > -1 > > > > Thanks, > > JIm > > > > Jim Mott > > Mellanox Technologies Ltd. > > mail: [EMAIL PROTECTED] > > Phone: 512-294-5481 > > > > > > -----Original Message----- > > From: Weikuan Yu [mailto:[EMAIL PROTECTED] > > Sent: Thursday, January 24, 2008 9:09 AM > > To: Scott Weitzenkamp (sweitzen) > > Cc: Jim Mott; [EMAIL PROTECTED]; > [email protected] > > Subject: Re: [ofa-general] RE: [ewg] Not seeing any SDP performance > > changes inOFED 1.3 beta, and I get Oops when enabling > sdp_zcopy_thresh > > > > Hi, Scott, > > > > I have been running SDP tests across two woodcrest nodes > with 4x DDR > > cards using OFED-1.2.5.4. The card/firmware info is below. > > > > CA 'mthca0' > > CA type: MT25208 > > Number of ports: 2 > > Firmware version: 5.1.400 > > Hardware version: a0 > > Node GUID: 0x0002c90200228e0c > > System image GUID: 0x0002c90200228e0f > > > > I could not get a bandwidth more than 5Gbps like you have > shown here. > > Wonder if I need to upgrade to the latest software or firmware? Any > > suggestions? > > > > Thanks, > > --Weikuan > > > > > > TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to > > 192.168.225.77 > > (192.168 > > .225.77) port 0 AF_INET > > Recv Send Send Utilization > > Service > > Demand > > Socket Socket Message Elapsed Send Recv Send > > Recv > > Size Size Size Time Throughput local > remote local > > remote > > bytes bytes bytes secs. 10^6bits/s % S % S us/KB > > us/KB > > > > 131072 131072 131072 10.00 4918.95 21.29 24.99 > 1.418 > > 1.665 > > > > > > Scott Weitzenkamp (sweitzen) wrote: > > > Jim, > > > > > > I am trying OFED-1.3-20071231-0600 and RHEL4 x86_64 on a dual CPU > > > (single core each CPU) Xeon system. I do not see any performance > > > improvement (either throughput or CPU utilization) using > > netperf when > > I > > > set /sys/module/ib_sdp/sdp_zcopy_thresh to 16384. Can > you elaborate > > on > > > your HCA type, and performance improvement you see? > > > > > > Here's an example netperf command line when using a > Cheetah DDR HCA > > and > > > 1.2.917 firmware (I have also tried ConnectX and 2.3.000 firmware > > too): > > > > > > [EMAIL PROTECTED] ~]$ LD_PRELOAD=libsdp.so netperf241 > -v2 -4 -H > > > 192.168.1.201 -l 30 -t TCP_STREAM -c -C -- -m 65536 > > > TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to > > 192.168.1.201 > > > (192.168.1.201) port 0 AF_INET : histogram : demo > > > > > > Recv Send Send Utilization > > Service > > > Demand > > > Socket Socket Message Elapsed Send > Recv Send > > > Recv > > > Size Size Size Time Throughput local > remote local > > > remote > > > bytes bytes bytes secs. 10^6bits/s % S % S > us/KB > > > us/KB > > > > > > 87380 16384 65536 30.01 7267.70 55.06 > 61.27 1.241 > > > 1.381 > > > > > > Alignment Offset Bytes Bytes Sends Bytes > > > Recvs > > > Local Remote Local Remote Xfered Per Per > > > Send Recv Send Recv Send (avg) > > Recv (avg) > > > 8 8 0 0 2.726e+10 65536.00 415942 > > 48106.01 > > > 566648 > > > > > > _______________________________________________ general mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
