I've tested on RHEL4 and RHEL5, and see no sdp_zcopy_thresh improvement for any message size, as measured with netperf, for any Arbel or ConnectX HCA.
Scott > -----Original Message----- > From: Jim Mott [mailto:[EMAIL PROTECTED] > Sent: Thursday, January 24, 2008 7:57 AM > To: Weikuan Yu; Scott Weitzenkamp (sweitzen) > Cc: ewg@lists.openfabrics.org; [EMAIL PROTECTED] > Subject: RE: [ofa-general] RE: [ewg] Not seeing any SDP > performance changes inOFED 1.3 beta, and I get Oops when > enabling sdp_zcopy_thresh > > Hi, > 64K is borderline for seeing bzcopy effect. Using an AMD > 6000+ (3 Ghz > dual core) in Asus M2A-VM motherboard with ConnectX running > 2.3 firmware > and OFED 1.3-rc3 stack running on 2.6.23.8 kernel.org kernel, > I ran the > test for 128K: > 5546 sdp_zcopy_thresh=0 (off) > 8709 sdp_zcopy_thresh=65536 > > For these tests, I just have LD_PRELOAD set in my environment. > > ======================= > > I see that TCP_MAXSEG is not being handled by libsdp and will > look into > it. > > > [EMAIL PROTECTED] ~]# modprobe ib_sdp > [EMAIL PROTECTED] ~]# netperf -v2 -4 -H 193.168.10.198 -l 30 -t TCP_STREAM -c > -C -- -m 128K > TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to > 193.168.10.198 > (193.168.10.198) port 0 AF_INET > netperf: get_tcp_info: getsockopt TCP_MAXSEG: errno 92 > Recv Send Send Utilization Service > Demand > Socket Socket Message Elapsed Send Recv Send > Recv > Size Size Size Time Throughput local remote local > remote > bytes bytes bytes secs. 10^6bits/s % S % S us/KB > us/KB > > 87380 16384 131072 30.01 5545.69 51.47 14.43 1.521 > 1.706 > > Alignment Offset Bytes Bytes Sends Bytes > Recvs > Local Remote Local Remote Xfered Per Per > Send Recv Send Recv Send (avg) Recv (avg) > 8 8 0 0 2.08e+10 131072.00 158690 33135.60 > 627718 > > Maximum > Segment > Size (bytes) > -1 > [EMAIL PROTECTED] ~]# echo 65536 > >/sys/module/ib_sdp/parameters/sdp_zcopy_thresh > [EMAIL PROTECTED] ~]# netperf -v2 -4 -H 193.168.10.198 -l 30 -t TCP_STREAM -c > -C -- -m 128K > TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to > 193.168.10.198 > (193.168.10.198) port 0 AF_INET > netperf: get_tcp_info: getsockopt TCP_MAXSEG: errno 92 > Recv Send Send Utilization Service > Demand > Socket Socket Message Elapsed Send Recv Send > Recv > Size Size Size Time Throughput local remote local > remote > bytes bytes bytes secs. 10^6bits/s % S % S us/KB > us/KB > > 87380 16384 131072 30.01 8708.58 50.63 14.55 0.953 > 1.095 > > Alignment Offset Bytes Bytes Sends Bytes > Recvs > Local Remote Local Remote Xfered Per Per > Send Recv Send Recv Send (avg) Recv (avg) > 8 8 0 0 3.267e+10 131072.00 249228 26348.30 > 1239807 > > Maximum > Segment > Size (bytes) > -1 > > Thanks, > JIm > > Jim Mott > Mellanox Technologies Ltd. > mail: [EMAIL PROTECTED] > Phone: 512-294-5481 > > > -----Original Message----- > From: Weikuan Yu [mailto:[EMAIL PROTECTED] > Sent: Thursday, January 24, 2008 9:09 AM > To: Scott Weitzenkamp (sweitzen) > Cc: Jim Mott; ewg@lists.openfabrics.org; [EMAIL PROTECTED] > Subject: Re: [ofa-general] RE: [ewg] Not seeing any SDP performance > changes inOFED 1.3 beta, and I get Oops when enabling sdp_zcopy_thresh > > Hi, Scott, > > I have been running SDP tests across two woodcrest nodes with 4x DDR > cards using OFED-1.2.5.4. The card/firmware info is below. > > CA 'mthca0' > CA type: MT25208 > Number of ports: 2 > Firmware version: 5.1.400 > Hardware version: a0 > Node GUID: 0x0002c90200228e0c > System image GUID: 0x0002c90200228e0f > > I could not get a bandwidth more than 5Gbps like you have shown here. > Wonder if I need to upgrade to the latest software or firmware? Any > suggestions? > > Thanks, > --Weikuan > > > TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to > 192.168.225.77 > (192.168 > .225.77) port 0 AF_INET > Recv Send Send Utilization > Service > Demand > Socket Socket Message Elapsed Send Recv Send > Recv > Size Size Size Time Throughput local remote local > remote > bytes bytes bytes secs. 10^6bits/s % S % S us/KB > us/KB > > 131072 131072 131072 10.00 4918.95 21.29 24.99 1.418 > 1.665 > > > Scott Weitzenkamp (sweitzen) wrote: > > Jim, > > > > I am trying OFED-1.3-20071231-0600 and RHEL4 x86_64 on a dual CPU > > (single core each CPU) Xeon system. I do not see any performance > > improvement (either throughput or CPU utilization) using > netperf when > I > > set /sys/module/ib_sdp/sdp_zcopy_thresh to 16384. Can you elaborate > on > > your HCA type, and performance improvement you see? > > > > Here's an example netperf command line when using a Cheetah DDR HCA > and > > 1.2.917 firmware (I have also tried ConnectX and 2.3.000 firmware > too): > > > > [EMAIL PROTECTED] ~]$ LD_PRELOAD=libsdp.so netperf241 -v2 -4 -H > > 192.168.1.201 -l 30 -t TCP_STREAM -c -C -- -m 65536 > > TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to > 192.168.1.201 > > (192.168.1.201) port 0 AF_INET : histogram : demo > > > > Recv Send Send Utilization > Service > > Demand > > Socket Socket Message Elapsed Send Recv Send > > Recv > > Size Size Size Time Throughput local remote local > > remote > > bytes bytes bytes secs. 10^6bits/s % S % S us/KB > > us/KB > > > > 87380 16384 65536 30.01 7267.70 55.06 61.27 1.241 > > 1.381 > > > > Alignment Offset Bytes Bytes Sends Bytes > > Recvs > > Local Remote Local Remote Xfered Per Per > > Send Recv Send Recv Send (avg) > Recv (avg) > > 8 8 0 0 2.726e+10 65536.00 415942 > 48106.01 > > 566648 > > > _______________________________________________ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg