Shirley, Hassan - I am *very* interested in these results, and I want to at least mention that I'm doing similar NFS/RDMA testing, and getting some contrasting results.
> 699040 699040 16384 60.00 3668.07 (458MB/s) > cpu utilization was around 95%. On my dual-2.4GHz Xeon, with the relatively untuned NFS/RDMA client on 2.6.16.6, I am able to pull about 450MB/sec of read throughput at 35% total CPU. This is using 16 threads of NFS direct i/o (O_DIRECT) to a midrange NetApp server, I did achieve a similar result with the Linux NFS/RDMA server (but only after hotwiring the ext2 interface because I don't have the spindles). I am using a dedicated filesystem test to generate the load, and also iozone. These NFS/RDMA direct reads use RDMA writes from the server to the client. Also, this was with client hyperthreading disabled and a dual-processor Dell, I could reboot with a single CPU to get more comparable results. But, the throughput was limited by server CPU (100%), the client was actually loafing a little bit. I thought it was interesting that a filesystem achieves the same throughput at better overhead than a dedicated network test. :-) And I haven't played with interrupt affinity at all. Tom. At 07:23 PM 5/8/2006, Shirley Ma wrote: >I am testing most of my patches. Under > >1.Intel(R) Xeon(TM) CPU 2.80GHz, one cpu, >2. fw-23108-3_4_000-MHXL-CF128-T.bin >3. pci-x without msi_x enabled >4. kernel 2.6.16 >5. netperf-2.4.0 >6. SVN 68XX+several IPoIB patches > >The best result I got so far: > >Testing with the following command line: >netperf -l 60 -H 10.1.1.100 -t TCP_STREAM -i 10,2 -I 95,5 -- -m 16384 -s >349520 -S 349520 > >TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.1.1.100 >(10.1.1.100) port 0 AF_INET : +/-2.5% @ 95% conf. >Recv Send Send >Socket Socket Message Elapsed >Size Size Size Time Throughput >bytes bytes bytes secs. 10^6bits/sec > >699040 699040 16384 60.00 3668.07 (458MB/s) > >cpu utilization was around 95%. > >Thanks >Shirley Ma >IBM Linux Technology Center >15300 SW Koll Parkway >Beaverton, OR 97006-6063 >Phone(Fax): (503) 578-7638 > > > > >"Hassan M. Jafri" <[EMAIL PROTECTED]> >Sent by: [EMAIL PROTECTED] > >05/08/2006 03:52 PM >To >[email protected] >cc >Subject >Re: [openib-general] ip over ib throughtput > > > > >I cant crank out more than 150 MB/sec with my 2.0 GHz xeons. verbs level >benchmarks, however give decent numbers for bandwidth. With netperf, the >server side CPU usage is 99% which is much higher than other posted >bandwidth results on this thread. Any suggestions? > >Here is the complete configuration for my bandwidth tests > >Kernel-2.6.15.4 >netperf-2.3-3 >OpenIB rev 6552 >MTLP23108-CF128 >Firmware 3.4.0 >MSI-X is enabled for the HCA > > >------------------------------ >Here is the netperf output > > > >TCP STREAM TEST to 192.168.2.2 >Recv Send Send Utilization Service >Demand >Socket Socket Message Elapsed Send Recv Send Recv >Size Size Size Time Throughput local remote local >remote >bytes bytes bytes secs. MBytes /s % T % T us/KB us/KB > >262142 262142 32768 10.01 151.32 59.66 99.84 7.700 >12.886 >------------------------------- > >Here is ib0 config for one of the nodes > >ib0 Link encap:UNSPEC HWaddr >00-02-04-04-FE-80-00-00-00-00-00-00-00-00-00-00 > inet addr:192.168.2.1 Bcast:192.168.2.255 Mask:255.255.255.0 > inet6 addr: fe80::202:c902:0:3ce9/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:2044 Metric:1 > RX packets:1724527 errors:0 dropped:0 overruns:0 frame:0 > TX packets:9685456 errors:0 dropped:2 overruns:0 carrier:0 > collisions:0 txqueuelen:128 > RX bytes:89830114 (85.6 MiB) TX bytes:2213308646 (2.0 GiB) > > > > > > > > >Michael S. Tsirkin wrote: >> Hi! >> What kind of performance do people see with ip over ib on gen2? >> I see about 100Mbyte/sec at 99% CPU utilisation on send, >> on an express card, Xeon 2.8GHz, SSE doorbells enabled. >> >> MST >> _______________________________________________ >> openib-general mailing list >> [email protected] >> http://openib.org/mailman/listinfo/openib-general >> >> To unsubscribe, please visit >> http://openib.org/mailman/listinfo/openib-general >> >_______________________________________________ >openib-general mailing list >[email protected] >http://openib.org/mailman/listinfo/openib-general > >To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > >_______________________________________________ >openib-general mailing list >[email protected] >http://openib.org/mailman/listinfo/openib-general > >To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general _______________________________________________ openib-general mailing list [email protected] http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
