On Wed, 2008-02-27 at 17:39 -0500, Wittawat Tantisiriroj wrote: > Hi, > We have set up a small InfiniBand cluster to do several network > storage experiments over IPoIB. However, we had the problem getting a > good throughput with IPoIB-connected mode. So, we followed the same > benchmark that Michael S. Tsirkin did in > "http://lists.openfabrics.org/pipermail/general/2006-November/029500.html";. > We realize that even with a simple scenario we still get only ~620MB/s > throughput from IPoIB-CM. I searched around with Google, but I cannot > find any information regarding IPoIB-CM throughput. > > My question is: > > - Is this throughput typical/normal in most system?
You can't say there is a typical result for this check -- it depends on the "strength" of your system. One thing you can do is watch how much of the CPU is used - you can use htop for that (it gives you per CPU utilization). If you use 100% CPU than stronger machines will give higher results. On my systems (AMD @2.4 Ghz) / mt25204 I get: [EMAIL PROTECTED] ~]# netperf -H 11.4.3.185 -f M TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 11.4.3.185 (11.4.3.185) port 0 AF_INET Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. MBytes/sec 87380 16384 16384 10.01 755.20 On other systems with Arbel (mt25218) I get 980 MB/s etc. > > - Is there any necessary tweak tuning with TCP, ib_ipoib or kernel > parameters in order to get ~900 MB/s? I have tried to TCP buffer size, > but it still does not improve the throughput. > > - Should I use a OFED distribution instead of a standard built-in with a > standard kernel? (I hope it does not matter) > > System > ===== > Processor: Intel(R) Pentium(R) D CPU 3.00GHz > Memory: 4GB > OS: Debian Etch with 2.6.24.2 kernel > > Network > ====== > Network card: Mellanox MT25204 (InfiniHost III Lx HCA) (4x, 10Gbps) > Switch: Mellanox Gazelle (MTS9600) 96 ports with 4X (10 Gb/s) each > Network Software Stack: Standard IPoIB built-in with the 2.6.24.2 kernel > IPoIB Configuration: Connected mode with MTU=65520 > > Benchmark > ======== > Server: ib265 > # ifconfig ib0 mtu 65520 > # netserver > > Client: ib266 > # ifconfig ib0 mtu 65520 > # netperf -H ib265 -f M > > TCP STREAM TEST to ib265 > Recv Send Send > Socket Socket Message Elapsed > Size Size Size Time Throughput > bytes bytes bytes secs. MBytes/sec > 87380 16384 16384 10.01 620.27 > > Thank in advance, > Wittawat > _______________________________________________ > general mailing list > [email protected] > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general _______________________________________________ general mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
