On Mon, May 08, 2006 at 05:52:17PM -0500, Hassan M. Jafri wrote: > I cant crank out more than 150 MB/sec with my 2.0 GHz xeons. verbs level > benchmarks, however give decent numbers for bandwidth. With netperf, the > server side CPU usage is 99% which is much higher than other posted > bandwidth results on this thread. Any suggestions? > > Here is the complete configuration for my bandwidth tests > > Kernel-2.6.15.4 > netperf-2.3-3
FWIW, 2.4.1 is the latest netperf version. See http://www.netperf.org/svn/netperf2 > OpenIB rev 6552 > MTLP23108-CF128 > Firmware 3.4.0 I was using 3.3.3...looks like I should update. > MSI-X is enabled for the HCA > > ------------------------------ > Here is the netperf output > > > > TCP STREAM TEST to 192.168.2.2 > Recv Send Send Utilization Service > Demand > Socket Socket Message Elapsed Send Recv Send Recv > Size Size Size Time Throughput local remote local > remote > bytes bytes bytes secs. MBytes /s % T % T us/KB us/KB > > 262142 262142 32768 10.01 151.32 59.66 99.84 7.700 > 12.886 I think Michael is probably right - NETFILTER or something else is burning additional CPU cycles. The "Send local" service demand is nearly 2X of what I was seeing a few monthes back. You need to use perfmon or oprofile to figure out where the time is being spent. You could also test SDP (see openib.org for notes on LD_PRELOAD=libsdp.so) to see how that performs - doesn't use NETFILTER. You might also try the netperf "-T" option to bind the netserver process to a different CPU. If two CPUs are available, IPoIB throughput is better with the interrupts handled on a different CPU than the one handling the data. grant _______________________________________________ openib-general mailing list [email protected] http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
