Quoting r. Grant Grundler <[EMAIL PROTECTED]>: > Subject: Re: Re: user-mode verbs on Itanium > > On Fri, Apr 29, 2005 at 11:38:46AM -0700, Grant Grundler wrote: > ... > > ionize:~# ibv_pingpong -s 64 -n 100000 10.0.0.51 > > local address: LID 0x000d, QPN 0x070406, PSN 0x972e9e > > remote address: LID 0x000b, QPN 0x030406, PSN 0x3c7543 > > 12800000 bytes in 1.65 seconds = 62.10 Mbit/sec > > 100000 iters in 1.65 seconds = 16.49 usec/iter > ... > > ionize:~# ibv_pingpong -s 65536 -n 10000 > > local address: LID 0x000d, QPN 0x0b0406, PSN 0xd79a37 > > remote address: LID 0x000b, QPN 0x070406, PSN 0x940ac7 > > 1310720000 bytes in 2.84 seconds = 3691.26 Mbit/sec > > 10000 iters in 2.84 seconds = 284.07 usec/iter > > It seemed the numbers were too low and of course "ionize" had the IB card > in a "1/2 bandwidth" slot (aka "single rope"). > 3691 Mbit/sec is about 90% utilization (461:512 MB/s) of the rope. > > Re-running the extremes, I'm getting better numbers: > [EMAIL PROTECTED]:~$ ibv_pingpong -s 65536 -n 100000 10.0.0.30 > local address: LID 0x000b, QPN 0x180406, PSN 0x7609fa > remote address: LID 0x0010, QPN 0x010406, PSN 0xf10582 > 13107200000 bytes in 17.99 seconds = 5830.14 Mbit/sec > 100000 iters in 17.99 seconds = 179.85 usec/iter > > 5830 Mbit/s still seems low for this box...only 728 MB/s. > I'm expecting at least 800 MB/s and about 900 MB/s > per link under "reasonable" conditions (ie > 1K size msgs). > Is the issue ibv_pingpong is a single threaded test? > Should I run a script to start/send several instances > of ibv_pingpong on different ports? > > > [EMAIL PROTECTED]:~$ ibv_pingpong -s 64 -n 1000000 10.0.0.30 > local address: LID 0x000b, QPN 0x190406, PSN 0x04c55b > remote address: LID 0x0010, QPN 0x020406, PSN 0x350ff5 > 128000000 bytes in 15.96 seconds = 64.17 Mbit/sec > 1000000 iters in 15.96 seconds = 15.96 usec/iter > > Is the 15.96 usec/iter the round trip time? > > thanks, > grant
I think ibv_pingpong is not the best latency benchmark - more of a test/usage example. Since it has no pipelining, its also not the best bandwidth benchmark. Grant, if you want a latency benchmark you may want to look at contrib/mellanox/perftest. Its a variation on the pingpong test, showing how you can get latency between 3 and 4usec one-way, by using inline data on sends, and polling on data. I expect to put up a bandwidth test one of these days. -- MST - Michael S. Tsirkin _______________________________________________ openib-general mailing list [email protected] http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
