On Fri, Apr 29, 2005 at 11:38:46AM -0700, Grant Grundler wrote: ... > ionize:~# ibv_pingpong -s 64 -n 100000 10.0.0.51 > local address: LID 0x000d, QPN 0x070406, PSN 0x972e9e > remote address: LID 0x000b, QPN 0x030406, PSN 0x3c7543 > 12800000 bytes in 1.65 seconds = 62.10 Mbit/sec > 100000 iters in 1.65 seconds = 16.49 usec/iter ... > ionize:~# ibv_pingpong -s 65536 -n 10000 > local address: LID 0x000d, QPN 0x0b0406, PSN 0xd79a37 > remote address: LID 0x000b, QPN 0x070406, PSN 0x940ac7 > 1310720000 bytes in 2.84 seconds = 3691.26 Mbit/sec > 10000 iters in 2.84 seconds = 284.07 usec/iter
It seemed the numbers were too low and of course "ionize" had the IB card in a "1/2 bandwidth" slot (aka "single rope"). 3691 Mbit/sec is about 90% utilization (461:512 MB/s) of the rope. Re-running the extremes, I'm getting better numbers: [EMAIL PROTECTED]:~$ ibv_pingpong -s 65536 -n 100000 10.0.0.30 local address: LID 0x000b, QPN 0x180406, PSN 0x7609fa remote address: LID 0x0010, QPN 0x010406, PSN 0xf10582 13107200000 bytes in 17.99 seconds = 5830.14 Mbit/sec 100000 iters in 17.99 seconds = 179.85 usec/iter 5830 Mbit/s still seems low for this box...only 728 MB/s. I'm expecting at least 800 MB/s and about 900 MB/s per link under "reasonable" conditions (ie > 1K size msgs). Is the issue ibv_pingpong is a single threaded test? Should I run a script to start/send several instances of ibv_pingpong on different ports? [EMAIL PROTECTED]:~$ ibv_pingpong -s 64 -n 1000000 10.0.0.30 local address: LID 0x000b, QPN 0x190406, PSN 0x04c55b remote address: LID 0x0010, QPN 0x020406, PSN 0x350ff5 128000000 bytes in 15.96 seconds = 64.17 Mbit/sec 1000000 iters in 15.96 seconds = 15.96 usec/iter Is the 15.96 usec/iter the round trip time? thanks, grant _______________________________________________ openib-general mailing list [email protected] http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
