On 26-Sep-11 11:27 AM, Yevgeny Kliteynik wrote: > On 22-Sep-11 12:09 AM, Jeff Squyres wrote: >> On Sep 21, 2011, at 4:24 PM, Sébastien Boisvert wrote: >> >>>> What happens if you run 2 ibv_rc_pingpong's on each node? Or N >>>> ibv_rc_pingpongs? >>> >>> With 11 ibv_rc_pingpong's >>> >>> http://pastebin.com/85sPcA47 >>> >>> Code to do that => https://gist.github.com/1233173 >>> >>> Latencies are around 20 microseconds. >> >> This seems to imply that the network is to blame for the higher latency...? > > Interesting... I'm getting the same latency with ibv_rc_pingpong. > I get 8.5 usec for a single ping-pong.
BTW, I've just checked this with performance guys - ibv_rc_pingpong is not used for performance measurement but only as IB network sanity check, therefore it was never meant to give optimal performance. Use ib_write_lat instead. -- YK > Please run 'ibclearcounters' to reset fabric counters, then > ibdiagnet to make sure that the fabric is clean. > If you have 4x QDR cluster, run ibdiagnet as follows: > > ibdiagnet --ls 10 --lw 4x > > Check that you don't have any errors/warnings. > > Then please run your script with ib_write_lat instead of ibv_rc_pingpong. > Just replace the command in the script and the rest would be fine. > > If the fabric is clean, you're supposed to get typical > latency of ~1.4 usec. > > -- YK > > >> I.e., if you run the same pattern with MPI processes and get 20us latency, >> that would tend to imply that the network itself is not performing well with >> that IO pattern. >> >>> My job seems to do well so far with ofud ! >>> >>> [sboisver12@colosse2 ray]$ qstat >>> job-ID prior name user state submit/start at queue >>> slots ja-task-ID >>> ----------------------------------------------------------------------------------------------------------------- >>> 3047460 0.55384 fish-Assem sboisver12 r 09/21/2011 15:02:25 >>> med@r104-n58 256 >> >> I would still be suspicious -- ofud is not well tested, and it can >> definitely hang if there are network drops. >> >
