Hello,Testing performance of OpenMPI over Infiniband I have the following result :
1) Hardware is : SilversStorm interface
2) Openmpi version is : (from ompi_info)
Open MPI: 1.0.2a9r9159
Open MPI SVN revision: r9159
Open RTE: 1.0.2a9r9159
Open RTE SVN revision: r9159
OPAL: 1.0.2a9r9159
OPAL SVN revision: r9159
3) Cluster with Bi-processors Opteron 248 2.2 GHz
Configure has been run with option --with-mvapi=path-to-mvapi
4) a C coded pinpong gives the following values :
LOOPS: 1000 BYTES: 4096 SECONDS: 0.085557 MBytes/sec: 95.749051
LOOPS: 1000 BYTES: 8192 SECONDS: 0.050657 MBytes/sec: 323.429912
LOOPS: 1000 BYTES: 16384 SECONDS: 0.084038 MBytes/sec: 389.918757
LOOPS: 1000 BYTES: 32768 SECONDS: 0.163161 MBytes/sec: 401.665104
LOOPS: 1000 BYTES: 65536 SECONDS: 0.306694 MBytes/sec: 427.370561
LOOPS: 1000 BYTES: 131072 SECONDS: 0.529589 MBytes/sec: 494.995011
LOOPS: 1000 BYTES: 262144 SECONDS: 0.952616 MBytes/sec: 550.366583
LOOPS: 1000 BYTES: 524288 SECONDS: 1.927987 MBytes/sec: 543.870859
LOOPS: 1000 BYTES: 1048576 SECONDS: 3.673732 MBytes/sec: 570.850562
LOOPS: 1000 BYTES: 2097152 SECONDS: 9.993185 MBytes/sec: 419.716435
LOOPS: 1000 BYTES: 4194304 SECONDS: 18.211958 MBytes/sec: 460.609893
LOOPS: 1000 BYTES: 8388608 SECONDS: 35.421490 MBytes/sec: 473.645124
My questions are :
a) Is OpenMPI doing in this case TCP/IP over IB ? (I guess so)
b) Is it possible to improve significantly these values by changing the
defaults ?
I have used several mca btl parameters but without improving the maximum bandwith.
For example : --mca btl mvapi --mca btl_mvapi_max_send_size 8388608 c) Is it possible that other IB hardware implementations have better performances with OpenMPI ?d) Is it possible to use specific IB drivers for optimal performance ? (should reach almost 800 MB/sec)
Thank you very much for your help Best Regards, Jean Latour
<<attachment: latour.vcf>>
