On Wed, 13 Aug 2008, George Bosilca wrote:
> Daniel,
>
> Open IB is one of the few devices that allow local communications (instead of
> using shared memory). As the latency looks OK, I supposed that small messages
> always use shared memory, while large ones get stripped over sm and openib.
> Can you run a test without openib to confirm this ?
This gives exactly the same scores as with OpenIB enabled. I.e. the openib
btl is doesn't interfere with the sm btl in a negative way, it just seems
that the sm btl is for some reason quite a bit slower than the OpenIB btl.
Daniël
[cvsupport@extern src]$ mpirun -np 8 --mca btl self,sm -hostfile hostfile \
./IMB-MPI1.openmpi -npmin 8 PingPong
#---------------------------------------------------
# Benchmarking PingPong
# #processes = 2
# ( 6 additional processes waiting in MPI_Barrier)
#---------------------------------------------------
#bytes #repetitions t[usec] Mbytes/sec
0 1000 0.85 0.00
1 1000 0.95 1.00
2 1000 0.95 2.00
4 1000 0.95 4.01
8 1000 0.95 8.00
16 1000 1.10 13.82
32 1000 1.13 27.00
64 1000 1.13 54.11
128 1000 1.25 97.62
256 1000 1.56 156.65
512 1000 1.94 251.23
1024 1000 2.94 332.16
2048 1000 4.48 435.81
4096 1000 6.96 561.08
8192 1000 11.18 698.98
16384 1000 18.80 831.01
32768 1000 34.60 903.31
65536 640 60.58 1031.71
131072 320 112.12 1114.91
262144 160 215.67 1159.17
524288 80 423.87 1179.61
1048576 40 858.35 1165.03
2097152 20 1765.20 1133.02
4194304 10 4049.80 987.70
