Dear Open MPIers,
Automatic tuning of socket buffers has been in the linux kernel since
2.4.17/2.6.7. That is some time ago. I remember, at the time, that we removed
the default setsockopt() for SO_SNDBUF and SO_RCVBUF in Scali MPI.
Today, running Open MPI 1.10.2 using the TCP BTL, on a 10Gbit/ ethernet, I get:
# OSU MPI Bi-Directional Bandwidth Test
# Size Bi-Bandwidth (MB/s)
1 1.72
2 4.58
4 9.06
8 18.43
16 35.68
32 68.47
64 135.20
128 259.30
256 450.59
512 703.55
1024 935.58
2048 1020.04
4096 1191.23
8192 1192.13
16384 1155.97
32768 1181.74
and by strace I see that Open MPI sets said buffer sizes:
setsockopt(12, SOL_SOCKET, SO_SNDBUF, [131072], 4) = 0
setsockopt(12, SOL_SOCKET, SO_RCVBUF, [131072], 4) = 0
Now, by adding “—mca btl_tcp_rcvbuf 0 —mca btl_tcp_sndbuf 0” to the above
command line, I get:
# OSU MPI Bi-Directional Bandwidth Test
# Size Bi-Bandwidth (MB/s)
1 1.60
2 4.56
4 9.03
8 11.66
16 35.54
32 68.36
64 133.70
128 247.69
256 466.75
512 885.40
1024 1557.51
2048 2115.40
4096 2226.65
8192 2288.82
16384 2318.11
32768 2334.19
(and strace shows no setsockopt for SO_{RCV,SND}BUF)
Roughly, the performance doubles.
Just a humble suggestion to remove the setting of the socket buffer sizes if
not already done in newer versions.
Thxs, Håkon
_______________________________________________
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
