Torsten Hoefler wrote:
Hi Joachim,
Is there a formula to calculate the number of socket connections per
node (S) that an OpenMPI application needs for running (via sockets) on
N nodes with P processes each? I guess something like
S = P * (N-1)*P
I would say that S=(N-1)*P (why do you assume that S=O(P^2)?) is an
For N nodes running P processes each, each single process sees (N-1)*P
remote processes. I'd say this results in (N-1)*P^2 connections *per node*.
upper bound for the MPI layer (each rank connects to all other ranks
besides the ones on its node (connect via sm) and there is no connection
'aggregation' on a single node yet). However, the connection
establishment is lazy (each connection is established with the first
send - see btl_tcp_endpoint.c:495). So you'll have the fully-connected
case after an MPI_Alltoall with a sufficient size or a similar send/recv
pattern :).
O.k. But what happens if a connection can not be established due to
resource limits? Are existing connections closed? O
You could try to use the hierarch collective component to minimize the
number of active connections (open sockets) per node.
Yes, this also can help to increase throughput for small data size.
regards, Joachim
--
Joachim Worringen, Software Architect, Dolphin Interconnect Solutions
phone ++49/(0)228/324 08 17 - http://www.dolphinics.com
