Hi Joachim, > >> Is there a formula to calculate the number of socket connections per > >> node (S) that an OpenMPI application needs for running (via sockets) on > >> N nodes with P processes each? I guess something like > >> > >> S = P * (N-1)*P > > I would say that S=(N-1)*P (why do you assume that S=O(P^2)?) is an > > For N nodes running P processes each, each single process sees (N-1)*P > remote processes. I'd say this results in (N-1)*P^2 connections *per node*. you're right, I didn't read the *per node* :).
> > upper bound for the MPI layer (each rank connects to all other ranks > > besides the ones on its node (connect via sm) and there is no connection > > 'aggregation' on a single node yet). However, the connection > > establishment is lazy (each connection is established with the first > > send - see btl_tcp_endpoint.c:495). So you'll have the fully-connected > > case after an MPI_Alltoall with a sufficient size or a similar send/recv > > pattern :). > > O.k. But what happens if a connection can not be established due to > resource limits? Are existing connections closed? O No, connections are never closed afaik (there is also the problem to determine which connection should be closed :). If the connection can not be established (e.g. socket() or connect() returns values < 0), the peer is marked as not reachable and the send fails (ompi exits with an error). > > You could try to use the hierarch collective component to minimize the > > number of active connections (open sockets) per node. > > Yes, this also can help to increase throughput for small data size. sure, and it could avoid potential flow-control problems for large data. Best Regards, Torsten -- "It is easier to write an incorrect program than understand a correct one." Alan J. Perlis.
