I suspect the OOB is working just fine and you are seeing the TCP/btl opening the other ports. There are two TCP elements at work here: the OOB (which sends management messages between daemons) and the BTL (which handles the MPI traffic). In addition to what you provided, you also need to provide the following params:
btl_tcp_port_range_v4: The number of ports where the TCP BTL will try to bind. This parameter together with the port min, define a range of ports where Open MPI will open sockets btl_tcp_port_min_v4: starting port to use I can’t answer the question about #ports to open - will have to leave that to someone else Ralph > On Jul 7, 2018, at 6:34 AM, Adam Sylvester <op8...@gmail.com> wrote: > > I'm using OpenMPI 2.1.0 on RHEL 7, communicating between ranks via TCP > > I have a new cluster to install my application on with tightly-controlled > firewalls. I can have them open up a range of TCP ports which MPI can > communicate over. I thought I could force MPI to stick to a range of ports > via "--mca oob_tcp_static_ports startPort-endPort" but this doesn't seem to > be working; I still seem MPI opening up TCP ports outside of this range to > communicate. I've also seen "--mca oob_tcp_dynamic_ports" on message boards; > I'm not sure what the difference is between these two but this flag doesn't > seem to do what I want either. > > Is there a way to lock the TCP port range down? As a general rule of thumb, > if I'm communicating between up to 50 instances on a 10 Gbps network moving > at several painful spots in the chain hundreds of GBs of data around, how > large should I make this port range (i.e. if Open MPI would normally open a > bunch of ports on each machine to improve the network transfer speed, I don't > want to slow it down by allowing it too narrow of a port range). Just need a > rough order of magnitude - 10 ports, 100 ports, 1000 ports? > > Thanks! > -Adam > _______________________________________________ > users mailing list > users@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/users _______________________________________________ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users