I would start at a lower level. Clear your error counters then run some fabric over the network, maybe using an IMB or OSU benchmark. Then look to see if any ports are very noisy - that usually indicates a cable needing a reseat or replacement.
Now start at a low level. Run IMB or OSU bandwidth or latency tests between pairs of nodes. Are any nodes particularly slow? Now run tests between groups of nodes which share a leaf switch. Finally, if this is really a problem which is being triggered by an application start by bisecting your network. Run the application on half the nodes, then the other half. My hunch is that you will find faulty cables. I can of course be very wrong and it is something that this application triggers. On Wed, 16 Feb 2022 at 19:28, Shan-ho Tsai via users < users@lists.open-mpi.org> wrote: > > Greetings, > > We are troubleshooting an IB network fabric issue that is causing some of > our MPI applications to failed with errors like this: > > -------------------------------------------------------------------------- > The InfiniBand retry count between two MPI processes has been > exceeded. "Retry count" is defined in the InfiniBand spec 1.2(section > 12.7.38): > > The total number of times that the sender wishes the receiver to > retry timeout, packet sequence, etc. errors before posting a > completion error. > > This error typically means that there is something awry within the > InfiniBand fabric itself. You should note the hosts on which this > error has occurred; it has been observed that rebooting or removing a > particular host from the job can sometimes resolve this issue. > > Two MCA parameters can be used to control Open MPI's behavior with > respect to the retry count: > > * btl_openib_ib_retry_count - The number of times the sender will > attempt to retry (defaulted to 7, the maximum value). > * btl_openib_ib_timeout - The local ACK timeout parameter (defaulted > to 20). The actual timeout value used is calculated as: > > 4.096 microseconds * (2^btl_openib_ib_timeout) > > See the InfiniBand spec 1.2 (section 12.7.34) for more details. > > Below is some information about the host that raised the error and the > peer to which it was connected: > > Local host: a3-6 > Local device: mlx5_0 > Peer host: a3-14 > > You may need to consult with your system administrator to get this > problem fixed. > -------------------------------------------------------------------------- > > > I would like to enable verbose logging for the MPI application to see if > that could help us pinpoint the IB communication issue (or the nodes with > the issue). > > I see many verbose logging options reported by "ompi_info -a | grep > verbose", but I am not sure which one(s) could be helpful here. Would any > of them be useful here or are there any other ways to enable verbose > logging to help with tracking down the issue? > > Thank you so much in advance. > > Best regards, > > ---------------------------------------------------- > Shan-Ho Tsai > University of Georgia, Athens GA > > >
