Greetings,
We are troubleshooting an IB network fabric issue that is causing some of our
MPI applications to failed with errors like this:
--------------------------------------------------------------------------
The InfiniBand retry count between two MPI processes has been
exceeded. "Retry count" is defined in the InfiniBand spec 1.2
(section 12.7.38):
The total number of times that the sender wishes the receiver to
retry timeout, packet sequence, etc. errors before posting a
completion error.
This error typically means that there is something awry within the
InfiniBand fabric itself. You should note the hosts on which this
error has occurred; it has been observed that rebooting or removing a
particular host from the job can sometimes resolve this issue.
Two MCA parameters can be used to control Open MPI's behavior with
respect to the retry count:
* btl_openib_ib_retry_count - The number of times the sender will
attempt to retry (defaulted to 7, the maximum value).
* btl_openib_ib_timeout - The local ACK timeout parameter (defaulted
to 20). The actual timeout value used is calculated as:
4.096 microseconds * (2^btl_openib_ib_timeout)
See the InfiniBand spec 1.2 (section 12.7.34) for more details.
Below is some information about the host that raised the error and the
peer to which it was connected:
Local host: a3-6
Local device: mlx5_0
Peer host: a3-14
You may need to consult with your system administrator to get this
problem fixed.
--------------------------------------------------------------------------
I would like to enable verbose logging for the MPI application to see if that
could help us pinpoint the IB communication issue (or the nodes with the issue).
I see many verbose logging options reported by "ompi_info -a | grep verbose",
but I am not sure which one(s) could be helpful here. Would any of them be
useful here or are there any other ways to enable verbose logging to help with
tracking down the issue?
Thank you so much in advance.
Best regards,
----------------------------------------------------
Shan-Ho Tsai
University of Georgia, Athens GA
