This material is working for nearly 10 years with several generations of nodes and OpenMPI without any problem. Today it is possible to found refurbished parts at low price on the web and it can help building small clusters. it is really more efficient than 10Gb ethernet for parallel codes due to the very low latency. Now I'm moving to 100-200Gb/s infiniband architectures... for the next 10 years. ;-)
Patrick Le 09/05/2020 à 16:09, Heinz, Michael William a écrit : > That it! I was trying to remember what the setting was but I haven’t > worked on those HCAs since around 2012, so it was faint. > > That said, I found the Intel TrueScale manual online > at > https://www.intel.com/content/dam/support/us/en/documents/network-and-i-o/fabric-products/OFED_Host_Software_UserGuide_G91902_06.pdf > <https://www.intel.com/content/dam/support/us/en/documents/network-and-i-o/fabric-products/OFED_Host_Software_UserGuide_G91902_06.pdf#page72> > > TS is the same hardware as the old QLogic QDR HCAs so the manual might > be helpful to you in the future. > > Sent from my iPad > >> On May 9, 2020, at 9:52 AM, Patrick Bégou via users >> <users@lists.open-mpi.org> wrote: >> >> >> Le 08/05/2020 à 21:56, Prentice Bisbal via users a écrit : >>> >>> We often get the following errors when more than one job runs on the >>> same compute node. We are using Slurm with OpenMPI. The IB cards are >>> QLogic using PSM: >>> >>> 10698ipath_userinit: assign_context command failed: Network is down >>> node01.10698can't open /dev/ipath, network down (err=26) >>> node01.10703ipath_userinit: assign_context command failed: Network >>> is down >>> node01.10703can't open /dev/ipath, network down (err=26) >>> node01.10701ipath_userinit: assign_context command failed: Network >>> is down >>> node01.10701can't open /dev/ipath, network down (err=26) >>> node01.10700ipath_userinit: assign_context command failed: Network >>> is down >>> node01.10700can't open /dev/ipath, network down (err=26) >>> node01.10697ipath_userinit: assign_context command failed: Network >>> is down >>> node01.10697can't open /dev/ipath, network down (err=26) >>> -------------------------------------------------------------------------- >>> PSM was unable to open an endpoint. Please make sure that the >>> network link is >>> active on the node and the hardware is functioning. >>> >>> Error: Could not detect network connectivity >>> -------------------------------------------------------------------------- >>> >>> Any Ideas how to fix this? >>> >>> -- >>> Prentice >> >> >> Hi Prentice, >> >> This is not openMPI related but merely due to your hardware. I've not >> many details but I think this occurs when several jobs share the same >> node and you have a large number of cores on these nodes (> 14). If >> this is the case: >> >> On Qlogic (I'm using such a hardware at this time) you have 16 >> channel for communication on each HBA and, if I remember what I had >> read many years ago, 2 are dedicated to the system. When launching >> MPI applications, each process of a job request for it's own >> dedicated channel if available, else they share ALL the available >> channels. So if a second job starts on the same node it do not >> remains any available channel. >> >> To avoid this situation I force sharing the channels (my nodes have >> 20 codes) by 2 MPI processes. You can set this with a simple >> environment variable. On all my cluster nodes I create the file: >> >> */etc/profile.d/ibsetcontext.sh* >> >> And it contains: >> >> # allow 4 processes to share an hardware MPI context >> # in infiniband with PSM >> *export PSM_RANKS_PER_CONTEXT=2* >> >> Of course if some people manage to oversubscribe on the cores (more >> than one process by core) it could rise again the problem but we do >> not oversubscribe. >> >> Hope this can help you. >> >> Patrick >>
