George, Thanks. I Google'd "openmpi ompi_display_comm" and found this discussion https://github.com/open-mpi/ompi/issues/30, which looks promising. Alas, the OMPI I'm using at the moment is 3.1.4 from the OpenHPC CentOS 7 collection, so it produced nothing. It looks like I'll have to wait for them to grab a later version of OMPI, or I'll have to do it myself.
Thanks for your spot-on advice. Larry Baker US Geological Survey 650-329-5608 ba...@usgs.gov<mailto:ba...@usgs.gov> On Aug 20 2021, at 9:28:25 AM, George Bosilca <bosi...@icl.utk.edu<mailto:bosi...@icl.utk.edu>> wrote: Larry, There is no simple answer to your question as it depends on many, software and hardware, factors. A user selectable PML (our high level messaging layer) component will decide what protocol to be used to move the data around using what hardware. At this level you have a choice between OB1 (which will then use BTLs to move data), UCX (which will use it’s own internal capabilities, TLs), or CM (which will then default to an MTL, aka specialized for each hardware). For any PML other then OB1, the steps leading to the route selection is made internally by the software component used by the corresponding PML/MTL, and OMPI has basically no control on the decision. As a user you might be able to drive this decision using specialized parameters for the underlying library, but as far as I know there is no way to extract this information in a portable way. Let’s now assume you use the OB1 PML, which in turn will use the BTLs to drive the data movement. The decision on what BTL to use will then be driven by the exclusivity, latency and bandwidth exposed by the BTLs themselves. Some BTL will play nicely with others (aka sharing the load between 2 peers, or allowing multiple instances of itself), while others will prefer a more dedicated connection. One of the reasons OMPI does not have a proper mechanism to expose the routing decisions to the user, is this multiplicity of possibilities. The best we have right now is the ompi_display_comm MCA, that will show you the initialized BTLs on the run (which is a superset of what will be used). Try adding `--mca ompi_display_comm MPI_Finalize` to your mpirun command. Hopefully this answers your inquiry, at least partially. George. On Fri, Aug 20, 2021 at 11:39 Baker, Lawrence M via devel <devel@lists.open-mpi.org<mailto:devel@lists.open-mpi.org>> wrote: Florent, I agree with your description for the information I have already found through the use of -mca btl_base_verbose 10. I am requesting some way of obtaining the device selection and configuration information without the rest of the tracing. As far as routing, I was not referring to the OS network routing, but the routing layer that must exist in OpenMPI. When OMPI must send a message from one process to another, it has to pick a path, namely the one among what may be several possibilities that it has decided is the shortest when mpirun started the run. The internal OMPI routing table would be a summary of all the device choices made my mpirun, I assume. Thank you, Larry Baker US Geological Survey 650-329-5608 ba...@usgs.gov<mailto:ba...@usgs.gov> > On Aug 19 2021, at 11:30:58 PM, GERMAIN, Florent > <florent.germ...@atos.net<mailto:florent.germ...@atos.net>> wrote: > > > > This email has been received from outside of DOI - Use caution before > clicking on links, opening attachments, or responding. > > > > Hello, > Open MPI have given to you the only information it has: message is given to > mlx4_0:1. > As you are using InfiniBand network, the routing tables are handled by the ib > runtime. > You can gather information about ib routing through the ibroute command. > I hope it will help you. > > Florent Germain > Ingénieur de développement > florent.germ...@atos.net<mailto:florent.germ...@atos.net> > 1 Rue de Provence > 38130 Echirolles > Atos.net<http://Atos.net> > > -----Message d'origine----- > De : devel > <devel-boun...@lists.open-mpi.org<mailto:devel-boun...@lists.open-mpi.org>> > De la part de Baker, Lawrence M via devel > Envoyé : vendredi 20 août 2021 04:19 > À : Geoffrey Paulsen via devel > <devel@lists.open-mpi.org<mailto:devel@lists.open-mpi.org>> > Cc : Baker, Lawrence M <ba...@usgs.gov<mailto:ba...@usgs.gov>> > Objet : [OMPI devel] How to display device selection or routing info > > I am running into dead ends trying to find an mpirun option to display the > device selected for each process. I use > > mpirun --report-bindings > > to view the bindings to nodes/cpus/cores. But I cannot find an equivalent > option that displays the hardware path used for OpenMPI messaging. Or the > device parameters that go with that, if they are programmable. I have found > I can see the openib message selecting InfiniBand when I add > > -mca btl_base_verbose 10 > > [compute-0-30.local:02597] [rank=0] openib: using port mlx4_0:1 > > But, I get lots of other messages that are just noise for this purpose. > > There must be a routing table somewhere being constructed (ORTE?). I could > not find an MCA option to show that. > > Is there a better way to get this information? My Google searching has not > turned up anything. If not, I'd like to put in a request for an mpirun > option similar to --report-bindings that displays the routing table and/or > device bindings and programmable settings used. > > Thank you very much. > > Larry Baker > US Geological Survey > 650-329-5608 > ba...@usgs.gov<mailto:ba...@usgs.gov> > >
