The openib btl should be able to run alongside cm/mxm or yalla. If I have time this weekend I will get on the mustang and see what the problem is. The best answer is to change the openmpi-mca-params.conf in the install to have pml = ob1. I have seen little to no benefit with using MXM on mustang. In fact, the default configuration (which uses UD) gets terrible bandwidth.
-Nathan On Thu, Apr 21, 2016 at 01:48:46PM +0300, Alina Sklarevich wrote: > David, thanks for the info you provided. > I will try to dig in further to see what might be causing this issue. > In the meantime, maybe Nathan can please comment about the openib btl > behavior here? > Thanks, > Alina. > On Wed, Apr 20, 2016 at 8:01 PM, David Shrader <dshra...@lanl.gov> wrote: > > Hello Alina, > > Thank you for the information about how the pml components work. I knew > that the other components were being opened and ultimately closed in > favor of yalla, but I didn't realize that initial open would cause a > persistent change in the ompi runtime. > > Here's the information you requested about the ib network: > > - MOFED version: > We are using the Open Fabrics Software as bundled by RedHat, and my ib > network folks say we're running something close to v1.5.4 > - ibv_devinfo: > [dshrader@mu0001 examples]$ ibv_devinfo > hca_id: mlx4_0 > transport: InfiniBand (0) > fw_ver: 2.9.1000 > node_guid: 0025:90ff:ff16:78d8 > sys_image_guid: 0025:90ff:ff16:78db > vendor_id: 0x02c9 > vendor_part_id: 26428 > hw_ver: 0xB0 > board_id: SM_2121000001000 > phys_port_cnt: 1 > port: 1 > state: PORT_ACTIVE (4) > max_mtu: 4096 (5) > active_mtu: 4096 (5) > sm_lid: 250 > port_lid: 366 > port_lmc: 0x00 > link_layer: InfiniBand > > I still get the seg fault when specifying the hca: > > $> mpirun -n 1 -mca btl_openib_receive_queues > X,4096,1024:X,12288,512:X,65536,512 -mca btl_openib_if_include mlx4_0 > ./hello_c.x > Hello, world, I am 0 of 1, (Open MPI v1.10.2, package: Open MPI > dshra...@mu-fey.lanl.gov Distribution, ident: 1.10.2, repo rev: > v1.10.1-145-g799148f, Jan 21, 2016, 135) > > -------------------------------------------------------------------------- > mpirun noticed that process rank 0 with PID 10045 on node mu0001 exited > on signal 11 (Segmentation fault). > > -------------------------------------------------------------------------- > > I don't know if this helps, but the first time I tried the command I > mistyped the hca name. This got me a warning, but no seg fault: > > $> mpirun -n 1 -mca btl_openib_receive_queues > X,4096,1024:X,12288,512:X,65536,512 -mca btl_openib_if_include ml4_0 > ./hello_c.x > > -------------------------------------------------------------------------- > WARNING: One or more nonexistent OpenFabrics devices/ports were > specified: > > Host: mu0001 > MCA parameter: mca_btl_if_include > Nonexistent entities: ml4_0 > > These entities will be ignored. You can disable this warning by > setting the btl_openib_warn_nonexistent_if MCA parameter to 0. > > -------------------------------------------------------------------------- > Hello, world, I am 0 of 1, (Open MPI v1.10.2, package: Open MPI > dshra...@mu-fey.lanl.gov Distribution, ident: 1.10.2, repo rev: > v1.10.1-145-g799148f, Jan 21, 2016, 135) > > So, telling the openib btl to use the actual hca didn't get the seg > fault to go away, but giving it a dummy value did. > > Thanks, > David > > On 04/20/2016 08:13 AM, Alina Sklarevich wrote: > > Hi David, > I was able to reproduce the issue you reported. > When the command line doesn't specify the components to use, ompi will > try to load/open all the ones available (and close them in the end) > and then choose the components according to their priority and whether > or not they were opened successfully. > This means that even if pml yalla was the one running, other > components were opened and closed as well. > The parameter you are using - btl_openib_receive_queues, doesn't have > an effect on pml yalla. It only affects the openib btl which is used > by pml ob1. > Using the verbosity of btl_base_verbose I see that when the > segmentation fault happens, the code doesn't reach the phase of > unloading the openib btl so perhaps the problem originates there > (since pml yalla was already unloaded). > Can you please try adding this mca parameter to your command line to > specify the HCA you are using? > -mca btl_openib_if_include <hca> > It made the segv go away for me. > Can you please attach the output of ibv_devinfo and write the MOFED > version you are using? > Thank you, > Alina. > On Wed, Apr 20, 2016 at 2:53 PM, Joshua Ladd <jladd.m...@gmail.com> > wrote: > > Hi, David > > We are looking into your report. > > Best, > > Josh > On Tue, Apr 19, 2016 at 4:41 PM, David Shrader <dshra...@lanl.gov> > wrote: > > Hello, > > I have been investigating using XRC on a cluster with a mellanox > interconnect. I have found that in a certain situation I get a seg > fault. I am using 1.10.2 compiled with gcc 5.3.0, and the simplest > configure line that I have found that still results in the seg > fault is as follows: > > $> ./configure --with-hcoll --with-mxm --prefix=... > > I do have mxm 3.4.3065 and hcoll 3.3.768 installed in to system > space (/usr/lib64). If I use '--without-hcoll --without-mxm,' the > seg fault does not happen. > > The seg fault happens even when using examples/hello_c.c, so here > is an example of the seg fault using it: > > $> mpicc hello_c.c -o hello_c.x > $> mpirun -n 1 ./hello_c.x > Hello, world, I am 0 of 1, (Open MPI v1.10.2, package: Open MPI > dshra...@mu-fey.lanl.gov Distribution, ident: 1.10.2, repo rev: > v1.10.1-145-g799148f, Jan 21, 2016, 135) > $> mpirun -n 1 -mca btl_openib_receive_queues > X,4096,1024:X,12288,512:X,65536,512 > Hello, world, I am 0 of 1, (Open MPI v1.10.2, package: Open MPI > dshra...@mu-fey.lanl.gov Distribution, ident: 1.10.2, repo rev: > v1.10.1-145-g799148f, Jan 21, 2016, 135) > > -------------------------------------------------------------------------- > mpirun noticed that process rank 0 with PID 22819 on node mu0001 > exited on signal 11 (Segmentation fault). > > -------------------------------------------------------------------------- > > The seg fault happens no matter the number of ranks. I have tried > the above command with '-mca pml_base_verbose,' and it shows that > I am using the yalla pml: > > $> mpirun -n 1 -mca btl_openib_receive_queues > X,4096,1024:X,12288,512:X,65536,512 -mca pml_base_verbose 100 > ./hello_c.x > ...output snipped... > [mu0001.localdomain:22825] select: component cm not selected / > finalized > [mu0001.localdomain:22825] select: component ob1 not selected / > finalized > [mu0001.localdomain:22825] select: component yalla selected > ...output snipped... > > -------------------------------------------------------------------------- > mpirun noticed that process rank 0 with PID 22825 on node mu0001 > exited on signal 11 (Segmentation fault). > > -------------------------------------------------------------------------- > > Interestingly enough, if I tell mpirun what pml to use, the seg > fault goes away. The following command does not get the seg fault: > > $> mpirun -n 1 -mca btl_openib_receive_queues > X,4096,1024:X,12288,512:X,65536,512 -mca pml yalla ./hello_c.x > > Passing either ob1 or cm to '-mca pml' also works. So it seems > that the seg fault comes about when the yalla pml is chosen by > default, when mxm/hcoll is involved, and using XRC. I'm not sure > if mxm is to blame, however, as using '-mca pml cm -mca mtl mxm' > with the XRC parameters doesn't throw the seg fault. > > Other information... > OS: RHEL 6.7-based (TOSS) > OpenFabrics: RedHat provided > Kernel: 2.6.32-573.8.1.2chaos.ch5.4.x86_64 > Config.log and 'ompi_info --all' are in the tarball ompi.tar.bz2 > which is attached. > > Is there something else I should be doing with the yalla pml when > using XRC? Regardless, I hope reporting the seg fault is useful. > > Thanks, > David > > -- > David Shrader > HPC-ENV High Performance Computer Systems > Los Alamos National Lab > Email: dshrader <at> lanl.gov > > _______________________________________________ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2016/04/18786.php > > _______________________________________________ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2016/04/18788.php > > _______________________________________________ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2016/04/18789.php > > -- > David Shrader > HPC-ENV High Performance Computer Systems > Los Alamos National Lab > Email: dshrader <at> lanl.gov > > _______________________________________________ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2016/04/18793.php > _______________________________________________ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2016/04/18801.php
pgpFswJzLCfh6.pgp
Description: PGP signature
