HI, When the segmentation fault happens, I get the following trace:
(gdb) bt #0 0x00007fffee4f007d in ibv_close_xrcd (xrcd=0x200000000) at /usr/include/infiniband/verbs.h:1227 #1 0x00007fffee4f055f in mca_btl_openib_close_xrc_domain (device=0xfb20c0) at btl_openib_xrc.c:104 #2 0x00007fffee4da073 in device_destruct (device=0xfb20c0) at btl_openib_component.c:978 #3 0x00007fffee4ce9f7 in opal_obj_run_destructors (object=0xfb20c0) at ../../../../opal/class/opal_object.h:460 #4 0x00007fffee4d4f82 in mca_btl_openib_finalize_resources (btl=0xfbbc40) at btl_openib.c:1703 #5 0x00007fffee4d511c in mca_btl_openib_finalize (btl=0xfbbc40) at btl_openib.c:1730 #6 0x00007ffff76b26d6 in mca_btl_base_close () at base/btl_base_frame.c:192 #7 0x00007ffff769c73d in mca_base_framework_close (framework=0x7ffff795bda0) at mca_base_framework.c:214 #8 0x00007ffff7d448e2 in mca_bml_base_close () at base/bml_base_frame.c:130 #9 0x00007ffff769c73d in mca_base_framework_close (framework=0x7ffff7fe4f00) at mca_base_framework.c:214 #10 0x00007ffff7cd4d18 in ompi_mpi_finalize () at runtime/ompi_mpi_finalize.c:415 #11 0x00007ffff7cfee0b in PMPI_Finalize () at pfinalize.c:47 #12 0x0000000000400880 in main () Looks like the problem originates from the openib btl finalize flow (if openib wasn't chosen for the run). This doesn't happen however when ob1 is specified from the command line as David mentioned. Btl openib behaves differently in these cases - in mca_btl_openib_finalize_resources specifically. When pml yalla is specified from the command line, this flow isn't invoked at all so in this case the segv doesn't happen as well. Thanks, Alina. On Thu, Apr 21, 2016 at 6:55 PM, Nathan Hjelm <hje...@lanl.gov> wrote: > > In 1.10.x is possible for the BTLs to be in use by ether ob1 or an > oshmem component. In 2.x one-sided components can also use BTLs. The MTL > interface doesn't not provide support for accessing hardware atomics and > RDMA. As for UD it stands for Unconnected Datagram. Its usage gets > better messaage rates for small messages but really hurts bandwidth. Our > applications are bandwidth bound and not message rate bound so we should > be using XRC not UD. > > -Nathan > > On Thu, Apr 21, 2016 at 09:33:06AM -0600, David Shrader wrote: > > Hey Nathan, > > > > I thought only 1 pml could be loaded at a time, and the only pml that > > could use btl's was ob1. If that is the case, how can the openib btl > run > > at the same time as cm and yalla? > > > > Also, what is UD? > > > > Thanks, > > David > > > > On 04/21/2016 09:25 AM, Nathan Hjelm wrote: > > > > The openib btl should be able to run alongside cm/mxm or yalla. If I > > have time this weekend I will get on the mustang and see what the > > problem is. The best answer is to change the openmpi-mca-params.conf in > > the install to have pml = ob1. I have seen little to no benefit with > > using MXM on mustang. In fact, the default configuration (which uses UD) > > gets terrible bandwidth. > > > > -Nathan > > > > On Thu, Apr 21, 2016 at 01:48:46PM +0300, Alina Sklarevich wrote: > > > > David, thanks for the info you provided. > > I will try to dig in further to see what might be causing this issue. > > In the meantime, maybe Nathan can please comment about the openib btl > > behavior here? > > Thanks, > > Alina. > > On Wed, Apr 20, 2016 at 8:01 PM, David Shrader <dshra...@lanl.gov> > wrote: > > > > Hello Alina, > > > > Thank you for the information about how the pml components work. I > knew > > that the other components were being opened and ultimately closed > in > > favor of yalla, but I didn't realize that initial open would cause > a > > persistent change in the ompi runtime. > > > > Here's the information you requested about the ib network: > > > > - MOFED version: > > We are using the Open Fabrics Software as bundled by RedHat, and > my ib > > network folks say we're running something close to v1.5.4 > > - ibv_devinfo: > > [dshrader@mu0001 examples]$ ibv_devinfo > > hca_id: mlx4_0 > > transport: InfiniBand (0) > > fw_ver: 2.9.1000 > > node_guid: 0025:90ff:ff16:78d8 > > sys_image_guid: 0025:90ff:ff16:78db > > vendor_id: 0x02c9 > > vendor_part_id: 26428 > > hw_ver: 0xB0 > > board_id: SM_2121000001000 > > phys_port_cnt: 1 > > port: 1 > > state: PORT_ACTIVE (4) > > max_mtu: 4096 (5) > > active_mtu: 4096 (5) > > sm_lid: 250 > > port_lid: 366 > > port_lmc: 0x00 > > link_layer: InfiniBand > > > > I still get the seg fault when specifying the hca: > > > > $> mpirun -n 1 -mca btl_openib_receive_queues > > X,4096,1024:X,12288,512:X,65536,512 -mca btl_openib_if_include > mlx4_0 > > ./hello_c.x > > Hello, world, I am 0 of 1, (Open MPI v1.10.2, package: Open MPI > > dshra...@mu-fey.lanl.gov Distribution, ident: 1.10.2, repo rev: > > v1.10.1-145-g799148f, Jan 21, 2016, 135) > > > -------------------------------------------------------------------------- > > mpirun noticed that process rank 0 with PID 10045 on node mu0001 > exited > > on signal 11 (Segmentation fault). > > > -------------------------------------------------------------------------- > > > > I don't know if this helps, but the first time I tried the command > I > > mistyped the hca name. This got me a warning, but no seg fault: > > > > $> mpirun -n 1 -mca btl_openib_receive_queues > > X,4096,1024:X,12288,512:X,65536,512 -mca btl_openib_if_include > ml4_0 > > ./hello_c.x > > > -------------------------------------------------------------------------- > > WARNING: One or more nonexistent OpenFabrics devices/ports were > > specified: > > > > Host: mu0001 > > MCA parameter: mca_btl_if_include > > Nonexistent entities: ml4_0 > > > > These entities will be ignored. You can disable this warning by > > setting the btl_openib_warn_nonexistent_if MCA parameter to 0. > > > -------------------------------------------------------------------------- > > Hello, world, I am 0 of 1, (Open MPI v1.10.2, package: Open MPI > > dshra...@mu-fey.lanl.gov Distribution, ident: 1.10.2, repo rev: > > v1.10.1-145-g799148f, Jan 21, 2016, 135) > > > > So, telling the openib btl to use the actual hca didn't get the seg > > fault to go away, but giving it a dummy value did. > > > > Thanks, > > David > > > > On 04/20/2016 08:13 AM, Alina Sklarevich wrote: > > > > Hi David, > > I was able to reproduce the issue you reported. > > When the command line doesn't specify the components to use, > ompi will > > try to load/open all the ones available (and close them in the > end) > > and then choose the components according to their priority and > whether > > or not they were opened successfully. > > This means that even if pml yalla was the one running, other > > components were opened and closed as well. > > The parameter you are using - btl_openib_receive_queues, doesn't > have > > an effect on pml yalla. It only affects the openib btl which is > used > > by pml ob1. > > Using the verbosity of btl_base_verbose I see that when the > > segmentation fault happens, the code doesn't reach the phase of > > unloading the openib btl so perhaps the problem originates there > > (since pml yalla was already unloaded). > > Can you please try adding this mca parameter to your command > line to > > specify the HCA you are using? > > -mca btl_openib_if_include <hca> > > It made the segv go away for me. > > Can you please attach the output of ibv_devinfo and write the > MOFED > > version you are using? > > Thank you, > > Alina. > > On Wed, Apr 20, 2016 at 2:53 PM, Joshua Ladd < > jladd.m...@gmail.com> > > wrote: > > > > Hi, David > > > > We are looking into your report. > > > > Best, > > > > Josh > > On Tue, Apr 19, 2016 at 4:41 PM, David Shrader < > dshra...@lanl.gov> > > wrote: > > > > Hello, > > > > I have been investigating using XRC on a cluster with a > mellanox > > interconnect. I have found that in a certain situation I get > a seg > > fault. I am using 1.10.2 compiled with gcc 5.3.0, and the > simplest > > configure line that I have found that still results in the > seg > > fault is as follows: > > > > $> ./configure --with-hcoll --with-mxm --prefix=... > > > > I do have mxm 3.4.3065 and hcoll 3.3.768 installed in to > system > > space (/usr/lib64). If I use '--without-hcoll > --without-mxm,' the > > seg fault does not happen. > > > > The seg fault happens even when using examples/hello_c.c, so > here > > is an example of the seg fault using it: > > > > $> mpicc hello_c.c -o hello_c.x > > $> mpirun -n 1 ./hello_c.x > > Hello, world, I am 0 of 1, (Open MPI v1.10.2, package: Open > MPI > > dshra...@mu-fey.lanl.gov Distribution, ident: 1.10.2, repo > rev: > > v1.10.1-145-g799148f, Jan 21, 2016, 135) > > $> mpirun -n 1 -mca btl_openib_receive_queues > > X,4096,1024:X,12288,512:X,65536,512 > > Hello, world, I am 0 of 1, (Open MPI v1.10.2, package: Open > MPI > > dshra...@mu-fey.lanl.gov Distribution, ident: 1.10.2, repo > rev: > > v1.10.1-145-g799148f, Jan 21, 2016, 135) > > > -------------------------------------------------------------------------- > > mpirun noticed that process rank 0 with PID 22819 on node > mu0001 > > exited on signal 11 (Segmentation fault). > > > -------------------------------------------------------------------------- > > > > The seg fault happens no matter the number of ranks. I have > tried > > the above command with '-mca pml_base_verbose,' and it shows > that > > I am using the yalla pml: > > > > $> mpirun -n 1 -mca btl_openib_receive_queues > > X,4096,1024:X,12288,512:X,65536,512 -mca pml_base_verbose 100 > > ./hello_c.x > > ...output snipped... > > [mu0001.localdomain:22825] select: component cm not selected > / > > finalized > > [mu0001.localdomain:22825] select: component ob1 not > selected / > > finalized > > [mu0001.localdomain:22825] select: component yalla selected > > ...output snipped... > > > -------------------------------------------------------------------------- > > mpirun noticed that process rank 0 with PID 22825 on node > mu0001 > > exited on signal 11 (Segmentation fault). > > > -------------------------------------------------------------------------- > > > > Interestingly enough, if I tell mpirun what pml to use, the > seg > > fault goes away. The following command does not get the seg > fault: > > > > $> mpirun -n 1 -mca btl_openib_receive_queues > > X,4096,1024:X,12288,512:X,65536,512 -mca pml yalla > ./hello_c.x > > > > Passing either ob1 or cm to '-mca pml' also works. So it > seems > > that the seg fault comes about when the yalla pml is chosen > by > > default, when mxm/hcoll is involved, and using XRC. I'm not > sure > > if mxm is to blame, however, as using '-mca pml cm -mca mtl > mxm' > > with the XRC parameters doesn't throw the seg fault. > > > > Other information... > > OS: RHEL 6.7-based (TOSS) > > OpenFabrics: RedHat provided > > Kernel: 2.6.32-573.8.1.2chaos.ch5.4.x86_64 > > Config.log and 'ompi_info --all' are in the tarball > ompi.tar.bz2 > > which is attached. > > > > Is there something else I should be doing with the yalla pml > when > > using XRC? Regardless, I hope reporting the seg fault is > useful. > > > > Thanks, > > David > > > > -- > > David Shrader > > HPC-ENV High Performance Computer Systems > > Los Alamos National Lab > > Email: dshrader <at> lanl.gov > > > > _______________________________________________ > > devel mailing list > > de...@open-mpi.org > > Subscription: > http://www.open-mpi.org/mailman/listinfo.cgi/devel > > Link to this post: > > > http://www.open-mpi.org/community/lists/devel/2016/04/18786.php > > > > _______________________________________________ > > devel mailing list > > de...@open-mpi.org > > Subscription: > http://www.open-mpi.org/mailman/listinfo.cgi/devel > > Link to this post: > > > http://www.open-mpi.org/community/lists/devel/2016/04/18788.php > > > > _______________________________________________ > > devel mailing list > > de...@open-mpi.org > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > > Link to this post: > http://www.open-mpi.org/community/lists/devel/2016/04/18789.php > > > > -- > > David Shrader > > HPC-ENV High Performance Computer Systems > > Los Alamos National Lab > > Email: dshrader <at> lanl.gov > > > > _______________________________________________ > > devel mailing list > > de...@open-mpi.org > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > > Link to this post: > > http://www.open-mpi.org/community/lists/devel/2016/04/18793.php > > > > _______________________________________________ > > devel mailing list > > de...@open-mpi.org > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > > Link to this post: > http://www.open-mpi.org/community/lists/devel/2016/04/18801.php > > > > _______________________________________________ > > devel mailing list > > de...@open-mpi.org > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > > Link to this post: > http://www.open-mpi.org/community/lists/devel/2016/04/18804.php > > > > -- > > David Shrader > > HPC-ENV High Performance Computer Systems > > Los Alamos National Lab > > Email: dshrader <at> lanl.gov > > > _______________________________________________ > > devel mailing list > > de...@open-mpi.org > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > > Link to this post: > http://www.open-mpi.org/community/lists/devel/2016/04/18805.php > > > _______________________________________________ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2016/04/18806.php >
