David, thanks for the info you provided.
I will try to dig in further to see what might be causing this issue.
In the meantime, maybe Nathan can please comment about the openib btl
behavior here?
Thanks,
Alina.
On Wed, Apr 20, 2016 at 8:01 PM, David Shrader <dshra...@lanl.gov> wrote:
Hello Alina,
Thank you for the information about how the pml components work. I knew
that the other components were being opened and ultimately closed in
favor of yalla, but I didn't realize that initial open would cause a
persistent change in the ompi runtime.
Here's the information you requested about the ib network:
- MOFED version:
We are using the Open Fabrics Software as bundled by RedHat, and my ib
network folks say we're running something close to v1.5.4
- ibv_devinfo:
[dshrader@mu0001 examples]$ ibv_devinfo
hca_id: mlx4_0
transport: InfiniBand (0)
fw_ver: 2.9.1000
node_guid: 0025:90ff:ff16:78d8
sys_image_guid: 0025:90ff:ff16:78db
vendor_id: 0x02c9
vendor_part_id: 26428
hw_ver: 0xB0
board_id: SM_2121000001000
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 250
port_lid: 366
port_lmc: 0x00
link_layer: InfiniBand
I still get the seg fault when specifying the hca:
$> mpirun -n 1 -mca btl_openib_receive_queues
X,4096,1024:X,12288,512:X,65536,512 -mca btl_openib_if_include mlx4_0
./hello_c.x
Hello, world, I am 0 of 1, (Open MPI v1.10.2, package: Open MPI
dshra...@mu-fey.lanl.gov Distribution, ident: 1.10.2, repo rev:
v1.10.1-145-g799148f, Jan 21, 2016, 135)
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 10045 on node mu0001 exited
on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
I don't know if this helps, but the first time I tried the command I
mistyped the hca name. This got me a warning, but no seg fault:
$> mpirun -n 1 -mca btl_openib_receive_queues
X,4096,1024:X,12288,512:X,65536,512 -mca btl_openib_if_include ml4_0
./hello_c.x
--------------------------------------------------------------------------
WARNING: One or more nonexistent OpenFabrics devices/ports were
specified:
Host: mu0001
MCA parameter: mca_btl_if_include
Nonexistent entities: ml4_0
These entities will be ignored. You can disable this warning by
setting the btl_openib_warn_nonexistent_if MCA parameter to 0.
--------------------------------------------------------------------------
Hello, world, I am 0 of 1, (Open MPI v1.10.2, package: Open MPI
dshra...@mu-fey.lanl.gov Distribution, ident: 1.10.2, repo rev:
v1.10.1-145-g799148f, Jan 21, 2016, 135)
So, telling the openib btl to use the actual hca didn't get the seg
fault to go away, but giving it a dummy value did.
Thanks,
David
On 04/20/2016 08:13 AM, Alina Sklarevich wrote:
Hi David,
I was able to reproduce the issue you reported.
When the command line doesn't specify the components to use, ompi will
try to load/open all the ones available (and close them in the end)
and then choose the components according to their priority and whether
or not they were opened successfully.
This means that even if pml yalla was the one running, other
components were opened and closed as well.
The parameter you are using - btl_openib_receive_queues, doesn't have
an effect on pml yalla. It only affects the openib btl which is used
by pml ob1.
Using the verbosity of btl_base_verbose I see that when the
segmentation fault happens, the code doesn't reach the phase of
unloading the openib btl so perhaps the problem originates there
(since pml yalla was already unloaded).
Can you please try adding this mca parameter to your command line to
specify the HCA you are using?
-mca btl_openib_if_include <hca>
It made the segv go away for me.
Can you please attach the output of ibv_devinfo and write the MOFED
version you are using?
Thank you,
Alina.
On Wed, Apr 20, 2016 at 2:53 PM, Joshua Ladd <jladd.m...@gmail.com>
wrote:
Hi, David
We are looking into your report.
Best,
Josh
On Tue, Apr 19, 2016 at 4:41 PM, David Shrader <dshra...@lanl.gov>
wrote:
Hello,
I have been investigating using XRC on a cluster with a mellanox
interconnect. I have found that in a certain situation I get a seg
fault. I am using 1.10.2 compiled with gcc 5.3.0, and the simplest
configure line that I have found that still results in the seg
fault is as follows:
$> ./configure --with-hcoll --with-mxm --prefix=...
I do have mxm 3.4.3065 and hcoll 3.3.768 installed in to system
space (/usr/lib64). If I use '--without-hcoll --without-mxm,' the
seg fault does not happen.
The seg fault happens even when using examples/hello_c.c, so here
is an example of the seg fault using it:
$> mpicc hello_c.c -o hello_c.x
$> mpirun -n 1 ./hello_c.x
Hello, world, I am 0 of 1, (Open MPI v1.10.2, package: Open MPI
dshra...@mu-fey.lanl.gov Distribution, ident: 1.10.2, repo rev:
v1.10.1-145-g799148f, Jan 21, 2016, 135)
$> mpirun -n 1 -mca btl_openib_receive_queues
X,4096,1024:X,12288,512:X,65536,512
Hello, world, I am 0 of 1, (Open MPI v1.10.2, package: Open MPI
dshra...@mu-fey.lanl.gov Distribution, ident: 1.10.2, repo rev:
v1.10.1-145-g799148f, Jan 21, 2016, 135)
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 22819 on node mu0001
exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
The seg fault happens no matter the number of ranks. I have tried
the above command with '-mca pml_base_verbose,' and it shows that
I am using the yalla pml:
$> mpirun -n 1 -mca btl_openib_receive_queues
X,4096,1024:X,12288,512:X,65536,512 -mca pml_base_verbose 100
./hello_c.x
...output snipped...
[mu0001.localdomain:22825] select: component cm not selected /
finalized
[mu0001.localdomain:22825] select: component ob1 not selected /
finalized
[mu0001.localdomain:22825] select: component yalla selected
...output snipped...
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 22825 on node mu0001
exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
Interestingly enough, if I tell mpirun what pml to use, the seg
fault goes away. The following command does not get the seg fault:
$> mpirun -n 1 -mca btl_openib_receive_queues
X,4096,1024:X,12288,512:X,65536,512 -mca pml yalla ./hello_c.x
Passing either ob1 or cm to '-mca pml' also works. So it seems
that the seg fault comes about when the yalla pml is chosen by
default, when mxm/hcoll is involved, and using XRC. I'm not sure
if mxm is to blame, however, as using '-mca pml cm -mca mtl mxm'
with the XRC parameters doesn't throw the seg fault.
Other information...
OS: RHEL 6.7-based (TOSS)
OpenFabrics: RedHat provided
Kernel: 2.6.32-573.8.1.2chaos.ch5.4.x86_64
Config.log and 'ompi_info --all' are in the tarball ompi.tar.bz2
which is attached.
Is there something else I should be doing with the yalla pml when
using XRC? Regardless, I hope reporting the seg fault is useful.
Thanks,
David
--
David Shrader
HPC-ENV High Performance Computer Systems
Los Alamos National Lab
Email: dshrader <at> lanl.gov
_______________________________________________
devel mailing list
de...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this post:
http://www.open-mpi.org/community/lists/devel/2016/04/18786.php
_______________________________________________
devel mailing list
de...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this post:
http://www.open-mpi.org/community/lists/devel/2016/04/18788.php
_______________________________________________
devel mailing list
de...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this post:
http://www.open-mpi.org/community/lists/devel/2016/04/18789.php
--
David Shrader
HPC-ENV High Performance Computer Systems
Los Alamos National Lab
Email: dshrader <at> lanl.gov
_______________________________________________
devel mailing list
de...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this post:
http://www.open-mpi.org/community/lists/devel/2016/04/18793.php
_______________________________________________
devel mailing list
de...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this post:
http://www.open-mpi.org/community/lists/devel/2016/04/18801.php
