Ralph,
It doesn't look to me like there is a lack of a dl component.
It looks to me like libinfinipath is doing something bad when dl_open()ed
by mtl_ofi.
Regardless, the requested verbose output is attached.
As for cores, I cannot get any because somebody is catching SIGABRT.
What I do see is files like
ring_c.80s-19092,sl7x.localdomain.btr
which contain the same backtrace as in the attachment.
So, I tried "mpirun -mca btl sm,self -np 1 gdb examples/ring_c" and was
able to get the backtrace given below.
It is worth noting that this is apparently *not* a regression.
I find that I get the same error with the released 1.10.0 built just now.
I know that I did *NOT* see this problem with my original testing of
1.10.0rc7 (still have the build and its logs)
However, trying today to run ring_c from my 1.10.0rc7 build fails in the
same manner.
So, something has changed on this system since my last testing there.
Thanks to /var/log/yum.log I can tell you exactly the 143 changes since I
tested 1.10.0rc7 if you care.
The main change appears to be a switch from a MOFED-based install to the
OFED packaged with RHEL7.
So, I am currently willing to bet that I've got a broken MOFED/OFED hybrid
installation.
If that is the case (and I am looking into it now) then this is NOT an Open
MPI problem.
I will try to let you know what I discover.
-Paul
ring_c: route/tc.c:973: rtnl_tc_register: Assertion `0' failed.
Program received signal SIGABRT, Aborted.
0x00007ffff74905e9 in __GI_raise (sig=sig@entry=6)
at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
(gdb) where
#0 0x00007ffff74905e9 in __GI_raise (sig=sig@entry=6)
at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007ffff7491cf8 in __GI_abort () at abort.c:90
#2 0x00007ffff7489556 in __assert_fail_base (
fmt=0x7ffff75d9168 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
assertion=assertion@entry=0x7ffff16d3230 "0",
file=file@entry=0x7ffff16d2a50 "route/tc.c", line=line@entry=973,
function=function@entry=0x7ffff16d2b90 <__PRETTY_FUNCTION__.13204>
"rtnl_tc_register") at assert.c:92
#3 0x00007ffff7489602 in __GI___assert_fail (
assertion=assertion@entry=0x7ffff16d3230 "0",
file=file@entry=0x7ffff16d2a50 "route/tc.c", line=line@entry=973,
function=function@entry=0x7ffff16d2b90 <__PRETTY_FUNCTION__.13204>
"rtnl_tc_register") at assert.c:101
#4 0x00007ffff16be3d9 in rtnl_tc_register (ops=0x7ffff18e86c0
<blackhole_ops>)
at route/tc.c:973
#5 0x00007ffff7dea503 in call_init (env=0x7fffffffdc58,
argv=0x7fffffffdc48,
argc=1, l=<optimized out>) at dl-init.c:82
#6 _dl_init (main_map=main_map@entry=0x6ae5f0, argc=1,
argv=0x7fffffffdc48,
env=0x7fffffffdc58) at dl-init.c:131
#7 0x00007ffff7deeb44 in dl_open_worker (a=a@entry=0x7fffffffd168)
at dl-open.c:566
#8 0x00007ffff7dea314 in _dl_catch_error (
objname=objname@entry=0x7fffffffd158,
errstring=errstring@entry=0x7fffffffd160,
mallocedp=mallocedp@entry=0x7fffffffd150,
operate=operate@entry=0x7ffff7dee7b0 <dl_open_worker>,
args=args@entry=0x7fffffffd168) at dl-error.c:177
#9 0x00007ffff7dee25b in _dl_open (
file=0x6ae520
"/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/openmpi/mca_mtl_ofi.so",
mode=-2147483391, caller_dlopen=<optimized out>,
nsid=-2, argc=1, argv=0x7fffffffdc48, env=0x7fffffffdc58) at
dl-open.c:656
#10 0x00007ffff6c7402b in dlopen_doit (a=a@entry=0x7fffffffd370) at
dlopen.c:66
#11 0x00007ffff7dea314 in _dl_catch_error (objname=0x641600,
errstring=0x641608, mallocedp=0x6415f8,
operate=0x7ffff6c73fd0 <dlopen_doit>, args=0x7fffffffd370)
at dl-error.c:177
#12 0x00007ffff6c7462d in _dlerror_run (
operate=operate@entry=0x7ffff6c73fd0 <dlopen_doit>,
args=args@entry=0x7fffffffd370) at dlerror.c:163
#13 0x00007ffff6c740c1 in __dlopen (file=<optimized out>, mode=<optimized
out>)
at dlopen.c:87
#14 0x00007ffff6ef8ec4 in do_dlopen (
fname=0x6ae520
"/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/openmpi/mca_mtl_ofi.so",
flags=257, handle=0x7fffffffd488,
err_msg=0x7fffffffd538)
at
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/openmpi-1.10.1rc1/opal/mca/dl/dlopen/dl_dlopen_module.c:38
#15 0x00007ffff6ef909c in dlopen_open (
fname=0x6ac0ba
"/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/openmpi/mca_mtl_ofi",
use_ext=true, private_namespace=false,
handle=0x7fffffffd5c8, err_msg=0x7fffffffd538)
at
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/openmpi-1.10.1rc1/opal/mca/dl/dlopen/dl_dlopen_module.c:95
#16 0x00007ffff6ef8bfa in opal_dl_open (
fname=0x6ac0ba
"/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/openmpi/mca_mtl_ofi",
use_ext=true, private_namespace=false,
handle=0x7fffffffd5c8, err_msg=0x7fffffffd538)
at
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/openmpi-1.10.1rc1/opal/mca/dl/base/dl_base_fns.c:32
#17 0x00007ffff6ed0265 in open_component (target_file=0x6ab010,
found_components=0x7fffffffd7a0)
at
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/openmpi-1.10.1rc1/opal/mca/base/mca_base_component_find.c:558
#18 0x00007ffff6ecfd45 in find_dyn_components (path=0x0,
type_name=0x7ffff7b891e0 "mtl", names=0x0, include_mode=true,
found_components=0x7fffffffd7a0)
at
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/openmpi-1.10.1rc1/opal/mca/base/mca_base_component_find.c:446
#19 0x00007ffff6ecf444 in mca_base_component_find (directory=0x0,
type=0x7ffff7b891e0 "mtl",
static_components=0x7ffff7dc73a8 <mca_mtl_base_static_components>,
requested_components=0x0, found_components=0x7fffffffd7a0,
open_dso_components=true)
at
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/openmpi-1.10.1rc1/opal/mca/base/mca_base_component_find.c:190
#20 0x00007ffff6ee157a in mca_base_framework_components_register (
framework=0x7ffff7dc2560 <ompi_mtl_base_framework>,
flags=MCA_BASE_REGISTER_DEFAULT)
at
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/openmpi-1.10.1rc1/opal/mca/base/mca_base_components_register.c:57
#21 0x00007ffff6ee1edd in mca_base_framework_register (
framework=0x7ffff7dc2560 <ompi_mtl_base_framework>,
flags=MCA_BASE_REGISTER_DEFAULT)
at
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/openmpi-1.10.1rc1/opal/mca/base/mca_base_framework.c:115
#22 0x00007ffff6ee1f4b in mca_base_framework_open (
framework=0x7ffff7dc2560 <ompi_mtl_base_framework>,
flags=MCA_BASE_OPEN_DEFAULT)
at
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/openmpi-1.10.1rc1/opal/mca/base/mca_base_framework.c:134
#23 0x00007ffff30cabf5 in mca_pml_cm_component_open ()
at
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/openmpi-1.10.1rc1/ompi/mca/pml/cm/pml_cm_component.c:120
#24 0x00007ffff6ed2e58 in open_components (
framework=0x7ffff7dc4e60 <ompi_pml_base_framework>)
at
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/openmpi-1.10.1rc1/opal/mca/base/mca_base_components_open.c:173
#25 0x00007ffff6ed2b4b in mca_base_framework_components_open (
framework=0x7ffff7dc4e60 <ompi_pml_base_framework>,
flags=MCA_BASE_OPEN_DEFAULT)
at
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/openmpi-1.10.1rc1/opal/mca/base/mca_base_components_open.c:66
#26 0x00007ffff7b3efa7 in mca_pml_base_open (flags=MCA_BASE_OPEN_DEFAULT)
at
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/openmpi-1.10.1rc1/ompi/mca/pml/base/pml_base_frame.c:187
#27 0x00007ffff6ee1fe6 in mca_base_framework_open (
framework=0x7ffff7dc4e60 <ompi_pml_base_framework>,
flags=MCA_BASE_OPEN_DEFAULT)
at
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/openmpi-1.10.1rc1/opal/mca/base/mca_base_framework.c:158
#28 0x00007ffff7a90755 in ompi_mpi_init (argc=1, argv=0x7fffffffdc48,
requested=0, provided=0x7fffffffdafc)
at
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/openmpi-1.10.1rc1/ompi/runtime/ompi_mpi_init.c:578
#29 0x00007ffff7ad00b6 in PMPI_Init (argc=0x7fffffffdb3c,
argv=0x7fffffffdb30)
at pinit.c:84
#30 0x0000000000400969 in main (argc=1, argv=0x7fffffffdc48) at ring_c.c:19
I
On Sat, Oct 3, 2015 at 4:07 PM, Ralph Castain <r...@open-mpi.org> wrote:
> Paul
>
> At a glance, it looks to me like no opal_dl component was selected. Any
> chance you could add -mca dl_base_verbose 100 and get a line number for us
> from one of those core files?
>
>
> On Oct 3, 2015, at 10:41 AM, Paul Hargrove <phhargr...@lbl.gov> wrote:
>
>
> On Sat, Oct 3, 2015 at 3:13 AM, Jeff Squyres (jsquyres) <
> jsquy...@cisco.com> wrote:
>
>> v1.10.1 is primarily a bug-fix release. rc1 has been released; it's in
>> the usual place:
>>
>> http://www.open-mpi.org/software/ompi/v1.10/
>>
>> Please test!
>>
>
>
> My tests experienced a failure (SEGV) on a fairly vanilla Linux VM running
> Scientific Linux 7 (RHEL7 clone).
> See attached output.
>
> -Paul
>
> --
> Paul H. Hargrove phhargr...@lbl.gov
> Computer Languages & Systems Software (CLaSS) Group
> Computer Science Department Tel: +1-510-495-2352
> Lawrence Berkeley National Laboratory Fax: +1-510-486-6900
> <ring_c.txt>_______________________________________________
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2015/10/18131.php
>
>
>
> _______________________________________________
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2015/10/18134.php
>
--
Paul H. Hargrove phhargr...@lbl.gov
Computer Languages & Systems Software (CLaSS) Group
Computer Science Department Tel: +1-510-495-2352
Lawrence Berkeley National Laboratory Fax: +1-510-486-6900
$ mpirun -mca btl sm,self -mca dl_base_verbose 100 -np 2 examples/ring_c
[sl7x.localdomain:19088] mca: base: components_register: registering dl
components
[sl7x.localdomain:19088] mca: base: components_register: found loaded component
dlopen
[sl7x.localdomain:19088] mca: base: components_register: component dlopen
register function successful
[sl7x.localdomain:19088] mca: base: components_open: opening dl components
[sl7x.localdomain:19088] mca: base: components_open: found loaded component
dlopen
[sl7x.localdomain:19088] mca: base: components_open: component dlopen open
function successful
[sl7x.localdomain:19088] mca:base:select: Auto-selecting dl components
[sl7x.localdomain:19088] mca:base:select:( dl) Querying component [dlopen]
[sl7x.localdomain:19088] mca:base:select:( dl) Query of component [dlopen]
set priority to 80
[sl7x.localdomain:19088] mca:base:select:( dl) Selected component [dlopen]
[sl7x.localdomain:19091] mca: base: components_register: registering dl
components
[sl7x.localdomain:19091] mca: base: components_register: found loaded component
dlopen
[sl7x.localdomain:19091] mca: base: components_register: component dlopen
register function successful
[sl7x.localdomain:19091] mca: base: components_open: opening dl components
[sl7x.localdomain:19091] mca: base: components_open: found loaded component
dlopen
[sl7x.localdomain:19091] mca: base: components_open: component dlopen open
function successful
[sl7x.localdomain:19091] mca:base:select: Auto-selecting dl components
[sl7x.localdomain:19091] mca:base:select:( dl) Querying component [dlopen]
[sl7x.localdomain:19091] mca:base:select:( dl) Query of component [dlopen]
set priority to 80
[sl7x.localdomain:19091] mca:base:select:( dl) Selected component [dlopen]
[sl7x.localdomain:19092] mca: base: components_register: registering dl
components
[sl7x.localdomain:19092] mca: base: components_register: found loaded component
dlopen
[sl7x.localdomain:19092] mca: base: components_register: component dlopen
register function successful
[sl7x.localdomain:19092] mca: base: components_open: opening dl components
[sl7x.localdomain:19092] mca: base: components_open: found loaded component
dlopen
[sl7x.localdomain:19092] mca: base: components_open: component dlopen open
function successful
[sl7x.localdomain:19092] mca:base:select: Auto-selecting dl components
[sl7x.localdomain:19092] mca:base:select:( dl) Querying component [dlopen]
[sl7x.localdomain:19092] mca:base:select:( dl) Query of component [dlopen]
set priority to 80
[sl7x.localdomain:19092] mca:base:select:( dl) Selected component [dlopen]
ring_c: route/tc.c:973: rtnl_tc_register: Assertion `0' failed.
ring_c:19091 terminated with signal 6 at PC=7f27ca2ab5e9 SP=7fff6c152748.
Backtrace:
/lib64/libc.so.6(gsignal+0x39)[0x7f27ca2ab5e9]
/lib64/libc.so.6(abort+0x148)[0x7f27ca2accf8]
/lib64/libc.so.6(+0x2e556)[0x7f27ca2a4556]
/lib64/libc.so.6(+0x2e602)[0x7f27ca2a4602]
/lib64/libnl-route-3.so.200(+0x213d9)[0x7f27c44d93d9]
/lib64/ld-linux-x86-64.so.2(+0xf503)[0x7f27cac05503]
/lib64/ld-linux-x86-64.so.2(+0x13b44)[0x7f27cac09b44]
/lib64/ld-linux-x86-64.so.2(+0xf314)[0x7f27cac05314]
/lib64/ld-linux-x86-64.so.2(+0x1325b)[0x7f27cac0925b]
/lib64/libdl.so.2(+0x102b)[0x7f27c9a8f02b]
/lib64/ld-linux-x86-64.so.2(+0xf314)[0x7f27cac05314]
/lib64/libdl.so.2(+0x162d)[0x7f27c9a8f62d]
/lib64/libdl.so.2(dlopen+0x31)[0x7f27c9a8f0c1]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(+0x81ec4)[0x7f27c9d13ec4]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(+0x8209c)[0x7f27c9d1409c]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(opal_dl_open+0x6e)[0x7f27c9d13bfa]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(+0x59265)[0x7f27c9ceb265]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(+0x58d45)[0x7f27c9cead45]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(mca_base_component_find+0x1d2)[0x7f27c9cea444]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(mca_base_framework_components_register+0x83)[0x7f27c9cfc57a]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(mca_base_framework_register+0x217)[0x7f27c9cfcedd]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(mca_base_framework_open+0x46)[0x7f27c9cfcf4b]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/openmpi/mca_pml_cm.so(+0x5bf5)[0x7f27c5ee5bf5]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(+0x5be58)[0x7f27c9cede58]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(mca_base_framework_components_open+0x69)[0x7f27c9cedb4b]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libmpi.so.12(+0x106fa7)[0x7f27ca959fa7]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(mca_base_framework_open+0xe1)[0x7f27c9cfcfe6]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libmpi.so.12(ompi_mpi_init+0x5ab)[0x7f27ca8ab755]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libmpi.so.12(MPI_Init+0x180)[0x7f27ca8eb0b6]
examples/ring_c[0x400969]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f27ca297af5]
examples/ring_c[0x400879]
ring_c: route/tc.c:973: rtnl_tc_register: Assertion `0' failed.
ring_c:19092 terminated with signal 6 at PC=7f8d2667d5e9 SP=7ffe1f096d98.
Backtrace:
/lib64/libc.so.6(gsignal+0x39)[0x7f8d2667d5e9]
/lib64/libc.so.6(abort+0x148)[0x7f8d2667ecf8]
/lib64/libc.so.6(+0x2e556)[0x7f8d26676556]
/lib64/libc.so.6(+0x2e602)[0x7f8d26676602]
/lib64/libnl-route-3.so.200(+0x213d9)[0x7f8d208ab3d9]
/lib64/ld-linux-x86-64.so.2(+0xf503)[0x7f8d26fd7503]
/lib64/ld-linux-x86-64.so.2(+0x13b44)[0x7f8d26fdbb44]
/lib64/ld-linux-x86-64.so.2(+0xf314)[0x7f8d26fd7314]
/lib64/ld-linux-x86-64.so.2(+0x1325b)[0x7f8d26fdb25b]
/lib64/libdl.so.2(+0x102b)[0x7f8d25e6102b]
/lib64/ld-linux-x86-64.so.2(+0xf314)[0x7f8d26fd7314]
/lib64/libdl.so.2(+0x162d)[0x7f8d25e6162d]
/lib64/libdl.so.2(dlopen+0x31)[0x7f8d25e610c1]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(+0x81ec4)[0x7f8d260e5ec4]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(+0x8209c)[0x7f8d260e609c]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(opal_dl_open+0x6e)[0x7f8d260e5bfa]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(+0x59265)[0x7f8d260bd265]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(+0x58d45)[0x7f8d260bcd45]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(mca_base_component_find+0x1d2)[0x7f8d260bc444]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(mca_base_framework_components_register+0x83)[0x7f8d260ce57a]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(mca_base_framework_register+0x217)[0x7f8d260ceedd]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(mca_base_framework_open+0x46)[0x7f8d260cef4b]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/openmpi/mca_pml_cm.so(+0x5bf5)[0x7f8d222b7bf5]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(+0x5be58)[0x7f8d260bfe58]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(mca_base_framework_components_open+0x69)[0x7f8d260bfb4b]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libmpi.so.12(+0x106fa7)[0x7f8d26d2bfa7]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(mca_base_framework_open+0xe1)[0x7f8d260cefe6]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libmpi.so.12(ompi_mpi_init+0x5ab)[0x7f8d26c7d755]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libmpi.so.12(MPI_Init+0x180)[0x7f8d26cbd0b6]
examples/ring_c[0x400969]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f8d26669af5]
examples/ring_c[0x400879]
ring_c:19091 terminated with signal 11 at PC=7f27c44d93e8 SP=7fff6c152098.
Backtrace:
/lib64/libnl-route-3.so.200(rtnl_tc_unregister+0x8)[0x7f27c44d93e8]
/lib64/ld-linux-x86-64.so.2(+0xfb7a)[0x7f27cac05b7a]
/lib64/libc.so.6(+0x38e79)[0x7f27ca2aee79]
/lib64/libc.so.6(+0x38ec5)[0x7f27ca2aeec5]
/lib64/libinfinipath.so.4(+0x3fb4)[0x7f27c5205fb4]
/lib64/libpthread.so.0(+0xf130)[0x7f27ca646130]
/lib64/libc.so.6(gsignal+0x39)[0x7f27ca2ab5e9]
/lib64/libc.so.6(abort+0x148)[0x7f27ca2accf8]
/lib64/libc.so.6(+0x2e556)[0x7f27ca2a4556]
/lib64/libc.so.6(+0x2e602)[0x7f27ca2a4602]
/lib64/libnl-route-3.so.200(+0x213d9)[0x7f27c44d93d9]
/lib64/ld-linux-x86-64.so.2(+0xf503)[0x7f27cac05503]
/lib64/ld-linux-x86-64.so.2(+0x13b44)[0x7f27cac09b44]
/lib64/ld-linux-x86-64.so.2(+0xf314)[0x7f27cac05314]
/lib64/ld-linux-x86-64.so.2(+0x1325b)[0x7f27cac0925b]
/lib64/libdl.so.2(+0x102b)[0x7f27c9a8f02b]
/lib64/ld-linux-x86-64.so.2(+0xf314)[0x7f27cac05314]
/lib64/libdl.so.2(+0x162d)[0x7f27c9a8f62d]
/lib64/libdl.so.2(dlopen+0x31)[0x7f27c9a8f0c1]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(+0x81ec4)[0x7f27c9d13ec4]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(+0x8209c)[0x7f27c9d1409c]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(opal_dl_open+0x6e)[0x7f27c9d13bfa]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(+0x59265)[0x7f27c9ceb265]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(+0x58d45)[0x7f27c9cead45]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(mca_base_component_find+0x1d2)[0x7f27c9cea444]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(mca_base_framework_components_register+0x83)[0x7f27c9cfc57a]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(mca_base_framework_register+0x217)[0x7f27c9cfcedd]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(mca_base_framework_open+0x46)[0x7f27c9cfcf4b]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/openmpi/mca_pml_cm.so(+0x5bf5)[0x7f27c5ee5bf5]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(+0x5be58)[0x7f27c9cede58]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(mca_base_framework_components_open+0x69)[0x7f27c9cedb4b]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libmpi.so.12(+0x106fa7)[0x7f27ca959fa7]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(mca_base_framework_open+0xe1)[0x7f27c9cfcfe6]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libmpi.so.12(ompi_mpi_init+0x5ab)[0x7f27ca8ab755]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libmpi.so.12(MPI_Init+0x180)[0x7f27ca8eb0b6]
examples/ring_c[0x400969]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f27ca297af5]
examples/ring_c[0x400879]
ring_c:19092 terminated with signal 11 at PC=7f8d208ab3e8 SP=7ffe1f0966d8.
Backtrace:
/lib64/libnl-route-3.so.200(rtnl_tc_unregister+0x8)[0x7f8d208ab3e8]
/lib64/ld-linux-x86-64.so.2(+0xfb7a)[0x7f8d26fd7b7a]
/lib64/libc.so.6(+0x38e79)[0x7f8d26680e79]
/lib64/libc.so.6(+0x38ec5)[0x7f8d26680ec5]
/lib64/libinfinipath.so.4(+0x3fb4)[0x7f8d215d7fb4]
/lib64/libpthread.so.0(+0xf130)[0x7f8d26a18130]
/lib64/libc.so.6(gsignal+0x39)[0x7f8d2667d5e9]
/lib64/libc.so.6(abort+0x148)[0x7f8d2667ecf8]
/lib64/libc.so.6(+0x2e556)[0x7f8d26676556]
/lib64/libc.so.6(+0x2e602)[0x7f8d26676602]
/lib64/libnl-route-3.so.200(+0x213d9)[0x7f8d208ab3d9]
/lib64/ld-linux-x86-64.so.2(+0xf503)[0x7f8d26fd7503]
/lib64/ld-linux-x86-64.so.2(+0x13b44)[0x7f8d26fdbb44]
/lib64/ld-linux-x86-64.so.2(+0xf314)[0x7f8d26fd7314]
/lib64/ld-linux-x86-64.so.2(+0x1325b)[0x7f8d26fdb25b]
/lib64/libdl.so.2(+0x102b)[0x7f8d25e6102b]
/lib64/ld-linux-x86-64.so.2(+0xf314)[0x7f8d26fd7314]
/lib64/libdl.so.2(+0x162d)[0x7f8d25e6162d]
/lib64/libdl.so.2(dlopen+0x31)[0x7f8d25e610c1]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(+0x81ec4)[0x7f8d260e5ec4]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(+0x8209c)[0x7f8d260e609c]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(opal_dl_open+0x6e)[0x7f8d260e5bfa]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(+0x59265)[0x7f8d260bd265]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(+0x58d45)[0x7f8d260bcd45]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(mca_base_component_find+0x1d2)[0x7f8d260bc444]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(mca_base_framework_components_register+0x83)[0x7f8d260ce57a]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(mca_base_framework_register+0x217)[0x7f8d260ceedd]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(mca_base_framework_open+0x46)[0x7f8d260cef4b]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/openmpi/mca_pml_cm.so(+0x5bf5)[0x7f8d222b7bf5]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(+0x5be58)[0x7f8d260bfe58]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(mca_base_framework_components_open+0x69)[0x7f8d260bfb4b]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libmpi.so.12(+0x106fa7)[0x7f8d26d2bfa7]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libopen-pal.so.13(mca_base_framework_open+0xe1)[0x7f8d260cefe6]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libmpi.so.12(ompi_mpi_init+0x5ab)[0x7f8d26c7d755]
/home/phargrov/OMPI/openmpi-1.10.1rc1-linux-x86_64-sl7x/INST/lib/libmpi.so.12(MPI_Init+0x180)[0x7f8d26cbd0b6]
examples/ring_c[0x400969]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f8d26669af5]
examples/ring_c[0x400879]
-------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code.. Per user-direction, the job has been aborted.
-------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus
causing
the job to be terminated. The first process to do so was:
Process name: [[2387,1],0]
Exit code: 1
--------------------------------------------------------------------------
[sl7x.localdomain:19088] mca: base: close: component dlopen closed
[sl7x.localdomain:19088] mca: base: close: unloading component dlopen
