I've grabbed last nights tarball (1.2b3r12981) and tried using the
shared mem transport on btl and mx,self on mtl, same results. What I
don't get is that, sometimes it works, and sometimes it doesn't (for
either). For example, I can run it 10 times successfully then incr the
-np from 7 to 10 across 3 nodes, and it'll immediately fail.
Here's an example of one run right after another.
$ mpirun --prefix /usr/local/openmpi-1.2b3r12981/ -x
LD_LIBRARY_PATH=${LD_LIBRARY_PATH} --hostfile ./h25-27 -np 10 --mca mtl
mx,self ./cpi
Process 0 of 10 is on node-25
Process 4 of 10 is on node-26
Process 1 of 10 is on node-25
Process 5 of 10 is on node-26
Process 2 of 10 is on node-25
Process 8 of 10 is on node-27
Process 6 of 10 is on node-26
Process 9 of 10 is on node-27
Process 7 of 10 is on node-26
Process 3 of 10 is on node-25
pi is approximately 3.1415926544231256, Error is 0.0000000008333325
wall clock time = 0.017513
$ mpirun --prefix /usr/local/openmpi-1.2b3r12981/ -x
LD_LIBRARY_PATH=${LD_LIBRARY_PATH} --hostfile ./h25-27 -np 10 --mca mtl
mx,self ./cpi
Signal:11 info.si_errno:0(Success) si_code:1(SEGV_MAPERR)
Failing at addr:(nil)
[0]
func:/usr/local/openmpi-1.2b3r12981/lib/libopen-pal.so.0(opal_backtrace_
print+0x1f) [0x2b8ddf3ccd3f]
[1] func:/usr/local/openmpi-1.2b3r12981/lib/libopen-pal.so.0
[0x2b8ddf3cb891]
[2] func:/lib/libpthread.so.0 [0x2b8ddf98f6c0]
[3] func:/opt/mx/lib/libmyriexpress.so(mx_open_endpoint+0x6df)
[0x2b8de25bf2af]
[4]
func:/usr/local/openmpi-1.2b3r12981/lib/openmpi/mca_btl_mx.so(mca_btl_mx
_component_init+0x5d7) [0x2b8de27dcd27]
[5]
func:/usr/local/openmpi-1.2b3r12981/lib/libmpi.so.0(mca_btl_base_select+
0x156) [0x2b8ddf125b46]
[6]
func:/usr/local/openmpi-1.2b3r12981/lib/openmpi/mca_bml_r2.so(mca_bml_r2
_component_init+0x11) [0x2b8de26d7491]
[7]
func:/usr/local/openmpi-1.2b3r12981/lib/libmpi.so.0(mca_bml_base_init+0x
7d) [0x2b8ddf12543d]
[8]
func:/usr/local/openmpi-1.2b3r12981/lib/openmpi/mca_pml_ob1.so(mca_pml_o
b1_component_init+0x6b) [0x2b8de23a4f8b]
[9]
func:/usr/local/openmpi-1.2b3r12981/lib/libmpi.so.0(mca_pml_base_select+
0x113) [0x2b8ddf12cea3]
[10]
func:/usr/local/openmpi-1.2b3r12981/lib/libmpi.so.0(ompi_mpi_init+0x45a)
[0x2b8ddf0f5bda]
[11] func:/usr/local/openmpi-1.2b3r12981/lib/libmpi.so.0(MPI_Init+0x83)
[0x2b8ddf116af3]
[12] func:./cpi(main+0x42) [0x400cd5]
[13] func:/lib/libc.so.6(__libc_start_main+0xe3) [0x2b8ddfab50e3]
[14] func:./cpi [0x400bd9]
*** End of error message ***
mpirun noticed that job rank 0 with PID 0 on node node-25 exited on
signal 11.
9 additional processes aborted (not shown)
-----Original Message-----
From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] On
Behalf Of Brian W. Barrett
Sent: Tuesday, January 02, 2007 4:11 PM
To: Open MPI Users
Subject: Re: [OMPI users] Ompi failing on mx only
Sorry to jump into the discussion late. The mx btl does not support
communication between processes on the same node by itself, so you have
to include the shared memory transport when using MX. This will
eventually be fixed, but likely not for the 1.2 release. So if you do:
mpirun --prefix /usr/local/openmpi-1.2b2 -x LD_LIBRARY_PATH --
hostfile ./h1-3 -np 2 --mca btl mx,sm,self ./cpi
It should work much better. As for the MTL, there is a bug in the MX
MTL for v1.2 that has been fixed, but after 1.2b2 that could cause the
random failures you were seeing. It will work much better after
1.2b3 is released (or if you are feeling really lucky, you can try out
the 1.2 nightly tarballs).
The MTL is a new feature in v1.2. It is a different communication
abstraction designed to support interconnects that have matching
implemented in the lower level library or in hardware (Myrinet/MX,
Portals, InfiniPath are currently implemented). The MTL allows us to
exploit the low latency and asynchronous progress these libraries can
provide, but does mean multi-nic abilities are reduced. Further, the
MTL is not well suited to interconnects like TCP or InfiniBand, so we
will continue supporting the BTL interface as well.
Brian
On Jan 2, 2007, at 2:44 PM, Grobe, Gary L. ((JSC-EV))[ESCG] wrote:
> About the -x, I've been trying it both ways and prefer the latter, and
> results for either are the same. But it's value is correct.
> I've attached the ompi_info from node-1 and node-2. Sorry for not
> zipping them, but they were small and I think I'd have firewall
> issues.
>
> $ mpirun --prefix /usr/local/openmpi-1.2b2 -x LD_LIBRARY_PATH --
> hostfile ./h13-15 -np 6 --mca pml cm ./cpi [node-14:19260] mx_connect
> fail for node-14:0 with key aaaaffff (error Endpoint closed or not
> connectable!) [node-14:19261] mx_connect fail for node-14:0 with key
> aaaaffff (error Endpoint closed or not connectable!) ...
>
> Is there any info anywhere's on MTL? Anyways, I've run w/ mtl, and
> sometimes it actually worked once. But now I can't reproduce it and
> it's throwing sig 7's, 11's, and 4's depending upon the number of
> procs I give it. But now that you mention mapper, I take it that's
> what SEGV_MAPERR might be referring to. I'm looking into the
>
> $ mpirun --prefix /usr/local/openmpi-1.2b2 -x LD_LIBRARY_PATH=$
> {LD_LIBRARY_PATH} --hostfile ./h1-3 -np 5 --mca mtl mx,self ./cpi
> Process 4 of 5 is on node-2 Process 0 of 5 is on node-1 Process 1 of 5
> is on node-1 Process 2 of 5 is on node-1 Process 3 of 5 is on node-1
> pi is approximately 3.1415926544231225, Error is 0.0000000008333294
> wall clock time = 0.019305
> Signal:11 info.si_errno:0(Success) si_code:1(SEGV_MAPERR) Failing at
> addr:0x2b88243862be mpirun noticed that job rank 0 with PID 0 on node
> node-1 exited on signal 1.
> 4 additional processes aborted (not shown) Or sometimes I'll get this
> error, just depending upon the number of procs ...
>
> mpirun --prefix /usr/local/openmpi-1.2b2 -x LD_LIBRARY_PATH=$
> {LD_LIBRARY_PATH} --hostfile ./h1-3 -np 7 --mca mtl mx,self ./cpi
> Signal:7 info.si_errno:0(Success) si_code:2() Failing at
> addr:0x2aaaaaaab000 [0]
> func:/usr/local/openmpi-1.2b2/lib/libopen-pal.so.0
> (opal_backtrace_print+0x1f) [0x2b9b7fa52d1f] [1]
> func:/usr/local/openmpi-1.2b2/lib/libopen-pal.so.0
> [0x2b9b7fa51871]
> [2] func:/lib/libpthread.so.0 [0x2b9b80013d00] [3]
> func:/usr/local/openmpi-1.2b2/lib/libmca_common_sm.so.0
> (mca_common_sm_mmap_init+0x1e3) [0x2b9b8270ef83] [4]
> func:/usr/local/openmpi-1.2b2/lib/openmpi/mca_mpool_sm.so
> [0x2b9b8260d0ff]
> [5] func:/usr/local/openmpi-1.2b2/lib/libmpi.so.0
> (mca_mpool_base_module_create+0x70) [0x2b9b7f7afac0] [6]
> func:/usr/local/openmpi-1.2b2/lib/openmpi/mca_btl_sm.so
> (mca_btl_sm_add_procs_same_base_addr+0x907) [0x2b9b83070517] [7]
> func:/usr/local/openmpi-1.2b2/lib/openmpi/mca_bml_r2.so
> (mca_bml_r2_add_procs+0x206) [0x2b9b82d5f576] [8]
> func:/usr/local/openmpi-1.2b2/lib/openmpi/mca_pml_ob1.so
> (mca_pml_ob1_add_procs+0xe3) [0x2b9b82a2d0a3] [9]
> func:/usr/local/openmpi-1.2b2/lib/libmpi.so.0(ompi_mpi_init
> +0x697) [0x2b9b7f77be07]
> [10] func:/usr/local/openmpi-1.2b2/lib/libmpi.so.0(MPI_Init+0x83)
> [0x2b9b7f79c943]
> [11] func:./cpi(main+0x42) [0x400cd5]
> [12] func:/lib/libc.so.6(__libc_start_main+0xf4) [0x2b9b8013a134] [13]
> func:./cpi [0x400bd9]
> *** End of error message ***
> Process 4 of 7 is on node-2
> Process 5 of 7 is on node-2
> Process 6 of 7 is on node-2
> Process 0 of 7 is on node-1
> Process 1 of 7 is on node-1
> Process 2 of 7 is on node-1
> Process 3 of 7 is on node-1
> pi is approximately 3.1415926544231239, Error is 0.0000000008333307
> wall clock time = 0.009331
> Signal:11 info.si_errno:0(Success) si_code:1(SEGV_MAPERR) Failing at
> addr:0x2b4ba33652be
> Signal:11 info.si_errno:0(Success) si_code:1(SEGV_MAPERR) Failing at
> addr:0x2b8685aba2be
> Signal:11 info.si_errno:0(Success) si_code:1(SEGV_MAPERR) Failing at
> addr:0x2b304ffbe2be mpirun noticed that job rank 0 with PID 0 on node
> node-1 exited on signal 1.
> 6 additional processes aborted (not shown)
>
> Ok, so I take it one is down. Would this be the cause for all the
> different errors I'm seeing?
>
> $ fm_status
> FMS Fabric status
>
> 17 hosts known
> 16 FMAs found
> 3 un-ACKed alerts
> Mapping is complete, last map generated by node-20 Database generation
> not yet complete.
>
>
> From: users-boun...@open-mpi.org [mailto:users-bounces@open- mpi.org]
> On Behalf Of Reese Faucette
> Sent: Tuesday, January 02, 2007 2:52 PM
> To: Open MPI Users
> Subject: Re: [OMPI users] Ompi failing on mx only
>
> Hi, Gary-
> This looks like a config problem, and not a code problem yet.
> Could you send the output of mx_info from node-1 and from node-2?
> Also, forgive me counter-asking a possibly dumb OMPI question, but is
> "-x LD_LIBRARY_PATH" really what you want, as opposed to "-x
> LD_LIBRARY_PATH=${LD_LIBRARY_PATH}" ? (I would not be surprised if
> not specifying a value defaults to this behavior, but have to ask).
>
> Also, have you tried MX MTL as opposed to BTL? --mca pml cm --mca mtl
> mx,self (it looks like you did)
>
> "[node-2:10464] mx_connect fail for node-2:0 with key aaaaffff "
> makes it look like your fabric may not be fully mapped or that you may
> have a down link.
>
> thanks,
> -reese
> Myricom, Inc.
>
> I was initially using 1.1.2 and moved to 1.2b2 because of a hang on
> MPI_Bcast() which 1.2b2 reports to fix, and seemed to have done so.
> My compute nodes are 2 dual core xeons on myrinet with mx. The problem
> is trying to get ompi running on mx only. My machine file is as
> follows ...
>
> node-1 slots=4 max-slots=4
> node-2 slots=4 max-slots=4
> node-3 slots=4 max-slots=4
>
> 'mpirun' with the minimum number of processes in order to get the
> error ...
> mpirun --prefix /usr/local/openmpi-1.2b2 -x LD_LIBRARY_PATH
> --hostfile ./h1-3 -np 2 --mca btl mx,self ./cpi
>
> I don't believe there'a anything wrong w/ the hardware as I can ping
> on mx between this failed node and the master fine. So I tried a
> different set of 3 nodes and I got the same error, it always fails on
> the 2nd node of any group of nodes I choose.
>
> <node-2.out>
> <node-1.out>
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
--
Brian Barrett
Open MPI Team, CCS-1
Los Alamos National Laboratory
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
