Tests good on 1.6.3 too.
Thanks Josh!
On 1/29/2013 9:17 AM, Steve Wise wrote:
I applied it to 1.6.4rc2 and it fixed the seg fault issue. Lemme try
1.6.3 too.
On 1/29/2013 9:11 AM, Joshua Ladd wrote:
It should apply cleanly to 1.6.3 branch, I tested it this morning.
From top level OMPI directory do
patch -p1 < yossi_openib.patch
Best,
Josh
Josh
-----Original Message-----
From: devel-boun...@open-mpi.org [mailto:devel-boun...@open-mpi.org]
On Behalf Of Jeff Squyres (jsquyres)
Sent: Tuesday, January 29, 2013 10:05 AM
To: Steve Wise
Cc: Open MPI Developers
Subject: Re: [OMPI devel] openib unloaded before last mem dereg
It's on the ticket that I just assigned to you. :-)
On Jan 29, 2013, at 10:03 AM, Steve Wise
<sw...@opengridcomputing.com> wrote:
Will do...once I get a patch.
STeve
On 1/29/2013 7:40 AM, Jeff Squyres (jsquyres) wrote:
Thanks Josh.
Steve -- if you can confirm that this fixes your problem in the
v1.6 series, we'll go ahead and commit the patch.
FWIW: the OpenFabrics startup code got a little cleanup/revamp on
the trunk/v1.7 -- I suspect that's why you're not seeing the
problem on trunk/v1.7 (e.g., look at the utility routines that were
abstracted out to ompi/mca/common/verbs).
On Jan 29, 2013, at 2:41 AM, Joshua Ladd <josh...@mellanox.com> wrote:
So, we (Mellanox) have observed this ourselves when no suitable
CPC can be found. Seems the BTL associated with this port is not
destroyed and the ref count is not decreased. Not sure why you
don't see the problem in 1.7. But we have a patch that I'll CMR
today. Please review our symptoms, diagnosis, and proposed change.
Ralph, maybe I can list you as a reviewer of the patch? I've
reviewed myself and it looks fine, but wouldn't mind having
another set of eyes on it since I don't want to be responsible for
breaking the OpenIB BTL.
Thanks,
Josh Ladd
Reported by Yossi:
Hi,
There is a bug in open mpi (openib component) when one of the
active ports is Ethernet.
The fix is attached, probably needs to be reviewed and submitted to
ompi
Error flow:
1. Openib component creates a btl instance for every active
port (including Ethernet)
2. Every btl holds a reference count to the device
(mca_btl_openib_device_t::btls)
3. Openib tries to create a "connection module" for every btl
4. It fails to create connection module for the Ethernet port
5. The btl for Ethernet port is not returned by openib
component, in the list of btl modules
6. The btl for Ethernet port is not destroyed during openib
component finalize
7. The device is not destroyed, because of the reference count
8. The memory pool created by the device is not destroyed
9. Later, rdma mpool module cleans up remaining pools during
its finalize
10. The memory pool created by openib is destroyed by rdma
mpool component finalize
11. The memory pool points to a function (openib_dereg_mr)
which is already unloaded from memory (because mca_btl_openib.so
was unloaded)
12. Segfault because of a call to invalid function
The fix: If a btl module is not going to be returned from openib
component init, destroy it.
-----Original Message-----
From: devel-boun...@open-mpi.org [mailto:devel-boun...@open-mpi.org]
On Behalf Of Ralph Castain
Sent: Monday, January 28, 2013 8:35 PM
To: Steve Wise
Cc: Open MPI Developers
Subject: Re: [OMPI devel] openib unloaded before last mem dereg
Out of curiosity, could you tell us how you configured OMPI?
On Jan 28, 2013, at 12:46 PM, Steve Wise
<sw...@opengridcomputing.com> wrote:
On 1/28/2013 2:04 PM, Ralph Castain wrote:
On Jan 28, 2013, at 11:55 AM, Steve Wise
<sw...@opengridcomputing.com> wrote:
Do you know if the rdmacm CPC is really being used for your
connection setup (vs other CPCs supported by IB)? Cuz iwarp
only supports rdmacm. Maybe that's the difference?
Dunno for certain, but I expect it is using the OOB cm since I
didn't direct it to do anything different. Like I said, I
suspect the problem is that the cluster doesn't have iWARP on it.
Definitely, or it could be the different CPC used for IWvs IB is
tickling the issue.
Steve.
On 1/28/2013 1:47 PM, Ralph Castain wrote:
Nope - still works just fine. I didn't receive that warning at
all, and it ran to completion without problem.
I suspect the problem is that the system I can use just isn't
configured like yours, and so I can't trigger the problem.
Afraid I can't be of help after all... :-(
On Jan 28, 2013, at 11:25 AM, Steve Wise
<sw...@opengridcomputing.com> wrote:
On 1/28/2013 12:48 PM, Ralph Castain wrote:
Hmmm...afraid I cannot replicate this using the current
state of the 1.6 branch (which is the 1.6.4rcN) on the only
IB-based cluster I can access.
Can you try it with a 1.6.4 tarball and see if you still see
the problem? Could be someone already fixed it.
I still hit it on 1.6.4rc2.
Note iWARP != IB so you may not have this issue on IB systems
for various reasons. Did you use the same mpirun line?
Namely using this:
--mca btl_openib_ipaddr_include "192.168.170.0/24"
(adjusted to your network config).
Because if I don't use ipaddr_include, then I don't see this
issue on my setup.
Also, did you see these logged:
Right after starting the job:
---------------------------------------------------------------
---
-------- No OpenFabrics connection schemes reported that they
were able to be used on a specific port. As such, the openib
BTL (OpenFabrics
support) will be disabled for this port.
Local host: hpc-hn1.ogc.int
Local device: cxgb4_0
Local port: 2
CPCs attempted: oob, xoob, rdmacm
---------------------------------------------------------------
---
--------
...
At the end of the job:
[hpc-hn1.ogc.int:07850] 5 more processes have sent help message
help-mpi-btl-openib-cpc-base.txt / no cpcs for port
I think these are benign, but prolly indicate a bug: the
mpirun is restricting the job to use port 1 only, so the CPCs
shouldn't be attempting port 2...
Steve.
On Jan 28, 2013, at 10:03 AM, Steve Wise
<sw...@opengridcomputing.com> wrote:
On 1/28/2013 11:48 AM, Ralph Castain wrote:
On Jan 28, 2013, at 9:12 AM, Steve Wise
<sw...@opengridcomputing.com> wrote:
On 1/25/2013 12:19 PM, Steve Wise wrote:
Hello,
I'm tracking an issue I see in openmpi-1.6.3. Running
this command on my chelsio iwarp/rdma setup causes a seg
fault every time:
/usr/mpi/gcc/openmpi-1.6.3-dbg/bin/mpirun --np 2 --host
hpc-hn1,hpc-cn2 --mca btl openib,sm,self --mca
btl_openib_ipaddr_include "192.168.170.0/24"
/usr/mpi/gcc/openmpi-1.6.3/tests/IMB-3.2/IMB-MPI1 pingpong
The segfault is during finalization, and I've debugged
this to the point were I see a call to dereg_mem() after
the openib blt is unloaded via dlclose(). dereg_mem()
dereferences a function pointer to call the btl-specific
dereg function, in this case it is openib_dereg_mr().
However, since that btl has already been unloaded, the
deref causes a seg fault. Happens every time with the
above mpi job.
Now, I tried this same experiment with openmpi-1.7rc6
and I don't see the seg fault, and I don't see a call to
dereg_mem() after the openib btl is unloaded. That's
all well good. :) But I'd like to get this fix pushed
into 1.6 since that is the current stable release.
Question: Can someone point me to the fix in 1.7?
Thanks,
Steve.
It appears that in ompi_mpi_finalize(),
mca_pml_base_close() is called which unloads the openib
btl. Then further down in ompi_mpi_finalize(),
mca_mpool_base_close() is called which ends up calling
dereg_mem() which seg faults trying to call into the
unloaded openib btl.
That definitely sounds like a bug
Anybody have thoughts? Anybody care? :)
I care! It needs to be fixed - I'll take a look. Probably
something that forgot to be cmr'd.
Great! If you want me to try out a fix or gather more
debug, just hollar.
Thanks,
Steve.
_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
--
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/
_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
