On Jan 28, 2013, at 11:55 AM, Steve Wise <sw...@opengridcomputing.com> wrote:
> Do you know if the rdmacm CPC is really being used for your connection setup > (vs other CPCs supported by IB)? Cuz iwarp only supports rdmacm. Maybe > that's the difference? Dunno for certain, but I expect it is using the OOB cm since I didn't direct it to do anything different. Like I said, I suspect the problem is that the cluster doesn't have iWARP on it. > > Steve. > > On 1/28/2013 1:47 PM, Ralph Castain wrote: >> Nope - still works just fine. I didn't receive that warning at all, and it >> ran to completion without problem. >> >> I suspect the problem is that the system I can use just isn't configured >> like yours, and so I can't trigger the problem. Afraid I can't be of help >> after all... :-( >> >> >> On Jan 28, 2013, at 11:25 AM, Steve Wise <sw...@opengridcomputing.com> wrote: >> >>> On 1/28/2013 12:48 PM, Ralph Castain wrote: >>>> Hmmm...afraid I cannot replicate this using the current state of the 1.6 >>>> branch (which is the 1.6.4rcN) on the only IB-based cluster I can access. >>>> >>>> Can you try it with a 1.6.4 tarball and see if you still see the problem? >>>> Could be someone already fixed it. >>> I still hit it on 1.6.4rc2. >>> >>> Note iWARP != IB so you may not have this issue on IB systems for various >>> reasons. Did you use the same mpirun line? Namely using this: >>> >>> --mca btl_openib_ipaddr_include "192.168.170.0/24" >>> >>> (adjusted to your network config). >>> >>> Because if I don't use ipaddr_include, then I don't see this issue on my >>> setup. >>> >>> Also, did you see these logged: >>> >>> Right after starting the job: >>> >>> -------------------------------------------------------------------------- >>> No OpenFabrics connection schemes reported that they were able to be >>> used on a specific port. As such, the openib BTL (OpenFabrics >>> support) will be disabled for this port. >>> >>> Local host: hpc-hn1.ogc.int >>> Local device: cxgb4_0 >>> Local port: 2 >>> CPCs attempted: oob, xoob, rdmacm >>> -------------------------------------------------------------------------- >>> ... >>> >>> At the end of the job: >>> >>> [hpc-hn1.ogc.int:07850] 5 more processes have sent help message >>> help-mpi-btl-openib-cpc-base.txt / no cpcs for port >>> >>> >>> I think these are benign, but prolly indicate a bug: the mpirun is >>> restricting the job to use port 1 only, so the CPCs shouldn't be attempting >>> port 2... >>> >>> Steve. >>> >>> >>>> On Jan 28, 2013, at 10:03 AM, Steve Wise <sw...@opengridcomputing.com> >>>> wrote: >>>> >>>>> On 1/28/2013 11:48 AM, Ralph Castain wrote: >>>>>> On Jan 28, 2013, at 9:12 AM, Steve Wise <sw...@opengridcomputing.com> >>>>>> wrote: >>>>>> >>>>>>> On 1/25/2013 12:19 PM, Steve Wise wrote: >>>>>>>> Hello, >>>>>>>> >>>>>>>> I'm tracking an issue I see in openmpi-1.6.3. Running this command on >>>>>>>> my chelsio iwarp/rdma setup causes a seg fault every time: >>>>>>>> >>>>>>>> /usr/mpi/gcc/openmpi-1.6.3-dbg/bin/mpirun --np 2 --host >>>>>>>> hpc-hn1,hpc-cn2 --mca btl openib,sm,self --mca >>>>>>>> btl_openib_ipaddr_include "192.168.170.0/24" >>>>>>>> /usr/mpi/gcc/openmpi-1.6.3/tests/IMB-3.2/IMB-MPI1 pingpong >>>>>>>> >>>>>>>> The segfault is during finalization, and I've debugged this to the >>>>>>>> point were I see a call to dereg_mem() after the openib blt is >>>>>>>> unloaded via dlclose(). dereg_mem() dereferences a function pointer >>>>>>>> to call the btl-specific dereg function, in this case it is >>>>>>>> openib_dereg_mr(). However, since that btl has already been unloaded, >>>>>>>> the deref causes a seg fault. Happens every time with the above mpi >>>>>>>> job. >>>>>>>> >>>>>>>> Now, I tried this same experiment with openmpi-1.7rc6 and I don't see >>>>>>>> the seg fault, and I don't see a call to dereg_mem() after the openib >>>>>>>> btl is unloaded. That's all well good. :) But I'd like to get this >>>>>>>> fix pushed into 1.6 since that is the current stable release. >>>>>>>> >>>>>>>> Question: Can someone point me to the fix in 1.7? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Steve. >>>>>>> It appears that in ompi_mpi_finalize(), mca_pml_base_close() is called >>>>>>> which unloads the openib btl. Then further down in >>>>>>> ompi_mpi_finalize(), mca_mpool_base_close() is called which ends up >>>>>>> calling dereg_mem() which seg faults trying to call into the unloaded >>>>>>> openib btl. >>>>>>> >>>>>> That definitely sounds like a bug >>>>>> >>>>>>> Anybody have thoughts? Anybody care? :) >>>>>> I care! It needs to be fixed - I'll take a look. Probably something that >>>>>> forgot to be cmr'd. >>>>> Great! If you want me to try out a fix or gather more debug, just hollar. >>>>> >>>>> Thanks, >>>>> >>>>> Steve. >>>>> >
