Re: [OMPI users] locked memory and queue pairs

2016-03-17 Thread Michael Di Domenico
On Thu, Mar 17, 2016 at 12:15 PM, Cabral, Matias A wrote: > I was looking for lines like" [nodexyz:17085] selected cm best priority 40" > and " [nodexyz:17099] select: component psm selected" this may have turned up more then i expected. i recompiled openmpi v1.8.4

Re: [OMPI users] locked memory and queue pairs

2016-03-17 Thread Michael Di Domenico
On Thu, Mar 17, 2016 at 12:52 PM, Jeff Squyres (jsquyres) wrote: > Can you send all the information listed here? > > https://www.open-mpi.org/community/help/ > > (including the full output from the run with the PML/BTL/MTL/etc. verbosity) > > This will allow Matias to look

Re: [OMPI users] locked memory and queue pairs

2016-03-17 Thread Jeff Squyres (jsquyres)
Michael -- Can you send all the information listed here? https://www.open-mpi.org/community/help/ (including the full output from the run with the PML/BTL/MTL/etc. verbosity) This will allow Matias to look through all the relevant info, potentially with fewer back-n-forth emails. Thanks!

Re: [OMPI users] locked memory and queue pairs

2016-03-17 Thread Michael Di Domenico
On Thu, Mar 17, 2016 at 12:15 PM, Cabral, Matias A wrote: > I was looking for lines like" [nodexyz:17085] selected cm best priority 40" > and " [nodexyz:17099] select: component psm selected" i see cm best priority 20, which seems to relate to ob1 being selected. i

Re: [OMPI users] locked memory and queue pairs

2016-03-17 Thread Cabral, Matias A
016 5:52 AM To: Open MPI Users <us...@open-mpi.org> Subject: Re: [OMPI users] locked memory and queue pairs On Wed, Mar 16, 2016 at 4:49 PM, Cabral, Matias A <matias.a.cab...@intel.com> wrote: > I didn't go into the code to see who is actually calling this error message, > but

Re: [OMPI users] locked memory and queue pairs

2016-03-17 Thread Gilles Gouaillardet
also, limits.conf is set when starting a ssh session. it is not useful for services started at boot time, and ulimit -l unlimited should be added in the startup script /etc/init.d/xxx or /etc/sysconfig/xxx Cheers, Gilles On Thursday, March 17, 2016, Dave Love wrote: >

Re: [OMPI users] locked memory and queue pairs

2016-03-17 Thread Dave Love
Michael Di Domenico writes: > On Wed, Mar 16, 2016 at 12:12 PM, Elken, Tom wrote: >> Hi Mike, >> >> In this file, >> $ cat /etc/security/limits.conf >> ... >> < do you see at the end ... > >> >> * hard memlock unlimited >> * soft memlock unlimited >>

Re: [OMPI users] locked memory and queue pairs

2016-03-17 Thread Michael Di Domenico
On Wed, Mar 16, 2016 at 4:49 PM, Cabral, Matias A wrote: > I didn't go into the code to see who is actually calling this error message, > but I suspect this may be a generic error for "out of memory" kind of thing > and not specific to the que pair. To confirm please

Re: [OMPI users] locked memory and queue pairs

2016-03-16 Thread Cabral, Matias A
rom: users [mailto:users-boun...@open-mpi.org] On Behalf Of Michael Di Domenico Sent: Wednesday, March 16, 2016 1:25 PM To: Open MPI Users <us...@open-mpi.org> Subject: Re: [OMPI users] locked memory and queue pairs On Wed, Mar 16, 2016 at 3:37 PM, Cabral, Matias A <matias.a.cab...@intel.com>

Re: [OMPI users] locked memory and queue pairs

2016-03-16 Thread Michael Di Domenico
On Wed, Mar 16, 2016 at 3:37 PM, Cabral, Matias A wrote: > Hi Michael, > > I may be missing some context, if you are using the qlogic cards you will > always want to use the psm mtl (-mca pml cm -mca mtl psm) and not openib btl. > As Tom suggest, confirm the limits

Re: [OMPI users] locked memory and queue pairs

2016-03-16 Thread Cabral, Matias A
thers" are not? Please share the command line and the error message. Thanks, _MAC >> Begin forwarded message: >> >> From: Michael Di Domenico <mdidomeni...@gmail.com> >> Subject: Re: [OMPI users] locked memory and queue pairs >> Date: March 16, 2016 at 11:32:01 AM

Re: [OMPI users] locked memory and queue pairs

2016-03-16 Thread Michael Di Domenico
On Wed, Mar 16, 2016 at 12:12 PM, Elken, Tom wrote: > Hi Mike, > > In this file, > $ cat /etc/security/limits.conf > ... > < do you see at the end ... > > > * hard memlock unlimited > * soft memlock unlimited > # -- All InfiniBand Settings End here -- > ? Yes. I double

Re: [OMPI users] locked memory and queue pairs

2016-03-16 Thread Elken, Tom
f Michael Di > Domenico > Sent: Thursday, March 10, 2016 8:55 AM > To: Open MPI Users > Subject: [OMPI users] locked memory and queue pairs > > when i try to run an openmpi job with >128 ranks (16 ranks per node) > using alltoall or alltoallv, i'm getting an error that the proc

Re: [OMPI users] locked memory and queue pairs

2016-03-16 Thread Michael Di Domenico
On Thu, Mar 10, 2016 at 11:54 AM, Michael Di Domenico wrote: > when i try to run an openmpi job with >128 ranks (16 ranks per node) > using alltoall or alltoallv, i'm getting an error that the process was > unable to get a queue pair. > > i've checked the max locked memory

[OMPI users] locked memory and queue pairs

2016-03-10 Thread Michael Di Domenico
when i try to run an openmpi job with >128 ranks (16 ranks per node) using alltoall or alltoallv, i'm getting an error that the process was unable to get a queue pair. i've checked the max locked memory settings across my machines; using ulimit -l in and outside of mpirun and they're all set to