[OMPI users] Warning about not enough registerable memory on SL6.6

Dear Open-MPI experts, I have updated my little cluster from Scientific Linux 6.5 to 6.6, this included extensive changes in the Infiniband drivers and a newer openmpi version (1.8.1). Now I'm getting this message on all nodes with more than 32 GB of RAM: WARNING: It appears that your

Re: [OMPI users] Warning about not enough registerable memory on SL6.6

Hi, This should be fixed in OMPI 1.8.3. Is it possible for you to give 1.8.3 a shot? Best, Josh On Mon, Dec 8, 2014 at 8:43 AM, Götz Waschk wrote: > Dear Open-MPI experts, > > I have updated my little cluster from Scientific Linux 6.5 to 6.6, > this included extensive

Re: [OMPI users] Converting --cpus-per-proc to --map-by for a hybrid code

lstopo is pretty! John

Re: [OMPI users] Warning about not enough registerable memory on SL6.6

Hi, I had tested 1.8.4rc1 and it wasn't fixed. I can try again though, maybe I had made an error. Regards, Götz Waschk On Mon, Dec 8, 2014 at 3:17 PM, Joshua Ladd wrote: > Hi, > > This should be fixed in OMPI 1.8.3. Is it possible for you to give 1.8.3 a > shot? > > Best,

Re: [OMPI users] Converting --cpus-per-proc to --map-by for a hybrid code

Thanks for sending that lstopo output - helped clarify things for me. I think I now understand the issue. Mostly a problem of my being rather dense when reading your earlier note. Try using —map-by node:PE=N to your cmd line. I think the problem is that we default to —map-by numa if you just

Re: [OMPI users] Converting --cpus-per-proc to --map-by for a hybrid code

OMP_NUM_THREADS=1 mpiexec -n 1 gnu_openmpi_a/one_c_prof.exe : 113 iterations OMP_NUM_THREADS=6 mpiexec -n 1 --map-by node:PE=6 : 639 iterations OMP_NUM_THREADS=6 mpiexec -n 2 --map-by node:PE=6 : 639 iterations OMP_NUM_THREADS=12 mpiexec -n 1 --map-by node:PE=12 : 1000 iterations

Re: [OMPI users] Warning about not enough registerable memory on SL6.6

Hi, here's another test with openmpi 1.8.3. With 1.8.1, 32GB was detected, now it is just 16: % mpirun -np 2 /usr/lib64/openmpi-intel/bin/mpitests-osu_get_bw -- WARNING: It appears that your OpenFabrics subsystem is

Re: [OMPI users] Warning about not enough registerable memory on SL6.6

Folks, FWIW, i observe a similar behaviour on my system. imho, the root cause is OFED has been upgraded from a (quite) older version to latest 3.12 version here is the relevant part of code (btl_openib.c from the master) : static uint64_t calculate_max_reg (void) { if (0 ==