Re: [OMPI users] issue compiling openmpi 3.2.1 with pmi and slurm

2018-10-10 Thread Gilles Gouaillardet
I digged a bit the configury logic and found ./configure --prefix=/usr/local/ --with-cuda --with-slurm --with-pmi=/usr/local/slurm should do the trick, if not ./configure --prefix=/usr/local/ --with-cuda --with-slurm --with-pmi=/usr/local/slurm --with-pmi-libdir=/usr/local/slurm/lib64

Re: [OMPI users] Cannot run MPI code on multiple cores with PBS

2018-10-10 Thread Bennet Fauber
There is a linux utility program `locate` that may be installed on your system. You could try $ locate ibv_devinfo Thus, mine returns $ locate ibv_devinfo /usr/bin/ibv_devinfo /usr/share/man/man1/ibv_devinfo.1.gz That should find it if it is on local disk and not in a network filesystem, and

Re: [OMPI users] issue compiling openmpi 3.2.1 with pmi and slurm

2018-10-10 Thread Charles A Taylor
In our config the "--with-pmi" points to the slurm “prefix” dir not the slurm libdir. The options below work for us with SLURM installed in “/opt/slurm”. I’ll note that after sharing this config with regard to another issue, it was recommended to drop the “/usr” in the “—with-foo=/usr” options

Re: [OMPI users] Cannot run MPI code on multiple cores with PBS

2018-10-10 Thread Castellana Michele
Dear John, I see, thank you for your reply. Unfortunately the cluster support is of poor quality, and it would take a while to get this information from them. Is there any way in which I can check this by myself? Also, it looks like ibv_devinfo does not exist on the cluster $ ibv_devinfo

Re: [OMPI users] --mca btl params

2018-10-10 Thread Jeff Squyres (jsquyres) via users
On Oct 9, 2018, at 8:55 PM, Noam Bernstein wrote: > >> That's basically what the output of >> ompi_info -a >> says. You actually probably want: ompi_info | grep btl That will show you the names and versions of the "btl" plugins that are available on your system. For example, this is what I

Re: [OMPI users] issue compiling openmpi 3.2.1 with pmi and slurm

2018-10-10 Thread Ross, Daniel B. via users
find /usr/local/slurm/ -name pmi.h /usr/local/slurm/include/slurm/pmi.h From: users On Behalf Of Bennet Fauber Sent: Wednesday, October 10, 2018 2:19 PM To: users@lists.open-mpi.org Subject: Re: [OMPI users] issue compiling openmpi 3.2.1 with pmi and slurm Yes, I was just wondering about the

Re: [OMPI users] issue compiling openmpi 3.2.1 with pmi and slurm

2018-10-10 Thread Bennet Fauber
Yes, I was just wondering about the path as specified, --with-pmi=/usr/local/slurm/include/slurm What do you get from $ find /usr/local/slurm -name pmi.h On Wed, Oct 10, 2018 at 2:07 PM Ross, Daniel B. via users < users@lists.open-mpi.org> wrote: > This is a slurm build and you point it

Re: [OMPI users] issue compiling openmpi 3.2.1 with pmi and slurm

2018-10-10 Thread Ross, Daniel B. via users
This is a slurm build and you point it to the slurm pmi directories. From: users on behalf of Bennet Fauber Sent: Wednesday, October 10, 2018 1:14 PM To: users@lists.open-mpi.org Subject: Re: [OMPI users] issue compiling openmpi 3.2.1 with pmi and slurm I

Re: [OMPI users] issue compiling openmpi 3.2.1 with pmi and slurm

2018-10-10 Thread Bennet Fauber
I thought the --with-pmi=/the/dir was meant to point to the top of a traditional FHS installation of PMI; e.g., /opt/pmi with subdirectories for bin, lib, include, man, etc. It looks like this is pointing only to the header file, based on the name. On Wed, Oct 10, 2018 at 11:27 AM Ralph H

Re: [OMPI users] issue compiling openmpi 3.2.1 with pmi and slurm

2018-10-10 Thread Ralph H Castain
Yeah, the CPPFLAGS didn’t get set - I suspect because the location being pointed to starts with “/usr/local” and therefore looks like a standard one. What you might do is set CPPFLAGS yourself to be “/usr/local/slurm/include/slurm” before running configure - should fix the problem. Might also

Re: [OMPI users] issue compiling openmpi 3.2.1 with pmi and slurm

2018-10-10 Thread Ross, Daniel B. via users
Here is the best I can do. The config.log is too large to email. grep pmi config.log $ ./configure --prefix=/usr/local/ --with-cuda --with-slurm --with-pmi=/usr/local/slurm/include/slurm --with pmi-libdir=/usr/local/slurm/lib64 configure:71744: result: '--prefix=/usr/local/' '--with-cuda'

Re: [OMPI users] no openmpi over IB on new CentOS 7 system

2018-10-10 Thread Noam Bernstein
> On Oct 10, 2018, at 4:51 AM, Dave Love wrote: > > RDMA was just broken in the last-but-one(?) RHEL7 kernel release, in > case that's the problem. (Fixed in 3.10.0-862.14.4.) I strongly suspect that this is it. In the process of getting everything organized to collect the info various

Re: [OMPI users] issue compiling openmpi 3.2.1 with pmi and slurm

2018-10-10 Thread Ralph H Castain
It appears that the CPPFLAGS isn’t getting set correctly as the component didn’t find the Slurm PMI-1 header file. Perhaps it would help if we saw the config.log output so we can see where OMPI thought the file was located. > On Oct 10, 2018, at 6:44 AM, Ross, Daniel B. via users > wrote: >

[OMPI users] issue compiling openmpi 3.2.1 with pmi and slurm

2018-10-10 Thread Ross, Daniel B. via users
I have been able to configure without issue using the following options: ./configure --prefix=/usr/local/ --with-cuda --with-slurm --with-pmi=/usr/local/slurm/include/slurm --with-pmi-libdir=/usr/local/slurm/lib64 Everything compiles just fine until I get this error: make[3]: Leaving directory

Re: [OMPI users] ompio on Lustre

2018-10-10 Thread Gabriel, Edgar
Well, good question. To be fair, the test passes if you run it with a lower number of processes. In addition, I had a couple of years back a discussion on that with one of the HDF5 developers, and it seemed to be ok to run it this way. That being said, after thinking about it a bit, I think the

Re: [OMPI users] no openmpi over IB on new CentOS 7 system

2018-10-10 Thread John Hearns via users
On that system please tell us what these return: ibstat ibstatus sminfo ibdiagnet On Wed, 10 Oct 2018 at 12:49, John Hearns wrote: > > Noam, what does ompi_info say - specifically which BTLs are available? > Stupid question though - this is a single system with no connection to a > switch?

Re: [OMPI users] no openmpi over IB on new CentOS 7 system

2018-10-10 Thread John Hearns via users
Noam, what does ompi_info say - specifically which BTLs are available? Stupid question though - this is a single system with no connection to a switch? You probably dont have an OpenSM subnet manager running then - could that be the root cause? On Wed, 10 Oct 2018 at 09:53, Dave Love wrote: > >

Re: [OMPI users] no openmpi over IB on new CentOS 7 system

2018-10-10 Thread Dave Love
RDMA was just broken in the last-but-one(?) RHEL7 kernel release, in case that's the problem. (Fixed in 3.10.0-862.14.4.) ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] ompio on Lustre

2018-10-10 Thread Dave Love
"Gabriel, Edgar" writes: > Ok, thanks. I usually run these test with 4 or 8, but the major item > is that atomicity is one of the areas that are not well supported in > ompio (along with data representations), so a failure in those tests > is not entirely surprising . If it's not expected to