Re: [OMPI users] relocating an installation
To be clear, --prefix and OPAL_PREFIX do two different things: 1. "mpirun --prefix" and "/abs/path/to/mpirun" (both documented in mpirun(1)) are used to set PATH and LD_LIBRARY_PATH on remote nodes when invoked via ssh/rsh. It's a way of not having to set your shell startup files to point to a specific Open MPI installation on remote nodes. This is helpful for users who tend to swap between multiple Open MPI installations. 2. OPAL_PREFIX overrides the compiled-in location in Open MPI binaries and libraries for where it should find things like plugins and help files. It is intended for use when you relocate Open MPI installations. These two can be used in conjunction with each other, if desired, but they are intended for two different purposes. > On Apr 10, 2019, at 9:06 AM, Reuti wrote: > > >> Am 10.04.2019 um 14:57 schrieb Dave Love : >> >> In fact, setting OPAL_PREFIX doesn't work for a relocated tree (with >> OMPI 1.10 or 3.0). You also need $OPAL_PREFIX/lib and >> $OPAL_PREFIX/lib/openmpi on LD_LIBRARY_PATH (assuming >> $MPI_LIB=$MPI_HOME/lib): >> >> $ OPAL_PREFIX=$(pwd)/usr/lib64/openmpi3 ./usr/lib64/openmpi3/bin/mpirun >> mpirun true >> ./usr/lib64/openmpi3/bin/mpirun: error while loading shared libraries: >> libopen-rte.so.40: cannot open shared object file: No such file or directory > > Does export this OPAL_PREFIX also to all child processes? For me using export > OPAL_PREFIX=… beforehand worked up to now. > > -- Reuti > ___ > users mailing list > users@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/users -- Jeff Squyres jsquy...@cisco.com ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] relocating an installation
> Am 10.04.2019 um 14:57 schrieb Dave Love : > > In fact, setting OPAL_PREFIX doesn't work for a relocated tree (with > OMPI 1.10 or 3.0). You also need $OPAL_PREFIX/lib and > $OPAL_PREFIX/lib/openmpi on LD_LIBRARY_PATH (assuming $MPI_LIB=$MPI_HOME/lib): > > $ OPAL_PREFIX=$(pwd)/usr/lib64/openmpi3 ./usr/lib64/openmpi3/bin/mpirun > mpirun true > ./usr/lib64/openmpi3/bin/mpirun: error while loading shared libraries: > libopen-rte.so.40: cannot open shared object file: No such file or directory Does export this OPAL_PREFIX also to all child processes? For me using export OPAL_PREFIX=… beforehand worked up to now. -- Reuti ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] relocating an installation
In fact, setting OPAL_PREFIX doesn't work for a relocated tree (with OMPI 1.10 or 3.0). You also need $OPAL_PREFIX/lib and $OPAL_PREFIX/lib/openmpi on LD_LIBRARY_PATH (assuming $MPI_LIB=$MPI_HOME/lib): $ OPAL_PREFIX=$(pwd)/usr/lib64/openmpi3 ./usr/lib64/openmpi3/bin/mpirun mpirun true ./usr/lib64/openmpi3/bin/mpirun: error while loading shared libraries: libopen-rte.so.40: cannot open shared object file: No such file or directory ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] relocating an installation
"Jeff Squyres (jsquyres) via users" writes: > Reuti's right. > > Sorry about the potentially misleading use of "--prefix" -- we > basically inherited that CLI option from a different MPI > implementation (i.e., people asked for it). So we were locked into > that meaning for the "--prefix" CLI options. Reading more closely, I see it's only for remote hosts anyhow, but I'd be surprised if only binaries and libraries were relocated from one tree, and only hosts that somehow were distinguished from the one invoking mpirun (et al). Regardless, is there some problem just documenting the relevant variables in the obvious places? ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] relocating an installation
Reuti writes: >> It should be documented. > > There is this FAQ entry: > > https://www.open-mpi.org/faq/?category=building#installdirs For what it's worth, I looked under "running" in the FAQ, as I was after a runtime switch. I expect FAQs to point to the actual documentation, though, and an environment variables section of the man pages seems the right place. [I know you're only providing info, thanks.] ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] relocating an installation
Reuti's right. Sorry about the potentially misleading use of "--prefix" -- we basically inherited that CLI option from a different MPI implementation (i.e., people asked for it). So we were locked into that meaning for the "--prefix" CLI options. > On Apr 9, 2019, at 9:14 AM, Reuti wrote: > >> >> Am 09.04.2019 um 14:52 schrieb Dave Love : >> >> Reuti writes: >> >>> export OPAL_PREFIX= >>> >>> to point it to the new location of installation before you start `mpiexec`. >> >> Thanks; that's now familiar, and I don't know how I missed it with >> strings. >> >> It should be documented. > > There is this FAQ entry: > > https://www.open-mpi.org/faq/?category=building#installdirs > > -- Reuti > > >> I'd have expected --prefix to have the same >> effect, and for there to be an MCA variable. Would there be some >> problem with either of those? >> ___ >> users mailing list >> users@lists.open-mpi.org >> https://lists.open-mpi.org/mailman/listinfo/users > > ___ > users mailing list > users@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/users -- Jeff Squyres jsquy...@cisco.com ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] relocating an installation
> Am 09.04.2019 um 14:52 schrieb Dave Love : > > Reuti writes: > >> export OPAL_PREFIX= >> >> to point it to the new location of installation before you start `mpiexec`. > > Thanks; that's now familiar, and I don't know how I missed it with > strings. > > It should be documented. There is this FAQ entry: https://www.open-mpi.org/faq/?category=building#installdirs -- Reuti > I'd have expected --prefix to have the same > effect, and for there to be an MCA variable. Would there be some > problem with either of those? > ___ > users mailing list > users@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/users ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] relocating an installation
Dave, Can you please post the configure command line of the Open MPI you are trying to relocate ? Cheers, Gilles Dave Love wrote: >Reuti writes: > >> export OPAL_PREFIX= >> >> to point it to the new location of installation before you start `mpiexec`. > >Thanks; that's now familiar, and I don't know how I missed it with >strings. > >It should be documented. I'd have expected --prefix to have the same >effect, and for there to be an MCA variable. Would there be some >problem with either of those? >___ >users mailing list >users@lists.open-mpi.org >https://lists.open-mpi.org/mailman/listinfo/users ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] relocating an installation
Reuti writes: > export OPAL_PREFIX= > > to point it to the new location of installation before you start `mpiexec`. Thanks; that's now familiar, and I don't know how I missed it with strings. It should be documented. I'd have expected --prefix to have the same effect, and for there to be an MCA variable. Would there be some problem with either of those? ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] relocating an installation
Hi, > Am 09.04.2019 um 13:31 schrieb Dave Love : > > Is it possible to use the environment or mpirun flags to run an OMPI > that's been relocated from where it was configured/installed? (Say > you've unpacked a system package that expects to be under /usr and want > to run it from home without containers etc.) I thought that was > possible, but I haven't found a way that works. Using --prefix doesn't > find help files, at least. export OPAL_PREFIX= to point it to the new location of installation before you start `mpiexec`. -- Reuti > ___ > users mailing list > users@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/users ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
[OMPI users] relocating an installation
Is it possible to use the environment or mpirun flags to run an OMPI that's been relocated from where it was configured/installed? (Say you've unpacked a system package that expects to be under /usr and want to run it from home without containers etc.) I thought that was possible, but I haven't found a way that works. Using --prefix doesn't find help files, at least. ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] Relocating an Installation
Upon reflecting on this more, I guess I see two issues. First, there's the issue of allowing the user to install our software on the system where ever they like. Some users may want to install it in their home directories, others may have a sys admin install it in a common location. This seems like a substantial reason to allow an OMPI installation to be relocated. So, I would say this was a very important capability. On the other hand, I don't have access to all the third party headers and libraries which are necessary to build some of the more interesting OMPI modules, such as the Infiniband and Myrinet drivers and many of the batch scheduling drivers (tm? LoadLeveler? PORTALS? Xgrid? [I'm not sure what these are]. And maybe a related question: One customer uses NQS (NQE?); can this be supported?) So, I would expect the user may want to compile at least some of OMPI himself (or herself) in order to activate these modules. Thus, perhaps I should supply a partially built installation which completes compilation as part of the installation process? This seems somewhat impractical since it would require compilers, headers and libraries, etc, on every machine on which our software is installed. I also don't know what distribution restrictions are placed on all the 3rd party software OMPI can link against. This may limit what can be redistributed with our product. So, I guess I'm open to suggestions on how best to distribute our software. Being able to relocate an installation and being able to build specific modules at installation time would appear to be very helpful capabilities. Many thanks, Allen On Fri, 2006-12-15 at 19:45 -0500, Jeff Squyres wrote: > Greetings Allen. > > This problem has not yet been resolved, but I'm quite sure we have an > open ticket about this on our bug tracker. I just replied to Patrick > on-list about a related issue (his use of --prefix); I'd have to > think about this a bit more, but a solution proposed by one of the > other OMPI developers in internal conversations may fix both issues. > It only hasn't been coded up because we didn't prioritize it high. > > So my question to you is -- how high of a priority is this for you? > Part of what makes it into each OMPI release is driven by what users > want/need, so input like this helps us prioritize the work. > > Thanks! > > > On Dec 13, 2006, at 10:37 AM, Allen Barnett wrote: > > > There was a thread back in November started by Patrick Jessee about > > relocating an installation after it was built (the subject was: > > removing > > hard-coded paths from OpenMPI shared libraries). I guess I'm in the > > same > > boat now. I would like to distribute our OpenMPI-based parallel > > solver; > > but I can't really dictate where a user will install our software. Has > > any one succeeded in building a version of OpenMPI which can be > > relocated? > -- Allen Barnett Transpire, Inc. E-Mail: al...@transpireinc.com Ph: 518-887-2930
Re: [OMPI users] Relocating an Installation
Greetings Allen. This problem has not yet been resolved, but I'm quite sure we have an open ticket about this on our bug tracker. I just replied to Patrick on-list about a related issue (his use of --prefix); I'd have to think about this a bit more, but a solution proposed by one of the other OMPI developers in internal conversations may fix both issues. It only hasn't been coded up because we didn't prioritize it high. So my question to you is -- how high of a priority is this for you? Part of what makes it into each OMPI release is driven by what users want/need, so input like this helps us prioritize the work. Thanks! On Dec 13, 2006, at 10:37 AM, Allen Barnett wrote: There was a thread back in November started by Patrick Jessee about relocating an installation after it was built (the subject was: removing hard-coded paths from OpenMPI shared libraries). I guess I'm in the same boat now. I would like to distribute our OpenMPI-based parallel solver; but I can't really dictate where a user will install our software. Has any one succeeded in building a version of OpenMPI which can be relocated? -- Jeff Squyres Server Virtualization Business Unit Cisco Systems
[OMPI users] Relocating an Installation
Hi: There was a thread back in November started by Patrick Jessee about relocating an installation after it was built (the subject was: removing hard-coded paths from OpenMPI shared libraries). I guess I'm in the same boat now. I would like to distribute our OpenMPI-based parallel solver; but I can't really dictate where a user will install our software. Has any one succeeded in building a version of OpenMPI which can be relocated? Thanks, Allen -- Allen Barnett Transpire, Inc. E-Mail: al...@transpireinc.com Ph: 518-887-2930
