My environment modules were already setting LD_LIBRARY_PATH to point
to my UCX lib directory.
The real problem was that OMPI's config/ompi_check_ucx.m4 was
recording the full path to the UCX library if it wasn't found in a
standard system location (e.g. /lib, /lib64, /usr/lib, etc.).
That is normally a good thing to do, since the chances that the
average mpirun user will have setup their LD_LIBRARY_PATH is lower
than the software installer would have done so correctly (I hope).
In my case however, I'm setting up the environment modules to enforce
the LD_LIBRARY_PATH to have an ABI compatible UCX available.
I found two ways to override ompi_check_ucx.m4 to just let the linker
find the UCX libraries on its own:
1) Modify UCX's lib/pkgconfig/ucx.pc to have "prefix = /usr" so that
ompi_check_ucx.m4 would think UCX was in a standard system location
(which is a lie).
2) Patch the OMPI configure system to allow me to force it to not
hard-code the path to UCX into the OMPI .so files such as
libmca_common_ucx.so
I chose the latter, since there might be other users of my UCX
installs, and I didn't want to possibly break them by having
pkg_config lie.
And, I was already having to patch an OMPI config/*.m4 file for a
different reason, and was thus already having to run "./autogen.pl -f"
anyway.
So, with the patch below, I can now do an OMPI configure line with
"... --with-ucx=from_runtime_env ..." and the resulting build will use
whatever UCX is found at link time.
Normally, that would be scary, since who knows what ancient version of
UCX might be sitting around from the Linux distro, but my environment
modules are enforcing the dependencies so that my UCX libraries will
be found, even if a user does a "module swap". Hey, a cool benefit of
lmod or modules v4+, but I digress.
Anyway, if there is some other OMPI sanctioned way to do this, please
let me know, or suggest a better way so I can upstream a patch. I
have vague memories of being able to force this kind of behavior by
doing something like "configure ... --with-ucx=/usr ...", but I
couldn't find any documentation for that, and in doing code inspection
of the m4 files revealed that such a feature (if it was an intended
feature) had bitrotted.
--- openmpi-4.1.0/config/ompi_check_ucx.m4.orig 2021-01-25
18:23:17.112499399 -0600
+++ openmpi-4.1.0/config/ompi_check_ucx.m4 2021-01-25 20:25:15.919338784 -0600
@@ -41,7 +41,7 @@
[ompi_check_ucx_dir=])],
[true])])
ompi_check_ucx_happy="no"
- AS_IF([test -z "$ompi_check_ucx_dir"],
+ AS_IF([test -z "$ompi_check_ucx_dir" || test
"$ompi_check_ucx_dir" = "from_runtime_env"],
[OPAL_CHECK_PACKAGE([ompi_check_ucx],
[ucp/api/ucp.h],
[ucp],
Oddly, the Open MPI configure script already has overrides for
ucx_CFLAGS, ucx_LIBS, and ucx_STATIC_LIBS, but nothing for something
like "ucx_LDFLAGS=''".
I didn't see a simple way to add support for such an override without
some more extensive changes to multiple m4 files.
On Sun, Jan 24, 2021 at 7:08 PM Gilles Gouaillardet via devel
<devel@lists.open-mpi.org> wrote:
>
> Tim,
>
> Have you tried using LD_LIBRARY_PATH?
> I guess "hardcoding the full path" means "link with -rpath", and IIRC,
> LD_LIBRARY_PATH
> overrides this setting.
>
>
> If this does not work, here something you can try (disclaimer: I did not)
>
> export LD_LIBRARY_PATH=/same/install/prefix/ucx/1.9.0/lib
> configure ... --with-ucx
> CPPFLAGS=-I/same/install/prefix/ucx/1.9.0/include
> LDFLAGS=-L/same/install/prefix/ucx/1.9.0/lib
>
> I expect the UCX components use libuct.so instead of
> /same/install/prefix/ucx/1.9.0/lib/libuct.so.
> If your users want the debug version, then you can simply change your
> LD_LIBRARY_PATH
> (module swap ucx should do the trick)
>
> Three caveats you should keep in mind:
> - it is your responsibility to ensure the debug and prod versions of
> UCX are ABI compatible
> - it will be mandatory to load a ucx module (otherwise Open MPI won't
> find UCX libraries)
> - this is a guess and I did not test this.
>
>
> An other option (I did not try) would be to install UCX on your build
> machine in /usr
> (since I expect /usr/lib/libuct.so is not hardcoded) and then use
> LD_LIBRARY_PATH
> (I assume your ucx module set it) to point to the UCX flavor of your choice).
>
> Cheers,
>
> Gilles
>
> On Mon, Jan 25, 2021 at 7:43 AM Tim Mattox via devel
> <devel@lists.open-mpi.org> wrote:
> >
> > I'm specifically wanting my users to be able to load a "debug" vs.
> > "tuned" UCX module, without me having to make two different Open MPI
> > installs... the combinatorics get bad after a few versions.... (I'm
> > already having multiple versions of Open MPI to handle the differences
> > in Fortran mpi mod files for various compilers.)
> > Here are the differences in the configure options between the two UCX
> > modules:
> > debug version: --enable-logging --enable-debug --enable-assertions
> > --enable-params-check --prefix=/same/install/prefix/ucx/1.9.0/debug
> > tuned version: --disable-logging --disable-debug --disable-assertions
> > --disable-params-check --prefix=/same/install/prefix/ucx/1.9.0/tuned
> >
> > We noticed that the --enable-debug option for UCX has a pretty
> > dramatic performance hit for one application (so far).
> > I've already tested that everything works fine if I replace UCX's .so
> > files manually in the filesystem, and the "new/changed" ones get
> > loaded, but a user can't make that kind of swap.
> > My hope is a user could type "module swap ucx/1.9.0/tuned
> > ucx/1.9.0/debug" when they want to enable debugging at the UCX layer.
> >
> > On Sun, Jan 24, 2021 at 4:43 PM Yossi Itigin <yos...@nvidia.com> wrote:
> > >
> > > Hi,
> > >
> > > One option is to use LD_PRELOAD to load all ucx libraries from a specific
> > > location
> > > For example: mpirun -x
> > > LD_PRELOAD=<path-to-libucp.so>:<path-to-libuct.so>:<path-to-libucs.so>:<path-to-libucm.so>
> > > ... <exe> <args>
> > >
> > > BTW, what is different about the other UCX configuration? Maybe this is
> > > something which can be resolved another way.
> > >
> > > --Yossi
> > >
> > > -----Original Message-----
> > > From: devel <devel-boun...@lists.open-mpi.org> On Behalf Of Tim Mattox
> > > via devel
> > > Sent: Sunday, 24 January 2021 23:18
> > > To: devel@lists.open-mpi.org
> > > Cc: Tim Mattox <tmat...@gmail.com>
> > > Subject: [OMPI devel] How to build Open MPI so the UCX used can be
> > > changed at runtime?
> > >
> > > Hello,
> > > I've run into an application that has its performance dramatically
> > > affected by some configuration options to the underlying UCX library.
> > > Is there a way to configure/build Open MPI so that which UCX library is
> > > used is determined at runtime (e.g. by an environment module), rather
> > > than having to configure/build different instances of Open MPI?
> > >
> > > When I configure Open MPI 4.1.0 with "--with-ucx" it is hardcoding the
> > > full path to the UCX .so library files to the UCX version it found at
> > > configure time.
> > > --
> > > Tim Mattox, Ph.D. - tmat...@gmail.com
> >
> >
> >
> > --
> > Tim Mattox, Ph.D. - tmat...@gmail.com
--
Tim Mattox, Ph.D. - tmat...@gmail.com
