I'm specifically wanting my users to be able to load a "debug" vs. "tuned" UCX module, without me having to make two different Open MPI installs... the combinatorics get bad after a few versions.... (I'm already having multiple versions of Open MPI to handle the differences in Fortran mpi mod files for various compilers.) Here are the differences in the configure options between the two UCX modules: debug version: --enable-logging --enable-debug --enable-assertions --enable-params-check --prefix=/same/install/prefix/ucx/1.9.0/debug tuned version: --disable-logging --disable-debug --disable-assertions --disable-params-check --prefix=/same/install/prefix/ucx/1.9.0/tuned
We noticed that the --enable-debug option for UCX has a pretty dramatic performance hit for one application (so far). I've already tested that everything works fine if I replace UCX's .so files manually in the filesystem, and the "new/changed" ones get loaded, but a user can't make that kind of swap. My hope is a user could type "module swap ucx/1.9.0/tuned ucx/1.9.0/debug" when they want to enable debugging at the UCX layer. On Sun, Jan 24, 2021 at 4:43 PM Yossi Itigin <yos...@nvidia.com> wrote: > > Hi, > > One option is to use LD_PRELOAD to load all ucx libraries from a specific > location > For example: mpirun -x > LD_PRELOAD=<path-to-libucp.so>:<path-to-libuct.so>:<path-to-libucs.so>:<path-to-libucm.so> > ... <exe> <args> > > BTW, what is different about the other UCX configuration? Maybe this is > something which can be resolved another way. > > --Yossi > > -----Original Message----- > From: devel <devel-boun...@lists.open-mpi.org> On Behalf Of Tim Mattox via > devel > Sent: Sunday, 24 January 2021 23:18 > To: devel@lists.open-mpi.org > Cc: Tim Mattox <tmat...@gmail.com> > Subject: [OMPI devel] How to build Open MPI so the UCX used can be changed at > runtime? > > Hello, > I've run into an application that has its performance dramatically affected > by some configuration options to the underlying UCX library. > Is there a way to configure/build Open MPI so that which UCX library is used > is determined at runtime (e.g. by an environment module), rather than having > to configure/build different instances of Open MPI? > > When I configure Open MPI 4.1.0 with "--with-ucx" it is hardcoding the full > path to the UCX .so library files to the UCX version it found at configure > time. > -- > Tim Mattox, Ph.D. - tmat...@gmail.com -- Tim Mattox, Ph.D. - tmat...@gmail.com
