Re: [OMPI packagers] Recommended combinations of pmix/prrte/openmpi
Thanks for the information. I'll start testing with the various RCs for
pmix 4.2.4, prrte 3.0.1, and openmpi 5.0.0.
On 3/18/23 20:14, Ralph Castain wrote:
First, sorry for all the confusion! I know it is a bit to parse as we
transition from OMPI v4 and earlier (which had its own embedded runtime
called ORTE) to OMPI v5, which includes a 3rd-party runtime called
PRRTE. You never split ORTE out of those earlier OMPI versions and so
these compatibility issues didn't previously exist. But now they will,
starting with OMPI v5.
If it is any consolation, we are also trying to work our way thru all
these combinations. It places constraints on multiple parties,
especially as PRRTE and PMIx are used by a number of other parties not
named OMPI. So none of us involved in the different projects enjoy total
freedom, and we are still figuring out how to manage all the moving pieces.
Bottom line is that you don't need to be worrying about PRRTE for OMPI
v4 and earlier. It is orthogonal to those releases.
We have previously identified a problem with OMPI's PMIx integration in
the v4.x series and a patch has been filed for it. This is the source of
the issue you cite. It isn't a PMIx problem, but rather a case of
replacing a previously non-standard function call with the standardized
macro. The patch (see https://github.com/open-mpi/ompi/pull/11472
<https://github.com/open-mpi/ompi/pull/11472>) fixes it for all PMIx
releases (both looking backwards and going forwards).
Looking forward to OMPI v5:
* PRRTE 2.x is not something that will be supported. Frankly, we do not
recommend anyone use that series.
* OMPI v5 is going to require a minimum of PRRTE v3.0.1, although OMPI
v5 might include v3.1.0 (the precise release to be included in the OMPI
v5 code remains TBD). Either PRRTE version will require a minimum of
PMIx v4.2.4 to properly function
* OMPI v5 will require a minimum of PMIx v4.2.4 - OMPI might ship with
the PMIx v5.0.0 release, but it will still work with v4.2.4
The problems you cite are unfortunately expected and part of the
learning process. Part of the delay in releasing OMPI v5 has been a
result of trying to resolve the logistics - hopefully, we are converging
on a stable system.
Ralph
On Mar 18, 2023, at 4:24 PM, Orion Poplawski wrote:
What are the current recommendations for compatible combinations of
pmix/prrte/openmpi?
I'm looking into updating each of these in Fedora and running into a
couple of issues. Currently in Fedora Rawhide we have:
- pmix 4.1.2
- prrte 2.0.2
- openmpi 4.1.5
After updating to pmix 4.2.3 I see the following:
- openmpi programs fail to run with:
mca_base_component_repository_open: unable to open mca_pmix_ext3x:
/usr/lib64/openmpi/lib/openmpi/mca_pmix_ext3x.so: undefined symbol:
pmix_value_load (ignored)
[[3260,0],0] ORTE_ERROR_LOG: Not found in file ess_hnp_module.c at
line 320
--
It looks like orte_init failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems. This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):
opal_pmix_base_select failed
--> Returned value Not found (-13) instead of ORTE_SUCCESS
- prrte 2.0.2 fails to build with:
make[2]: Entering directory
'/builddir/build/BUILD/prrte-2.0.2/src/tools/prted'
/bin/sh ../../../libtool --tag=CC --mode=link gcc
-DPRTE_CONFIGURE_USER="\"mockbuild\""
-DPRTE_CONFIGURE_HOST="\"2188cae5486f485888615d01f56cd6c9\""
-DPRTE_CONFIGURE_DATE="\"Fri Jan 20 00:00:00 UTC 2023\""
-DPRTE_BUILD_USER="\"$USER\""
-DPRTE_BUILD_HOST="\"${HOSTNAME:-`(hostname || uname -n) | sed 1q`}\""
-DPRTE_BUILD_DATE="\"`../../../config/getdate.sh`\""
-DPRTE_BUILD_CFLAGS="\"-DNDEBUG -O2 -flto=auto -ffat-lto-objects
-fexceptions -g -grecord-gcc-switches -pipe
-specs=/usr/lib/rpm/redhat/redhat-hardened-cc1
-fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1
-m64 -mtune=generic -fasynchronous-unwind-tables
-fstack-clash-protection -fcf-protection -fno-omit-frame-pointer
-mno-omit-leaf-frame-pointer -finline-functions -Wall
-Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3
-Wp,-D_GLIBCXX_ASSERTIONS\""
-DPRTE_BUILD_CPPFLAGS="\"-iquote/builddir/build/BUILD/prrte-2.0.2
-iquote/builddir/build/BUILD/prrte-2.0.2/src/include\""
-DPRTE_BUILD_LDFLAGS="\"-Wl,-z,relro -Wl,--as-needed -Wl,-z,now
-specs=/usr/lib/rpm/redhat/redhat-hardened-ld
-specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -Wl,--build-id=sha1
-specs=/usr/lib/rpm/redhat/redhat-package-notes\""
-DPRTE_BUILD
[OMPI packagers] Recommended combinations of pmix/prrte/openmpi
=/usr/lib/rpm/redhat/redhat-package-notes\"" "-DPRTE_BUILD_LIBS=\"-lm -levent_core -levent_pthreads -lpmix -lhwloc\"" -DPRTE_CC_ABSOLUTE=\"/usr/bin/gcc\" -DPRTE_GREEK_VERSION=\"\" -DPRTE_REPO_REV=\"v2.0.1-8-gaa57929\" "-DPRTE_RELEASE_DATE=\"Feb 10, 2022\"" -DNDEBUG -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -finline-functions -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -Wl,-z -Wl,relro -Wl,--as-needed -Wl,-z -Wl,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -Wl,--build-id=sha1 -specs=/usr/lib/rpm/redhat/redhat-package-notes -o .libs/prted prted.o ../../../src/.libs/libprrte.so -lm -levent_core -levent_pthreads -lpmix -lhwloc make[2]: Leaving directory '/builddir/build/BUILD/prrte-2.0.2/src/tools/prted' /usr/bin/ld: ../../../src/.libs/libprrte.so: undefined reference to `pmix_argv_append_unique_nosize' /usr/bin/ld: ../../../src/.libs/libprrte.so: undefined reference to `PMIx_Byte_object_load' /usr/bin/ld: ../../../src/.libs/libprrte.so: undefined reference to `pmix_argv_join' collect2: error: ld returned 1 exit status make[2]: *** [Makefile:792: prted] Error 1 So I'm guessing an update to prrte 3.0.0 is in order here. But will openmpi 4.1.5 work with prrte 3.0.0? I've also reported and discussed these issues here: https://github.com/openpmix/openpmix/issues/3022 but figured it would be of interest to other openmpi packagers. -- Orion Poplawski he/him/his - surely the least important thing about me IT Systems Manager 720-772-5637 NWRA, Boulder/CoRA Office FAX: 303-415-9702 3380 Mitchell Lane or...@nwra.com Boulder, CO 80301 https://www.nwra.com/ smime.p7s Description: S/MIME Cryptographic Signature ___ ompi-packagers mailing list ompi-packagers@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/ompi-packagers
Re: [OMPI packagers] Support timelines
Thank you, Ralph. That's helpful. On 9/16/21 8:19 AM, Ralph Castain wrote: Sorry nobody answered this right away. It isn't quite that simple as we don't do support on a timeline basis. We generally support two major releases at a time. So v4 would remain supported until v6 was released. Our original intent was to have one major release per year - so the policy would roughly translate to a 2 year period. However, as you have probably noted, v5 is very, very late from that perspective. Thus, there is no real timeline we can provide - only the intent that it be for two years, driven completely by our ability to stick to the one major release per year policy. HTH Ralph On Sep 14, 2021, at 8:02 PM, Orion Poplawski wrote: Is there any documentation that would indicate how long the 4.0 (or any particular release series) will be supported? This would be helpful for establishing downstream timelines. Thanks! -- Orion Poplawski he/him/his - surely the least important thing about me Manager of NWRA Technical Systems 720-772-5637 NWRA, Boulder/CoRA Office FAX: 303-415-9702 3380 Mitchell Lane or...@nwra.com Boulder, CO 80301 https://www.nwra.com/ ___ ompi-packagers mailing list ompi-packagers@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/ompi-packagers ___ ompi-packagers mailing list ompi-packagers@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/ompi-packagers -- Orion Poplawski he/him/his - surely the least important thing about me Manager of NWRA Technical Systems 720-772-5637 NWRA, Boulder/CoRA Office FAX: 303-415-9702 3380 Mitchell Lane or...@nwra.com Boulder, CO 80301 https://www.nwra.com/ smime.p7s Description: S/MIME Cryptographic Signature ___ ompi-packagers mailing list ompi-packagers@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/ompi-packagers
[OMPI packagers] Support timelines
Is there any documentation that would indicate how long the 4.0 (or any particular release series) will be supported? This would be helpful for establishing downstream timelines. Thanks! -- Orion Poplawski he/him/his - surely the least important thing about me Manager of NWRA Technical Systems 720-772-5637 NWRA, Boulder/CoRA Office FAX: 303-415-9702 3380 Mitchell Lane or...@nwra.com Boulder, CO 80301 https://www.nwra.com/ smime.p7s Description: S/MIME Cryptographic Signature ___ ompi-packagers mailing list ompi-packagers@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/ompi-packagers
Re: [OMPI packagers] New RTE for OMPI v5
From a Fedora perspective -
We prefer separate packages, but it is acceptable to bundle
libraries if required - especially if OMPI is only tested against a few
(or a single) version of PRRTE. However, I would say that if Fedora
ends up packaging PRRTE separately we would really rather that open-mpi
build against that.
HTH,
Orion
On 4/27/20 6:47 PM, Ralph Castain wrote:
Just as an FYI - just learned that a major resource manager vendor will be
utilizing PRRTE as their runtime for executing applications. So a significant
percentage of HPC systems are going to need a PRRTE package, hopefully
distributed by someone from the distro community.
Ralph
On Apr 27, 2020, at 12:48 PM, Jeff Squyres (jsquyres) via ompi-packagers
wrote:
Open MPI packagers --
Just to be clear: this is an open question to you for the upcoming Open MPI
v5.0.x series.
We'd really appreciate your feedback.
Thanks!
On Apr 14, 2020, at 12:40 PM, Ralph Castain wrote:
Just pinging you all to ensure you got this. I need to know if we need
to coordinate an official PRRTE release to coincide (and sync) with the release of
OMPI v5.0, or if you are okay with just using the embedded PRRTE included with the
OMPI v5.0 tarball.
If it helps, PRRTE depends upon hwloc, libevent, and PMIx - none of which are
internally embedded.
Thanks
Ralph
On Apr 1, 2020, at 8:40 PM, Ralph Castain wrote:
Hi folks
I just wanted to alert you to the fact that we are replacing the ORTE runtime environment
in Open MPI with an external package called PRRTE ("PMIx Reference RunTime
Environment"). We will be including a copy of that package in the OMPI v5 tarball,
just as we do libevent, hwloc, and PMIx.
PRRTE historically has not been generating official releases - there is an old
v1.0, but nothing on a regular release sequence. As part of this change in
OMPI, the PRRTE folks will begin generating official releases that OMPI will
use in their releases. So there will be correlation between the packages.
My question to you is: is this a package you would prefer to distribute
separately (as you do for PMIx and friends), or shall we just leave it as an
included package? PRRTE does get used by a fairly small community of people at
the national labs and a couple of universities, but it by no means has as
wide-ranging a following as OMPI.
Just need to know if we need to add a --with-prrte option to OMPI's configure
code so one could point it at an external PRRTE installation.
Ralph
___
ompi-packagers mailing list
ompi-packagers@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/ompi-packagers
___
ompi-packagers mailing list
ompi-packagers@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/ompi-packagers
--
Jeff Squyres
jsquy...@cisco.com
___
ompi-packagers mailing list
ompi-packagers@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/ompi-packagers
___
ompi-packagers mailing list
ompi-packagers@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/ompi-packagers
--
Orion Poplawski
Manager of NWRA Technical Systems 720-772-5637
NWRA, Boulder/CoRA Office FAX: 303-415-9702
3380 Mitchell Lane or...@nwra.com
Boulder, CO 80301 https://www.nwra.com/
smime.p7s
Description: S/MIME Cryptographic Signature
___
ompi-packagers mailing list
ompi-packagers@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/ompi-packagers
[OMPI packagers] openmpi and pmix compatibility
Are there any restrictions on what versions of pmix to use with particular versions of openmpi? In particular - is it okay to have a "newer" pmix (3.0.2) with openmpi 3.1.X or even openmpi 2.1.X? Thanks. -- Orion Poplawski Manager of NWRA Technical Systems 720-772-5637 NWRA, Boulder/CoRA Office FAX: 303-415-9702 3380 Mitchell Lane or...@nwra.com Boulder, CO 80301 https://www.nwra.com/ ___ ompi-packagers mailing list ompi-packagers@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/ompi-packagers
