Hi Stephan
> On Oct 12, 2018, at 2:25 AM, Stephan Krempel <krem...@par-tec.com> wrote: > > Hallo Ralph, > >> I assume this (--with-ompi-mpix-rte) is a typo as the correct option >> is —with-ompi-pmix-rte? > > You were right, this was a typo, with the correct option I now managed > to start an MPI helloworld program using OpenMPI and our own process > manager with pmix server. Hooray! If you want me to show support for your PM on our web site, please send me a little info about it. You are welcome to send it off-list if you prefer. > >> It all looks okay to me for the client, but I wonder if you >> remembered to call register_nspace and register_client on your server >> prior to starting the client? If not, the connection will be dropped >> - you could add PMIX_MCA_ptl_base_verbose=100 to your environment to >> see the detailed connection handshake. > > This has been a point that I could finally figure out from the prrte > code. To make it working you do not only need to call register_nspace > but also pass some specific information to it that OpenMPI considers to > be available (e.g. proc info with lrank). My apologies - we will document this better on the PMIx web site and provide some link to it on the OMPI web site. We actually do publish the info OMPI is expecting, but it isn’t in an obvious enough place. > > A remark to pmix at this point: pmix_bfrops_base_value_load() does > silently not handle PMIX_DATA_ARRAY type leading to not working makros > PMIX_VALUE_LOAD and PMIX_INFO_LOAD with that type. I think this is > unlucky and took me a while to figure out why it comes to a segfault > when pmix tried to process my PMIX_PROC_DATA infos. I’ll check that out - I don’t know why we wouldn’t handle it, so it is likely just an oversight. Regardless, it should return an error if it isn’t doing it. > > So thank you again for your help so far. > > > One point that remains open and is interesting for me is if I can > achieve the same with the 3.1.2 release of OpenMPI. Is it somehow > possible to configure it as there were the "--with-ompi-pmix-rte" > switch from version 4.x? I’m afraid we didn’t backport that capability to the v3.x branches. I’ll ask the relevant release managers if they’d like us to do so. Ralph > > Regards, > > Stephan > > >> >>> On Oct 9, 2018, at 3:14 PM, Stephan Krempel <krem...@par-tec.com> >>> wrote: >>> >>> Hi Ralf, >>> >>> After studying prrte a little bit, I tried something new and >>> followed >>> the description here using openmpi 4: >>> https://pmix.org/code/building-the-pmix-reference-server/ >>> >>> I configured openmpi 4.0.0rc3: >>> >>> ../configure --enable-debug --prefix [...] --with-pmix=[...] \ >>> --with-libevent=/usr --with-ompi-mpix-rte >>> >>> (I also tried to set --with-orte=no, but it then claims not to have >>> a >>> suitable rte and does not finish) >>> >>> I then started my own PMIx and spawned a client compiled with mpicc >>> of >>> the new openmpi installation with this environment: >>> >>> PMIX_NAMESPACE=namespace_3228_0_0 >>> PMIX_RANK=0 >>> PMIX_SERVER_URI2=pmix-server.3234;tcp4://127.0.0.1:49637 >>> PMIX_SERVER_URI21=pmix-server.3234;tcp4://127.0.0.1:49637 >>> PMIX_SERVER_URI=pmix-server:3234:/tmp/pmix-3234 >>> PMIX_SERVER_URI2USOCK=pmix-server:3234:/tmp/pmix-3234 >>> PMIX_SECURITY_MODE=native,none >>> PMIX_PTL_MODULE=tcp,usock >>> PMIX_BFROP_BUFFER_TYPE=PMIX_BFROP_BUFFER_FULLY_DESC >>> PMIX_GDS_MODULE=ds12,hash >>> PMIX_DSTORE_ESH_BASE_PATH=/tmp/pmix_dstor_3234 >>> >>> The client is not connecting to my pmix server and it's environment >>> after MPI_Init looks like that: >>> >>> PMIX_SERVER_URI2USOCK=pmix-server:3234:/tmp/pmix-3234 >>> PMIX_RANK=0 >>> PMIX_PTL_MODULE=tcp,usock >>> PMIX_SERVER_URI=pmix-server:3234:/tmp/pmix-3234 >>> PMIX_MCA_mca_base_component_show_load_errors=1 >>> PMIX_BFROP_BUFFER_TYPE=PMIX_BFROP_BUFFER_FULLY_DESC >>> PMIX_DSTORE_ESH_BASE_PATH=/tmp/ompi.landroval.1001/pid.3243/pmix_ds >>> tor_ >>> 3243 >>> PMIX_SERVER_URI2=864157696.0;tcp4://127.0.0.1:33619 >>> PMIX_SERVER_URI21=864157696.0;tcp4://127.0.0.1:33619 >>> PMIX_SECURITY_MODE=native,none >>> PMIX_NAMESPACE=864157697 >>> PMIX_GDS_MODULE=ds12,hash >>> ORTE_SCHIZO_DETECTION=ORTE >>> OMPI_COMMAND=./hello_env >>> OMPI_MCA_orte_precondition_transports=f28d6577f6b6ac08- >>> d92c0e73869e1cfa >>> OMPI_MCA_orte_launch=1 >>> OMPI_APP_CTX_NUM_PROCS=1 >>> OMPI_MCA_pmix=^s1,s2,cray,isolated >>> OMPI_MCA_ess=singleton >>> OMPI_MCA_orte_ess_num_procs=1 >>> >>> So something goes wrong but I do not have an idea what I am >>> missing. Do >>> you have an idea what I need to change? Do I have to set an MCA >>> parameter to tell OpenMPI not to start orted, or does it need >>> another >>> hint in the client environment beside the stuff comming from the >>> PMIx >>> server helper library? >>> >>> >>> Stephan >>> >>> >>> On Tuesday, Oct 10 2018, 08:33 -0700 Ralph H Castain wrote: >>>> Hi Stephan >>>> >>>> Thanks for the clarification - that helps a great deal. You are >>>> correct that OMPI’s orted daemons do more than just host the PMIx >>>> server library. However, they are only active if you launch the >>>> OMPI >>>> processes using mpirun. This is probably the source of the >>>> trouble >>>> you are seeing. >>>> >>>> Since you have a process launcher and have integrated the PMIx >>>> server >>>> support into your RM’s daemons, you really have no need for >>>> mpirun at >>>> all. You should just be able to launch the processes directly >>>> using >>>> your own launcher. The PMIx support will take care of the startup >>>> requirements. The application procs will not use the orted in >>>> such >>>> cases. >>>> >>>> So if your system is working fine with the PMIx example programs, >>>> then just launch the OMPI apps the same way and it should just >>>> work. >>>> >>>> On the Slurm side: I’m surprised that it doesn’t work without the >>>> —with-slurm option. An application proc doesn’t care about any of >>>> the >>>> Slurm-related code if PMIx is available. I might have access to a >>>> machine where I can check it… >>>> >>>> Ralph >>>> >>>> >>>>> On Oct 9, 2018, at 3:26 AM, Stephan Krempel <krem...@par-tec.co >>>>> m> >>>>> wrote: >>>>> >>>>> Ralph, Gilles, >>>>> >>>>> thanks for your input. >>>>> >>>>> Before I answer, let me shortly explain what my general >>>>> intention >>>>> is. >>>>> We do have our own resource manager and process launcher that >>>>> supports >>>>> different MPI implementations in different ways. I want to >>>>> adapt it >>>>> to >>>>> PMIx to cleanly support OpenMPI and hopefully other MPI >>>>> implementation >>>>> supporting PMIx in the future, too. >>>>> >>>>>> It sounds like what you really want to do is replace the >>>>>> orted, >>>>>> and >>>>>> have your orted open your PMIx server? In other words, you >>>>>> want >>>>>> to >>>>>> use the PMIx reference library to handle all the PMIx stuff, >>>>>> and >>>>>> provide your own backend functions to support the PMIx server >>>>>> calls? >>>>> >>>>> You are right, that was my original plan, and I already did it >>>>> so >>>>> far. >>>>> In my environment I already can launch processes that >>>>> successfully >>>>> call >>>>> PMIx client functions like put, get, fence and so on, all >>>>> handled >>>>> by my >>>>> servers using the PMIx server helper library. As far as I >>>>> implemented >>>>> the server functions now, all the example programs coming with >>>>> the >>>>> pmix >>>>> library are working fine. >>>>> >>>>> Then I tried to use that with OpenMPI and stumbled. >>>>> My first idea was to simply replace orted but after taking a >>>>> closer >>>>> look into OpenMPI it seems to me, that it uses/needs orted not >>>>> only >>>>> for >>>>> spawning and exchange of process information, but also for its >>>>> general >>>>> communication and collectives. Am I wrong with that? >>>>> >>>>> So replacing it completely is perhaps not what I want since I >>>>> do >>>>> not >>>>> intent to replace OpenMPIs whole communication stuff. But >>>>> perhaps I >>>>> do >>>>> mix up orte and orted here, not certain about that. >>>>> >>>>>> If so, then your best bet would be to edit the PRRTE code in >>>>>> orte/orted/pmix and replace it with your code. You’ll have to >>>>>> deal >>>>>> with the ORTE data objects and PRRTE’s launch procedure, but >>>>>> that >>>>>> is >>>>>> likely easier than trying to write your own version of >>>>>> “orted” >>>>>> from >>>>>> scratch. >>>>> >>>>> I think one problem here is, that I do not really understand >>>>> which >>>>> purposes orted fulfills overall especially beside implementing >>>>> the >>>>> PMIx >>>>> server side. Can you please give me a short overview? >>>>> >>>>>> As for Slurm: it behaves the same way as PRRTE. It has a >>>>>> plugin >>>>>> that >>>>>> implements the server backend functions, and the Slurm >>>>>> daemons >>>>>> “host” >>>>>> the plugin. What you would need to do is replace that plugin >>>>>> with >>>>>> your own. >>>>> >>>>> I understand that, but it also seems to need some special >>>>> support >>>>> by >>>>> the several slurm modules on the OpenMPI side that I do not >>>>> understand, >>>>> yet. At least when I tried OpenMPI without slurm support and >>>>> `srun --mpi=pmix_v2` it does not work but generates a message >>>>> that >>>>> slurm support in opemmpi is missing. >>>>> >>>>> >>>>> >>>>> Stephan >>>>> >>>>> >>>>> >>>>>>> On Oct 8, 2018, at 5:36 PM, Gilles Gouaillardet <gilles@ris >>>>>>> t.or >>>>>>> .jp> >>>>>>> wrote: >>>>>>> >>>>>>> Stephan, >>>>>>> >>>>>>> >>>>>>> Have you already checked https://github.com/pmix/prrte ? >>>>>>> >>>>>>> >>>>>>> This is the PMIx Reference RunTime Environment (PPRTE), >>>>>>> which >>>>>>> was >>>>>>> built on top of orted. >>>>>>> >>>>>>> Long story short, it deploys the PMIx server and then you >>>>>>> start >>>>>>> your MPI app with prun >>>>>>> An example is available at https://github.com/pmix/prrte/bl >>>>>>> ob/m >>>>>>> aste >>>>>>> r/contrib/travis/test_client.sh >>>>>>> >>>>>>> >>>>>>> Cheers, >>>>>>> >>>>>>> Gilles >>>>>>> >>>>>>> >>>>>>> On 10/9/2018 8:45 AM, Stephan Krempel wrote: >>>>>>>> Hallo everyone, >>>>>>>> >>>>>>>> I am currently implementing a PMIx server and I try to >>>>>>>> use it >>>>>>>> with >>>>>>>> OpenMPI. I do have an own mpiexec which starts my PMIx >>>>>>>> server >>>>>>>> and >>>>>>>> launches the processes. >>>>>>>> >>>>>>>> If I launch an executable linked against OpenMPI, during >>>>>>>> MPI_Init() the >>>>>>>> ORTE layer starts another PMIx server and overrides my >>>>>>>> PMIX_* >>>>>>>> environment so this new server is used instead of mine. >>>>>>>> >>>>>>>> So I am looking for a method to prevent orte(d) from >>>>>>>> starting >>>>>>>> a >>>>>>>> PMIx >>>>>>>> server. >>>>>>>> >>>>>>>> I already tried to understand what the slurm support is >>>>>>>> doing, >>>>>>>> since >>>>>>>> this is (at least in parts) what I think I need. Somehow >>>>>>>> when >>>>>>>> starting >>>>>>>> a job with srun --mpi=pmix_v2 the ess module pmi is >>>>>>>> started, >>>>>>>> but >>>>>>>> I was >>>>>>>> not able to enforce that manually by setting an MCA >>>>>>>> parameter >>>>>>>> (oss >>>>>>>> should be the correct one?!?) >>>>>>>> And I do not yet have a clue how the slurm support is >>>>>>>> working. >>>>>>>> >>>>>>>> So does anyone has a hint for me where I can find >>>>>>>> documentation >>>>>>>> or >>>>>>>> information concerning that or is there an easy way to >>>>>>>> achieve >>>>>>>> what I >>>>>>>> am trying to do that I missed? >>>>>>>> >>>>>>>> Thank you in advance. >>>>>>>> >>>>>>>> Regards, >>>>>>>> >>>>>>>> Stephan >>>>>>>> _______________________________________________ >>>>>>>> devel mailing list >>>>>>>> devel@lists.open-mpi.org >>>>>>>> https://lists.open-mpi.org/mailman/listinfo/devel >>>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> devel mailing list >>>>>>> devel@lists.open-mpi.org >>>>>>> https://lists.open-mpi.org/mailman/listinfo/devel >>>>>> >>>>>> _______________________________________________ >>>>>> devel mailing list >>>>>> devel@lists.open-mpi.org >>>>>> https://lists.open-mpi.org/mailman/listinfo/devel >>>>> >>>>> _______________________________________________ >>>>> devel mailing list >>>>> devel@lists.open-mpi.org >>>>> https://lists.open-mpi.org/mailman/listinfo/devel >>>> >>>> _______________________________________________ >>>> devel mailing list >>>> devel@lists.open-mpi.org >>>> https://lists.open-mpi.org/mailman/listinfo/devel >>> >>> _______________________________________________ >>> devel mailing list >>> devel@lists.open-mpi.org >>> https://lists.open-mpi.org/mailman/listinfo/devel >> >> _______________________________________________ >> devel mailing list >> devel@lists.open-mpi.org >> https://lists.open-mpi.org/mailman/listinfo/devel > -- > -- > Stephan Krempel > HPC Software Engineer > > ParTec Cluster Competence Center GmbH > Possartstraße 20 > 81679 München, Germany_______________________________________________ > devel mailing list > devel@lists.open-mpi.org <mailto:devel@lists.open-mpi.org> > https://lists.open-mpi.org/mailman/listinfo/devel > <https://lists.open-mpi.org/mailman/listinfo/devel>
_______________________________________________ devel mailing list devel@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/devel
