The ubuntu package is here: https://packages.ubuntu.com/focal/libpmix-dev
Yes, we rewrote the service files (see here<https://github.com/NVIDIA/nephele-packages/blob/master/slurm/debian/PACKAGE-control.slurmctld.service>) and we let debhelper install them to the appropriate location. It seems like you’re wanting to simply get a development build going rather than building packages for distribution. Nonetheless, reading through the packaging files here might help because it shows how to build recent slurm on recent ubuntu/Debian: https://github.com/NVIDIA/nephele-packages/tree/master/slurm From: slurm-users <[email protected]> On Behalf Of Avery Grieve Sent: Thursday, December 10, 2020 10:18 AM To: Slurm User Community List <[email protected]> Subject: Re: [slurm-users] slurm-wlm package OpenMPI PMIx implementation External email: Use caution opening links or attachments Hey Luke, Thanks for the response. I should have mentioned I'm on debian. What's the name of the ubuntu package for pmix? I'll see if I can track down the debian equivalent. When you build slurm from scratch you have to place the .service files into /etc/init.d and the daemon files in /etc/systemd/system, right? When I tried building from source it didn't do that for me (even as root). Not sure if intended or if I was missing something. Thanks -ave On Thu, Dec 10, 2020, 1:11 PM Luke Yeager <[email protected]<mailto:[email protected]>> wrote: Hi Avery, * pmix: we just use the standard Ubuntu packages on 20.04. Unfortunately the standard packages on 18.04 are too out of date for us. * openmpi: we build our own, using ./configure --with-pmix=internal … * slurm: we build our own, using ./configure --with-pmix=PATH … (see here<https://github.com/NVIDIA/nephele-packages/blob/42145aef4bbe2cff335a1fca222766232dab7aa7/slurm/debian/rules#L41>) Then we can set MpiDefault=pmix (see here<https://github.com/NVIDIA/nephele/blob/1d79977164d5ef1418466bfb322d59d502c18e8f/ansible/roles/slurm/templates/etc/slurm/slurm.conf.default#L87>) and it works. $ srun --mpi=list srun: MPI types are... srun: cray_shasta srun: pmi2 srun: pmix_v3 srun: pmix srun: none Hope that helps, Luke From: slurm-users <[email protected]<mailto:[email protected]>> On Behalf Of Avery Grieve Sent: Thursday, December 10, 2020 7:52 AM To: [email protected]<mailto:[email protected]> Subject: [slurm-users] slurm-wlm package OpenMPI PMIx implementation External email: Use caution opening links or attachments Hi Forum, I've been putting together an ARM cluster for fun/learning and I've been a bit lost about how to get OpenMPI and slurm to behave together. I have installed the slurm-wlm package <https://packages.debian.org/buster/slurm-wlm> from the Debian apt search and compiled OpenMPI from source on my compute nodes. OpenMPI has been compiled with the option --with-slurm and the configure time log indicates openmpi has pmix v3 built in. I thought that would be enough for slurm and calling a job with "srun -n 4 -N1 executable" (with slurm.conf having MpiDefault=pmix_v3) would be enough. Not the case, unfortunately as slurm doesn't have any idea what pmix_v3 means without being compiled against it I guess. I have also attempted to compile openmpi from source with the --with-pmi option but the slurm-wlm package doesn't install any of the libraries/headers (pmi.h pmi2.h pmix.h etc). Neither does any of the slurm-llnl develop packages, so I'm at a loss of what to do here. A few notes: OpenMPI is working across my compute nodes. I'm able to ssh to my compute node and start a job manually with mpirun that executes successfully across the nodes. My slurmctld and slurmd daemons work for single thread resource allocation (and presumably OpenMP multithreading, though I haven't tested this). Beyond compiling slurm from source (assuming this installs the pmi headers that I can use to build openmpi), which I have tried with no luck on my devices, is there a way to get slurm and openmpi to behave together using the precompiled package slurm-wlm? Thank you, ~Avery Grieve They/Them/Theirs please! University of Michigan
