On Wed, Jul 24, 2019 at 09:46:13PM +0000, Jeff Squyres (jsquyres) wrote: > On Jul 24, 2019, at 5:16 PM, Ralph Castain via users > <users@lists.open-mpi.org> wrote: > > > > It doesn't work that way, as you discovered. You need to add this > > information at the same place where vader currently calls modex send, and > > then retrieve it at the same place vader currently calls modex recv. Those > > macros don't do an immediate send/recv like you are thinking - the send > > simply adds the value to an aggregated payload, then the "fence" call > > distributes that payload to everyone, and then the read extracts the > > requested piece from that payload. > > Just to expand on what Ralph said, think of it like this: > > 1. each component/module does a modex "send", which just memcopies the data > blob > > 2. the "fence()" is deep within ompi_mpi_init(), which does the actual data > exchange of all the module blobs in an efficient manner > > 3. each component/module can then later do a modex "receive", which just > memcopies the relevant blob from the module blobs that were actually received > in step #2
Thanks for the clarifications how this works. I was expecting something completely different. > (BTW, "modex" = "module exchange") Ah, also good to know. I was not really sure what it means. I opened a pull request completely without modex : https://github.com/open-mpi/ompi/pull/6844 I would have preferred to have the user namespace detection earlier than the actual {put,get}(), but I was not able to make it work. I hope I included all relevant information about my changes in the PR. Let's continue the discussion there. Thanks everyone for the support. Adrian > >> On Jul 24, 2019, at 5:23 AM, Adrian Reber <adr...@lisas.de> wrote: > >> > >> On Mon, Jul 22, 2019 at 04:30:50PM +0000, Ralph Castain wrote: > >>>> On Jul 22, 2019, at 9:20 AM, Adrian Reber <adr...@lisas.de> wrote: > >>>> > >>>> I have most of the code ready, but I still have troubles doing > >>>> OPAL_MODEX_RECV. I am using the following lines, based on the code from > >>>> orte/test/mpi/pmix.c: > >>>> > >>>> OPAL_MODEX_SEND_VALUE(rc, OPAL_PMIX_LOCAL, "user_ns_id", &value, > >>>> OPAL_INT); > >>>> > >>>> This sets rc to 0. For receiving: > >>>> > >>>> OPAL_MODEX_RECV_VALUE(rc, "user_ns_id", &wildcard_rank, &ptr, OPAL_INT); > >>> > >>> You need to replace "wildcard_rank" with the process name of the proc who > >>> published the "user_ns_id" key. If/when we have mpirun provide the value, > >>> then you can retrieve it from the wildcard rank as it will be coming from > >>> the system and not an application proc > >> > >> So I can get the user namespace ID from all involved processes back to > >> the main process (MCA_BTL_VADER_LOCAL_RANK == 0). But now only this > >> process knows that the user namespace IDs are different and I have > >> trouble using MODEX to send the information (do not use cma) back to the > >> other involved processes. It seems am not able to used MODEX_{SEND,RECV} > >> at the same time. One process sent and waits then on receive from the > >> other processes. Something like this works > >> > >> PROC 0 PROC 1 > >> recv() sent() > >> > >> > >> Bit this does not work: > >> > >> PROC 0 PROC 1 > >> recv() sent() > >> sent() recv() > >> > >> If I start the recv() immediately after the send() on PROC 1 no messages > >> are delivered anymore and everything hangs, even if different MODEX keys > >> are used. It seems like MODEX can not fetch messages in another order > >> than it was sent. Is that so? > >> > >> Not sure how to tell the other processes to not use CMA, while some > >> processes are still transmitting their user namespace ID to PROC 0. > >> > >> Adrian > >> > >>>> and rc is always set to -13. Is this how it is supposed to work, or do I > >>>> have to do it differently? > >>>> > >>>> Adrian > >>>> > >>>> On Mon, Jul 22, 2019 at 02:03:20PM +0000, Ralph Castain via users wrote: > >>>>> If that works, then it might be possible to include the namespace ID in > >>>>> the job-info provided by PMIx at startup - would have to investigate, > >>>>> so please confirm that the modex option works first. > >>>>> > >>>>>> On Jul 22, 2019, at 1:22 AM, Gilles Gouaillardet via users > >>>>>> <users@lists.open-mpi.org> wrote: > >>>>>> > >>>>>> Adrian, > >>>>>> > >>>>>> > >>>>>> An option is to involve the modex. > >>>>>> > >>>>>> each task would OPAL_MODEX_SEND() its own namespace ID, and then > >>>>>> OPAL_MODEX_RECV() > >>>>>> > >>>>>> the one from its peers and decide whether CMA support can be enabled. > >>>>>> > >>>>>> > >>>>>> Cheers, > >>>>>> > >>>>>> > >>>>>> Gilles > >>>>>> > >>>>>> On 7/22/2019 4:53 PM, Adrian Reber via users wrote: > >>>>>>> I had a look at it and not sure if it really makes sense. > >>>>>>> > >>>>>>> In btl_vader_{put,get}.c it would be easy to check for the user > >>>>>>> namespace ID of the other process, but the function would then just > >>>>>>> return OPAL_ERROR a bit earlier instead of as a result of > >>>>>>> process_vm_{read,write}v(). Nothing would really change. > >>>>>>> > >>>>>>> A better place for the check would be > >>>>>>> mca_btl_vader_check_single_copy() > >>>>>>> but I do not know if at this point the PID of the other processes is > >>>>>>> already known. Not sure if I can check for the user namespace ID of > >>>>>>> the > >>>>>>> other processes. > >>>>>>> > >>>>>>> Any recommendations how to do this? > >>>>>>> > >>>>>>> Adrian > >>>>>>> > >>>>>>> On Sun, Jul 21, 2019 at 03:08:01PM -0400, Nathan Hjelm wrote: > >>>>>>>> Patches are always welcome. What would be great is a nice big > >>>>>>>> warning that CMA support is disabled because the processes are on > >>>>>>>> different namespaces. Ideally all MPI processes should be on the > >>>>>>>> same namespace to ensure the best performance. > >>>>>>>> > >>>>>>>> -Nathan > >>>>>>>> > >>>>>>>>> On Jul 21, 2019, at 2:53 PM, Adrian Reber via users > >>>>>>>>> <users@lists.open-mpi.org> wrote: > >>>>>>>>> > >>>>>>>>> For completeness I am mentioning my results also here. > >>>>>>>>> > >>>>>>>>> To be able to mount file systems in the container it can only work > >>>>>>>>> if > >>>>>>>>> user namespaces are used and even if the user IDs are all the same > >>>>>>>>> (in > >>>>>>>>> each container and on the host), to be able to ptrace the kernel > >>>>>>>>> also > >>>>>>>>> checks if the processes are in the same user namespace (in addition > >>>>>>>>> to > >>>>>>>>> being owned by the same user). This check - same user namespace - > >>>>>>>>> fails > >>>>>>>>> and so process_vm_readv() and process_vm_writev() will also fail. > >>>>>>>>> > >>>>>>>>> So Open MPI's checks are currently not enough to detect if 'cma' > >>>>>>>>> can be > >>>>>>>>> used. Checking for the same user namespace would also be necessary. > >>>>>>>>> > >>>>>>>>> Is this a use case important enough to accept a patch for it? > >>>>>>>>> > >>>>>>>>> Adrian > >>>>>>>>> > >>>>>>>>>> On Fri, Jul 12, 2019 at 03:42:15PM +0200, Adrian Reber via users > >>>>>>>>>> wrote: > >>>>>>>>>> Gilles, > >>>>>>>>>> > >>>>>>>>>> thanks again. Adding '--mca btl_vader_single_copy_mechanism none' > >>>>>>>>>> helps > >>>>>>>>>> indeed. > >>>>>>>>>> > >>>>>>>>>> The default seems to be 'cma' and that seems to use > >>>>>>>>>> process_vm_readv() > >>>>>>>>>> and process_vm_writev(). That seems to require CAP_SYS_PTRACE, but > >>>>>>>>>> telling Podman to give the process CAP_SYS_PTRACE with > >>>>>>>>>> '--cap-add=SYS_PTRACE' > >>>>>>>>>> does not seem to be enough. Not sure yet if this related to the > >>>>>>>>>> fact > >>>>>>>>>> that Podman is running rootless. I will continue to investigate, > >>>>>>>>>> but now > >>>>>>>>>> I know where to look. Thanks! > >>>>>>>>>> > >>>>>>>>>> Adrian > >>>>>>>>>> > >>>>>>>>>>> On Fri, Jul 12, 2019 at 06:48:59PM +0900, Gilles Gouaillardet via > >>>>>>>>>>> users wrote: > >>>>>>>>>>> Adrian, > >>>>>>>>>>> > >>>>>>>>>>> Can you try > >>>>>>>>>>> mpirun --mca btl_vader_copy_mechanism none ... > >>>>>>>>>>> > >>>>>>>>>>> Please double check the MCA parameter name, I am AFK > >>>>>>>>>>> > >>>>>>>>>>> IIRC, the default copy mechanism used by vader directly accesses > >>>>>>>>>>> the remote process address space, and this requires some > >>>>>>>>>>> permission (ptrace?) that might be dropped by podman. > >>>>>>>>>>> > >>>>>>>>>>> Note Open MPI might not detect both MPI tasks run on the same > >>>>>>>>>>> node because of podman. > >>>>>>>>>>> If you use UCX, then btl/vader is not used at all (pml/ucx is > >>>>>>>>>>> used instead) > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Cheers, > >>>>>>>>>>> > >>>>>>>>>>> Gilles > >>>>>>>>>>> > >>>>>>>>>>> Sent from my iPod > >>>>>>>>>>> > >>>>>>>>>>>> On Jul 12, 2019, at 18:33, Adrian Reber via users > >>>>>>>>>>>> <users@lists.open-mpi.org> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> So upstream Podman was really fast and merged a PR which makes my > >>>>>>>>>>>> wrapper unnecessary: > >>>>>>>>>>>> > >>>>>>>>>>>> Add support for --env-host : > >>>>>>>>>>>> https://github.com/containers/libpod/pull/3557 > >>>>>>>>>>>> > >>>>>>>>>>>> As commented in the PR I can now start mpirun with Podman > >>>>>>>>>>>> without a > >>>>>>>>>>>> wrapper: > >>>>>>>>>>>> > >>>>>>>>>>>> $ mpirun --hostfile ~/hosts --mca orte_tmpdir_base > >>>>>>>>>>>> /tmp/podman-mpirun podman run --env-host --security-opt > >>>>>>>>>>>> label=disable -v /tmp/podman-mpirun:/tmp/podman-mpirun > >>>>>>>>>>>> --userns=keep-id --net=host mpi-test /home/mpi/ring > >>>>>>>>>>>> Rank 0 has cleared MPI_Init > >>>>>>>>>>>> Rank 1 has cleared MPI_Init > >>>>>>>>>>>> Rank 0 has completed ring > >>>>>>>>>>>> Rank 0 has completed MPI_Barrier > >>>>>>>>>>>> Rank 1 has completed ring > >>>>>>>>>>>> Rank 1 has completed MPI_Barrier > >>>>>>>>>>>> > >>>>>>>>>>>> This is example was using TCP and on an InfiniBand based system > >>>>>>>>>>>> I have > >>>>>>>>>>>> to map the InfiniBand devices into the container. > >>>>>>>>>>>> > >>>>>>>>>>>> $ mpirun --mca btl ^openib --hostfile ~/hosts --mca > >>>>>>>>>>>> orte_tmpdir_base /tmp/podman-mpirun podman run --env-host -v > >>>>>>>>>>>> /tmp/podman-mpirun:/tmp/podman-mpirun --security-opt > >>>>>>>>>>>> label=disable --userns=keep-id --device /dev/infiniband/uverbs0 > >>>>>>>>>>>> --device /dev/infiniband/umad0 --device /dev/infiniband/rdma_cm > >>>>>>>>>>>> --net=host mpi-test /home/mpi/ring > >>>>>>>>>>>> Rank 0 has cleared MPI_Init > >>>>>>>>>>>> Rank 1 has cleared MPI_Init > >>>>>>>>>>>> Rank 0 has completed ring > >>>>>>>>>>>> Rank 0 has completed MPI_Barrier > >>>>>>>>>>>> Rank 1 has completed ring > >>>>>>>>>>>> Rank 1 has completed MPI_Barrier > >>>>>>>>>>>> > >>>>>>>>>>>> This is all running without root and only using Podman's rootless > >>>>>>>>>>>> support. > >>>>>>>>>>>> > >>>>>>>>>>>> Running multiple processes on one system, however, still gives > >>>>>>>>>>>> me an > >>>>>>>>>>>> error. If I disable vader I guess that Open MPI is using TCP for > >>>>>>>>>>>> localhost communication and that works. But with vader it fails. > >>>>>>>>>>>> > >>>>>>>>>>>> The first error message I get is a segfault: > >>>>>>>>>>>> > >>>>>>>>>>>> [test1:00001] *** Process received signal *** > >>>>>>>>>>>> [test1:00001] Signal: Segmentation fault (11) > >>>>>>>>>>>> [test1:00001] Signal code: Address not mapped (1) > >>>>>>>>>>>> [test1:00001] Failing at address: 0x7fb7b1552010 > >>>>>>>>>>>> [test1:00001] [ 0] > >>>>>>>>>>>> /lib64/libpthread.so.0(+0x12d80)[0x7f6299456d80] > >>>>>>>>>>>> [test1:00001] [ 1] > >>>>>>>>>>>> /usr/lib64/openmpi/lib/openmpi/mca_btl_vader.so(mca_btl_vader_send+0x3db)[0x7f628b33ab0b] > >>>>>>>>>>>> [test1:00001] [ 2] > >>>>>>>>>>>> /usr/lib64/openmpi/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_send_request_start_rdma+0x1fb)[0x7f62901d24bb] > >>>>>>>>>>>> [test1:00001] [ 3] > >>>>>>>>>>>> /usr/lib64/openmpi/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_send+0xfd6)[0x7f62901be086] > >>>>>>>>>>>> [test1:00001] [ 4] > >>>>>>>>>>>> /usr/lib64/openmpi/lib/libmpi.so.40(PMPI_Send+0x1bd)[0x7f62996f862d] > >>>>>>>>>>>> [test1:00001] [ 5] /home/mpi/ring[0x400b76] > >>>>>>>>>>>> [test1:00001] [ 6] > >>>>>>>>>>>> /lib64/libc.so.6(__libc_start_main+0xf3)[0x7f62990a3813] > >>>>>>>>>>>> [test1:00001] [ 7] /home/mpi/ring[0x4008be] > >>>>>>>>>>>> [test1:00001] *** End of error message *** > >>>>>>>>>>>> > >>>>>>>>>>>> Guessing that vader uses shared memory this is expected to fail, > >>>>>>>>>>>> with > >>>>>>>>>>>> all the namespace isolations in place. Maybe not with a > >>>>>>>>>>>> segfault, but > >>>>>>>>>>>> each container has its own shared memory. So next step was to > >>>>>>>>>>>> use the > >>>>>>>>>>>> host's ipc and pid namespace and mount /dev/shm: > >>>>>>>>>>>> > >>>>>>>>>>>> '-v /dev/shm:/dev/shm --ipc=host --pid=host' > >>>>>>>>>>>> > >>>>>>>>>>>> Which does not segfault, but still does not look correct: > >>>>>>>>>>>> > >>>>>>>>>>>> Rank 0 has cleared MPI_Init > >>>>>>>>>>>> Rank 1 has cleared MPI_Init > >>>>>>>>>>>> Rank 2 has cleared MPI_Init > >>>>>>>>>>>> [test1:17722] Read -1, expected 80000, errno = 1 > >>>>>>>>>>>> [test1:17722] Read -1, expected 80000, errno = 1 > >>>>>>>>>>>> [test1:17722] Read -1, expected 80000, errno = 1 > >>>>>>>>>>>> [test1:17722] Read -1, expected 80000, errno = 1 > >>>>>>>>>>>> [test1:17722] Read -1, expected 80000, errno = 1 > >>>>>>>>>>>> [test1:17722] Read -1, expected 80000, errno = 1 > >>>>>>>>>>>> [test1:17722] Read -1, expected 80000, errno = 1 > >>>>>>>>>>>> [test1:17722] Read -1, expected 80000, errno = 1 > >>>>>>>>>>>> [test1:17722] Read -1, expected 80000, errno = 1 > >>>>>>>>>>>> [test1:17722] Read -1, expected 80000, errno = 1 > >>>>>>>>>>>> [test1:17722] Read -1, expected 80000, errno = 1 > >>>>>>>>>>>> Rank 0 has completed ring > >>>>>>>>>>>> Rank 2 has completed ring > >>>>>>>>>>>> Rank 0 has completed MPI_Barrier > >>>>>>>>>>>> Rank 1 has completed ring > >>>>>>>>>>>> Rank 2 has completed MPI_Barrier > >>>>>>>>>>>> Rank 1 has completed MPI_Barrier > >>>>>>>>>>>> > >>>>>>>>>>>> This is using the Open MPI ring.c example with SIZE increased > >>>>>>>>>>>> from 20 to 20000. > >>>>>>>>>>>> > >>>>>>>>>>>> Any recommendations what vader needs to communicate correctly? > >>>>>>>>>>>> > >>>>>>>>>>>> Adrian > >>>>>>>>>>>> > >>>>>>>>>>>>> On Thu, Jul 11, 2019 at 12:07:35PM +0200, Adrian Reber via > >>>>>>>>>>>>> users wrote: > >>>>>>>>>>>>> Gilles, > >>>>>>>>>>>>> > >>>>>>>>>>>>> thanks for pointing out the environment variables. I quickly > >>>>>>>>>>>>> created a > >>>>>>>>>>>>> wrapper which tells Podman to re-export all OMPI_ and PMIX_ > >>>>>>>>>>>>> variables > >>>>>>>>>>>>> (grep "\(PMIX\|OMPI\)"). Now it works: > >>>>>>>>>>>>> > >>>>>>>>>>>>> $ mpirun --hostfile ~/hosts ./wrapper -v /tmp:/tmp > >>>>>>>>>>>>> --userns=keep-id --net=host mpi-test /home/mpi/hello > >>>>>>>>>>>>> > >>>>>>>>>>>>> Hello, world (2 procs total) > >>>>>>>>>>>>> --> Process # 0 of 2 is alive. ->test1 > >>>>>>>>>>>>> --> Process # 1 of 2 is alive. ->test2 > >>>>>>>>>>>>> > >>>>>>>>>>>>> I need to tell Podman to mount /tmp from the host into the > >>>>>>>>>>>>> container, as > >>>>>>>>>>>>> I am running rootless I also need to tell Podman to use the > >>>>>>>>>>>>> same user ID > >>>>>>>>>>>>> in the container as outside (so that the Open MPI files in > >>>>>>>>>>>>> /tmp) can be > >>>>>>>>>>>>> shared and I am also running without a network namespace. > >>>>>>>>>>>>> > >>>>>>>>>>>>> So this is now with the full Podman provided isolation except > >>>>>>>>>>>>> the > >>>>>>>>>>>>> network namespace. Thanks for you help! > >>>>>>>>>>>>> > >>>>>>>>>>>>> Adrian > >>>>>>>>>>>>> > >>>>>>>>>>>>>> On Thu, Jul 11, 2019 at 04:47:21PM +0900, Gilles Gouaillardet > >>>>>>>>>>>>>> via users wrote: > >>>>>>>>>>>>>> Adrian, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> the MPI application relies on some environment variables (they > >>>>>>>>>>>>>> typically > >>>>>>>>>>>>>> start with OMPI_ and PMIX_). > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> The MPI application internally uses a PMIx client that must be > >>>>>>>>>>>>>> able to > >>>>>>>>>>>>>> contact a PMIx server > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> (that is included in mpirun and the orted daemon(s) spawned on > >>>>>>>>>>>>>> the remote > >>>>>>>>>>>>>> hosts). > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> located on the same host. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> If podman provides some isolation between the app inside the > >>>>>>>>>>>>>> container (e.g. > >>>>>>>>>>>>>> /home/mpi/hello) > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> and the outside world (e.g. mpirun/orted), that won't be an > >>>>>>>>>>>>>> easy ride. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Cheers, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Gilles > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On 7/11/2019 4:35 PM, Adrian Reber via users wrote: > >>>>>>>>>>>>>>> I did a quick test to see if I can use Podman in combination > >>>>>>>>>>>>>>> with Open > >>>>>>>>>>>>>>> MPI: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> [test@test1 ~]$ mpirun --hostfile ~/hosts podman run > >>>>>>>>>>>>>>> quay.io/adrianreber/mpi-test /home/mpi/hello > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Hello, world (1 procs total) > >>>>>>>>>>>>>>> --> Process # 0 of 1 is alive. ->789b8fb622ef > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Hello, world (1 procs total) > >>>>>>>>>>>>>>> --> Process # 0 of 1 is alive. ->749eb4e1c01a > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> The test program (hello) is taken from > >>>>>>>>>>>>>>> https://raw.githubusercontent.com/openhpc/ohpc/obs/OpenHPC_1.3.8_Factory/tests/mpi/hello.c > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> The problem with this is that each process thinks it is > >>>>>>>>>>>>>>> process 0 of 1 > >>>>>>>>>>>>>>> instead of > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Hello, world (2 procs total) > >>>>>>>>>>>>>>> --> Process # 1 of 2 is alive. ->test1 > >>>>>>>>>>>>>>> --> Process # 0 of 2 is alive. ->test2 > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> My questions is how is the rank determined? What resources do > >>>>>>>>>>>>>>> I need to have > >>>>>>>>>>>>>>> in my container to correctly determine the rank. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> This is Podman 1.4.2 and Open MPI 4.0.1. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Adrian > >>>>>>>>>>>>>>> _______________________________________________ > >>>>>>>>>>>>>>> users mailing list > >>>>>>>>>>>>>>> users@lists.open-mpi.org > >>>>>>>>>>>>>>> https://lists.open-mpi.org/mailman/listinfo/users > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> _______________________________________________ > >>>>>>>>>>>>>> users mailing list > >>>>>>>>>>>>>> users@lists.open-mpi.org > >>>>>>>>>>>>>> https://lists.open-mpi.org/mailman/listinfo/users > >>>>>>>>>>>>> _______________________________________________ > >>>>>>>>>>>>> users mailing list > >>>>>>>>>>>>> users@lists.open-mpi.org > >>>>>>>>>>>>> https://lists.open-mpi.org/mailman/listinfo/users > >>>>>>>>>>>> _______________________________________________ > >>>>>>>>>>>> users mailing list > >>>>>>>>>>>> users@lists.open-mpi.org > >>>>>>>>>>>> https://lists.open-mpi.org/mailman/listinfo/users > >>>>>>>>>>> _______________________________________________ > >>>>>>>>>>> users mailing list > >>>>>>>>>>> users@lists.open-mpi.org > >>>>>>>>>>> https://lists.open-mpi.org/mailman/listinfo/users > >>>>>>>>>> _______________________________________________ > >>>>>>>>>> users mailing list > >>>>>>>>>> users@lists.open-mpi.org > >>>>>>>>>> https://lists.open-mpi.org/mailman/listinfo/users > >>>>>>>>> _______________________________________________ > >>>>>>>>> users mailing list > >>>>>>>>> users@lists.open-mpi.org > >>>>>>>>> https://lists.open-mpi.org/mailman/listinfo/users > >>>>>>> Adrian > >>>>>>> > >>>>>> _______________________________________________ > >>>>>> users mailing list > >>>>>> users@lists.open-mpi.org > >>>>>> https://lists.open-mpi.org/mailman/listinfo/users > >>>>> > >>>>> > >>>>> _______________________________________________ > >>>>> users mailing list > >>>>> users@lists.open-mpi.org > >>>>> https://lists.open-mpi.org/mailman/listinfo/users > > > > > > _______________________________________________ > > users mailing list > > users@lists.open-mpi.org > > https://lists.open-mpi.org/mailman/listinfo/users _______________________________________________ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
