Re: [OMPI users] RSH agent working but no output
Dear Ralph, I am using Open MPI 1.10. I will recompile with "—enable-debug" and do as you suggested. I get back in touch with more information soon. -- Mr. Filippo SPIGA, M.Sc. http://fspiga.github.io ~ skype: filippo.spiga «Nobody will drive us out of Cantor's paradise.» ~ David Hilbert * Disclaimer: "Please note this message and any attachments are CONFIDENTIAL and may be privileged or otherwise protected from disclosure. The contents are not to be disclosed to anyone other than the addressee. Unauthorized recipients are requested to preserve this confidentiality and to advise the sender immediately of any error in transmission." > On Sep 27, 2015, at 6:08 PM, Ralph Castain <r...@open-mpi.org> wrote: > > Is this a debug build of OMPI? In other words, was it configured with > —enable-debug? If so, you can get some further output to see what is causing > the problem by adding the following to your cmd line: > > —leave-session-attached —mca plm_base_verbose 5 —mca state_base_verbose 5 > > BTW: what version of OMPI are you using? > > >> On Sep 27, 2015, at 8:47 AM, Filippo Spiga <spiga.fili...@gmail.com> wrote: >> >> Hello, >> >> I am trying to run OpenMPI on a cluster where I am only allow to use RSH to >> spawn processes across compute nodes. I modified my mpirun command in this >> way: >> >> mpirun -np 48 --hostfile ./hostfile --mca plm_rsh_agent "rsh" >> /central_storage/osu_barrier >> >> the file "hostfile" contain the machine where I am running mpirun plus >> another node. Libraries and osu_barrier are located in a central storage >> mounted by all nodes. The processes osu_barrier are spawn correctly across >> two nodes but no output is returned and the command seems stuck. Any idea >> what can I look at to try to solve this issue? >> >> Thanks >> >> -- >> Mr. Filippo SPIGA, M.Sc. >> http://fspiga.github.io ~ skype: filippo.spiga >> >> «Nobody will drive us out of Cantor's paradise.» ~ David Hilbert >> >> * >> Disclaimer: "Please note this message and any attachments are CONFIDENTIAL >> and may be privileged or otherwise protected from disclosure. The contents >> are not to be disclosed to anyone other than the addressee. Unauthorized >> recipients are requested to preserve this confidentiality and to advise the >> sender immediately of any error in transmission." >> >> >> ___ >> users mailing list >> us...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> Link to this post: >> http://www.open-mpi.org/community/lists/users/2015/09/27683.php > > ___ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2015/09/27684.php
[OMPI users] RSH agent working but no output
Hello, I am trying to run OpenMPI on a cluster where I am only allow to use RSH to spawn processes across compute nodes. I modified my mpirun command in this way: mpirun -np 48 --hostfile ./hostfile --mca plm_rsh_agent "rsh" /central_storage/osu_barrier the file "hostfile" contain the machine where I am running mpirun plus another node. Libraries and osu_barrier are located in a central storage mounted by all nodes. The processes osu_barrier are spawn correctly across two nodes but no output is returned and the command seems stuck. Any idea what can I look at to try to solve this issue? Thanks -- Mr. Filippo SPIGA, M.Sc. http://fspiga.github.io ~ skype: filippo.spiga «Nobody will drive us out of Cantor's paradise.» ~ David Hilbert * Disclaimer: "Please note this message and any attachments are CONFIDENTIAL and may be privileged or otherwise protected from disclosure. The contents are not to be disclosed to anyone other than the addressee. Unauthorized recipients are requested to preserve this confidentiality and to advise the sender immediately of any error in transmission."
[OMPI users] OpenMPI 1.10.0 and old SUSE SLES 11 SP3
Hello everybody, I desperately trying to compile on a SLES 11 SP3 OpenMPI 1.10.0 with MXM and FCA support. export MXM_DIR=/opt/mellanox/mxm export KNEM_DIR=$(find /opt -maxdepth 1 -type d -name "knem*" -print0) export FCA_DIR=/opt/mellanox/fca export HCOLL_DIR=/opt/mellanox/hcoll rm -rf build mkdir build cd build ../configure --prefix=$FINAL_DIR --with-lsf --enable-mca-no-build=btl-usnic --with-verbs --with-hwloc=internal --with-fca=$FCA_DIR --with-mxm=$MXM_DIR --with-knem=$KNEM_DIR --enable-static I had to enable static otherwise MXM fails. Anyway, almost at the end it continues to fails because of -lz problem Creating mpi/man/man3/OpenMPI.3 man page... CCLD libmpi.la /usr/lib64/gcc/x86_64-suse-linux/4.3/../../../../x86_64-suse-linux/bin/ld: cannot find -lz collect2: ld returned 1 exit status make[2]: *** [libmpi.la] Error 1 make[2]: Leaving directory `/tmp/openmpi-1.10.0/build/ompi' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/tmp/openmpi-1.10.0/build/ompi' make: *** [all-recursive] Error 1 Any suggestion to overcome this issue? The kernel is not very old but the GCC is... $gcc --version gcc (SUSE Linux) 4.3.4 [gcc-4_3-branch revision 152973] Copyright (C) 2008 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. $ uname -a Linux testing 3.0.93-0.8-default #1 SMP Tue Aug 27 08:44:18 UTC 2013 (70ed288) x86_64 x86_64 x86_64 GNU/Linux I see libz.so under /lib64 ... -- Mr. Filippo SPIGA, M.Sc. http://fspiga.github.io ~ skype: filippo.spiga «Nobody will drive us out of Cantor's paradise.» ~ David Hilbert * Disclaimer: "Please note this message and any attachments are CONFIDENTIAL and may be privileged or otherwise protected from disclosure. The contents are not to be disclosed to anyone other than the addressee. Unauthorized recipients are requested to preserve this confidentiality and to advise the sender immediately of any error in transmission."
Re: [OMPI users] Missing file "openmpi/ompi/mpi/f77/constants.h"
Dear Jeff, dear Dave, thank you for your replies. I spent a bit more time on the issue, Open MPi is built correctly. The fact is that IPM is looking for a header files that does not exist anymore in the Open MPI 1.8 branch (but it possibly exists in older 1.x versions). So I made a change (basically a check over OPAL_MINOR_VERSION) to IPM source code and push for review to the main IPM repository so nobody else will have troubles with it in the future. Thanks again for your support Regards, Filippo On Jun 11, 2015, at 5:09 PM, Dave Love <d.l...@liverpool.ac.uk> wrote: > > Filippo Spiga <spiga.fili...@gmail.com> writes: > >> Dear OpenMPI experts, >> >> I am rebuilding IPM (https://github.com/nerscadmin/ipm) based on OpenMPI >> 1.8.5. However, despite OMPI is compiled with the "--with-devel-headers" >> option, IPM build fails because the file "openmpi/ompi/mpi/f77/constants.h" >> is missing > > Which version of IPM? 2.0.3 plus irrelevant patches builds against > Fedora 22's openmpi-1.8.4: > <http://copr.fedoraproject.org/coprs/loveshack/livhpc/build/97971/>. > ___ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2015/06/27097.php -- Mr. Filippo SPIGA, M.Sc. http://fspiga.github.io ~ skype: filippo.spiga «Nobody will drive us out of Cantor's paradise.» ~ David Hilbert * Disclaimer: "Please note this message and any attachments are CONFIDENTIAL and may be privileged or otherwise protected from disclosure. The contents are not to be disclosed to anyone other than the addressee. Unauthorized recipients are requested to preserve this confidentiality and to advise the sender immediately of any error in transmission."
[OMPI users] Missing file "openmpi/ompi/mpi/f77/constants.h"
Dear OpenMPI experts, I am rebuilding IPM (https://github.com/nerscadmin/ipm) based on OpenMPI 1.8.5. However, despite OMPI is compiled with the "--with-devel-headers" option, IPM build fails because the file "openmpi/ompi/mpi/f77/constants.h" is missing Shall I hack IPM because this is a deprecated file or I miss something to do during the instllation process of OMPI? Thank in advance. Regards, Filippo -- Mr. Filippo SPIGA, M.Sc. http://fspiga.github.io ~ skype: filippo.spiga «Nobody will drive us out of Cantor's paradise.» ~ David Hilbert * Disclaimer: "Please note this message and any attachments are CONFIDENTIAL and may be privileged or otherwise protected from disclosure. The contents are not to be disclosed to anyone other than the addressee. Unauthorized recipients are requested to preserve this confidentiality and to advise the sender immediately of any error in transmission."
Re: [OMPI users] Different HCA from different OpenMP threads (same rank using MPI_THREAD_MULTIPLE)
Thanks Rolf and Ralph for the replies! On Apr 6, 2015, at 10:37 PM, Ralph Castain <r...@open-mpi.org> wrote: > That said, you can certainly have your application specify a thread-level > binding. You’d have to do the heavy lifting yourself in the app, I’m afraid, > instead of relying on us to do it for you Ok, my application must do it and I am fine with it. But how? I mean, does Open MPi expose some API that allows such fine grain control? F -- Mr. Filippo SPIGA, M.Sc. http://filippospiga.info ~ skype: filippo.spiga «Nobody will drive us out of Cantor's paradise.» ~ David Hilbert * Disclaimer: "Please note this message and any attachments are CONFIDENTIAL and may be privileged or otherwise protected from disclosure. The contents are not to be disclosed to anyone other than the addressee. Unauthorized recipients are requested to preserve this confidentiality and to advise the sender immediately of any error in transmission."
[OMPI users] Different HCA from different OpenMP threads (same rank using MPI_THREAD_MULTIPLE)
Dear Open MPI developers, I wonder if there is a way to address this particular scenario using MPI_T or other strategies in Open MPI. I saw a similar discussion few days ago, I assume the same challenges are applied in this case but I just want to check. Here is the scenario: We have a system composed by dual rail Mellanox IB, two distinct Connect-IB cards per node each one sitting on a different PCI-E lane out of two distinct sockets. We are seeking a way to control MPI traffic thought each one of them directly into the application. In specific we have a single MPI rank per node that goes multi-threading using OpenMP. MPI_THREAD_MULTIPLE is used, each OpenMP thread may initiate MPI communication. We would like to assign IB-0 to thread 0 and IB-1 to thread 1. Via mpirun or env variables we can control which IB interface to use by binding it to a specific MPI rank (or by apply a policy that relate IB to MPi ranks). But if there is only one MPI rank active, how we can differentiate the traffic across multiple IB cards? Thanks in advance for any suggestion about this matter. Regards, Filippo -- Mr. Filippo SPIGA, M.Sc. http://filippospiga.info ~ skype: filippo.spiga «Nobody will drive us out of Cantor's paradise.» ~ David Hilbert * Disclaimer: "Please note this message and any attachments are CONFIDENTIAL and may be privileged or otherwise protected from disclosure. The contents are not to be disclosed to anyone other than the addressee. Unauthorized recipients are requested to preserve this confidentiality and to advise the sender immediately of any error in transmission."
Re: [OMPI users] Clarification about OpenMPI, slurm and PMI interface
Dear Mike, it sounds good... the description fits my purposes... I really miss
this when I was reading srun man page! I will give it a try
Thanks to everybody for the help and support!
F
> On Aug 21, 2014, at 7:58 PM, Mike Dubman <mi...@dev.mellanox.co.il> wrote:
>
> Hi FIlippo,
>
> I think you can use SLURM_LOCALID var (at least with slurm v14.03.4-2)
>
> $srun -N2 --ntasks-per-node 3 env |grep SLURM_LOCALID
> SLURM_LOCALID=1
> SLURM_LOCALID=2
> SLURM_LOCALID=0
> SLURM_LOCALID=0
> SLURM_LOCALID=1
> SLURM_LOCALID=2
> $
>
> Kind Regards,
> M
>
>
> On Thu, Aug 21, 2014 at 9:27 PM, Ralph Castain <r...@open-mpi.org> wrote:
>
> On Aug 21, 2014, at 10:58 AM, Filippo Spiga <spiga.fili...@gmail.com> wrote:
>
>> Dear Ralph
>>
>> On Aug 21, 2014, at 2:30 PM, Ralph Castain <r...@open-mpi.org> wrote:
>>> I'm afraid that none of the mapping or binding options would be available
>>> under srun as those only work via mpirun. You can pass MCA params in the
>>> environment of course, or in default MCA param files.
>>
>> I understand. I hopefully be able to still pass the LAMA mca options as
>> environment variables
>
> I'm afraid not - LAMA doesn't exist in Slurm, only in mpirun itself
>
>> I fear by default srun completely takes over the process binding.
>>
>>
>> I got another problem. On my cluster I have two GPU and two Ivy Bridge
>> processors. To maximize the PCIe bandwidth I want to allocate GPU 0 to
>> socket 0 and GPU 1 to socket 1. I use a script like this
>>
>> #!/bin/bash
>> lrank=$OMPI_COMM_WORLD_LOCAL_RANK
>> case ${lrank} in
>> 0)
>> export CUDA_VISIBLE_DEVICES=0
>> "$@"
>> ;;
>> 1)
>> export CUDA_VISIBLE_DEVICES=1
>> "$@"
>> ;;
>> esac
>>
>>
>> But OMP_COMM_WORLD_LOCAL_RANK is not defined is I use srun with PMI2 as
>> luncher. Is there any equivalent option/environment variable that will help
>> me achieve the same result?
>
> I'm afraid not - that's something we added. I'm unaware of any similar envar
> from Slurm, I'm afraid
>
>
>>
>> Thanks in advance!
>> F
>>
>> --
>> Mr. Filippo SPIGA, M.Sc.
>> http://filippospiga.info ~ skype: filippo.spiga
>>
>> «Nobody will drive us out of Cantor's paradise.» ~ David Hilbert
>>
>> *
>> Disclaimer: "Please note this message and any attachments are CONFIDENTIAL
>> and may be privileged or otherwise protected from disclosure. The contents
>> are not to be disclosed to anyone other than the addressee. Unauthorized
>> recipients are requested to preserve this confidentiality and to advise the
>> sender immediately of any error in transmission."
>>
>>
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post:
>> http://www.open-mpi.org/community/lists/users/2014/08/25119.php
>
>
> _______
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2014/08/25120.php
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2014/08/25121.php
--
Mr. Filippo SPIGA, M.Sc.
http://filippospiga.info ~ skype: filippo.spiga
«Nobody will drive us out of Cantor's paradise.» ~ David Hilbert
*
Disclaimer: "Please note this message and any attachments are CONFIDENTIAL and
may be privileged or otherwise protected from disclosure. The contents are not
to be disclosed to anyone other than the addressee. Unauthorized recipients are
requested to preserve this confidentiality and to advise the sender immediately
of any error in transmission."
Re: [OMPI users] Clarification about OpenMPI, slurm and PMI interface
Dear Ralph
On Aug 21, 2014, at 2:30 PM, Ralph Castain <r...@open-mpi.org> wrote:
> I'm afraid that none of the mapping or binding options would be available
> under srun as those only work via mpirun. You can pass MCA params in the
> environment of course, or in default MCA param files.
I understand. I hopefully be able to still pass the LAMA mca options as
environment variablesI fear by default srun completely takes over the
process binding.
I got another problem. On my cluster I have two GPU and two Ivy Bridge
processors. To maximize the PCIe bandwidth I want to allocate GPU 0 to socket 0
and GPU 1 to socket 1. I use a script like this
#!/bin/bash
lrank=$OMPI_COMM_WORLD_LOCAL_RANK
case ${lrank} in
0)
export CUDA_VISIBLE_DEVICES=0
"$@"
;;
1)
export CUDA_VISIBLE_DEVICES=1
"$@"
;;
esac
But OMP_COMM_WORLD_LOCAL_RANK is not defined is I use srun with PMI2 as
luncher. Is there any equivalent option/environment variable that will help me
achieve the same result?
Thanks in advance!
F
--
Mr. Filippo SPIGA, M.Sc.
http://filippospiga.info ~ skype: filippo.spiga
«Nobody will drive us out of Cantor's paradise.» ~ David Hilbert
*
Disclaimer: "Please note this message and any attachments are CONFIDENTIAL and
may be privileged or otherwise protected from disclosure. The contents are not
to be disclosed to anyone other than the addressee. Unauthorized recipients are
requested to preserve this confidentiality and to advise the sender immediately
of any error in transmission."
Re: [OMPI users] Clarification about OpenMPI, slurm and PMI interface
Hi Joshua,
On Aug 21, 2014, at 12:28 AM, Joshua Ladd <jladd.m...@gmail.com> wrote:
> When launching with mpirun in a SLURM environment, srun is only being used to
> launch the ORTE daemons (orteds.) Since the daemon will already exist on the
> node from which you invoked mpirun, this node will not be included in the
> list of nodes. SLURM's PMI library is not involved (that functionality is
> only necessary if you directly launch your MPI application with srun, in
> which case it is used to exchanged wireup info amongst slurmds.) This is the
> expected behavior.
>
> ~/ompi-top-level/orte/mca/plm/plm_slurm_module.c +294
> /* if the daemon already exists on this node, then
> * don't include it
> */
> if (node->daemon_launched) {
> continue;
> }
>
> Do you have a frontend node that you can launch from? What happens if you set
> "-np X" where X = 8*ppn. The alternative is to do a direct launch of the MPI
> application with srun.
I understand the logic and I understand with orted in the first node is not
needed. But since we use a batch system (SLURM) we do not want people to run
their mpirun commands directly fon a front-end. Typical scenario: all compute
node are running fine but we reboot all the login nodes to upgrade the linux
image because of a security update the kernel. We can keep the login nodes
offline potentially for hours without stop the system to work.
From our perspective, a front-end node is an additional burden. Of course login
node and front-end node can be two separated hosts but I am looking for a way
to keep our setup as-it-is without introducing structural changes.
Hi Ralph,
On Aug 21, 2014, at 12:36 AM, Ralph Castain <r...@open-mpi.org> wrote:
> Or you can add
>
>-nolocal|--nolocalDo not run any MPI applications on the local node
>
> to your mpirun command line and we won't run any application procs on the
> node where mpirun is executing
I tried but of course but mpirun complains. If it cannot run local (meaning on
the first node, tesla121) then only 7 nodes remains and I request in total 8.
So to use "--nolocal" I need to add another nodes. Since we allocate node
exclusively and for some users we charge the usage real money... this is not
ideal I am afraid.
srun seems the only solution to go. I need to understand how to pass most of
the --mca parameters to srun and to be sure I can pilot rmaps_lama_* options
as flexible as I do with normal mpirun. Then there are mxm, fca, hcollI am
not against srun in principle, my only stopping point it that the syntax is
only different that we might receive lot (too many) complains our users in
adopting this new way to submit because they are used to use classic mpirun
inside a sbatch script. Most of them will probably not switch to a different
method! So our hope to "silently" profile network, energy, I/O using SLURM
plugins also using Open MPI is lost...
F
--
Mr. Filippo SPIGA, M.Sc.
http://filippospiga.info ~ skype: filippo.spiga
«Nobody will drive us out of Cantor's paradise.» ~ David Hilbert
*
Disclaimer: "Please note this message and any attachments are CONFIDENTIAL and
may be privileged or otherwise protected from disclosure. The contents are not
to be disclosed to anyone other than the addressee. Unauthorized recipients are
requested to preserve this confidentiality and to advise the sender immediately
of any error in transmission."
[OMPI users] Clarification about OpenMPI, slurm and PMI interface
Dear Open MPI experts, I have a problem that is related to the integration of OpenMPI, slurm and PMI interface. I spent some time today with a colleague of mine trying to figure out why we were not able to obtain all H5 profile files (generated by acct_gather_profile) using Open MPI. When I say "all" I mean if I run using 8 nodes (e.g. tesla[121-128]) then I always systematically miss the file related to the first one (the first node in the allocation list, in this case tesla121). By comparing which processes are spawn on the compute nodes, I discovered that mpirun running on tesla121 calls srun only to spawn remotely new MPI processes to the other 7 nodes (maybe this is obvious, for me it was not)... fs395 617 0.0 0.0 106200 1504 ?S22:41 0:00 /bin/bash /var/spool/slurm-test/slurmd/job390044/slurm_script fs395 629 0.1 0.0 194552 5288 ?Sl 22:41 0:00 \_ mpirun -bind-to socket --map-by ppr:1:socket --host tesla121,tesla122,tesla123,tesla124,tesla125,tesla126,tes fs395 632 0.0 0.0 659740 9148 ?Sl 22:41 0:00 | \_ srun --ntasks-per-node=1 --kill-on-bad-exit --cpu_bind=none --nodes=7 --nodelist=tesla122,tesla123,tesla1 fs395 633 0.0 0.0 55544 1072 ?S22:41 0:00 | | \_ srun --ntasks-per-node=1 --kill-on-bad-exit --cpu_bind=none --nodes=7 --nodelist=tesla122,tesla123,te fs395 651 0.0 0.0 106072 1392 ?S22:41 0:00 | \_ /bin/bash ./run_linpack ./xhpl fs395 654 295 35.5 120113412 23289280 ? RLl 22:41 3:12 | | \_ ./xhpl fs395 652 0.0 0.0 106072 1396 ?S22:41 0:00 | \_ /bin/bash ./run_linpack ./xhpl fs395 656 307 35.5 120070332 23267728 ? RLl 22:41 3:19 | \_ ./xhpl The "xhpl" processes allocated on the first node of a job are not called by srun and because of this the SLURM profile plugin is not activated on the node!!! As result I always miss the first node profile information. Intel MPI does not have this behavior, mpiexec.hydra uses srun on the first node. I got to the conclusion that SLURM is configured properly, something is wrong in the way I lunch Open MPI using mpirun. If I disable SLURM support and I revert back to rsh (--mca plm rsh) everything work but there is not profiling because the SLURM plug-in is not activated. During the configure step, Open MPI 1.8.1 detects slurm and libmpi/libpmi2 correctly. Honestly, I would prefer to avoid to use srun as job luncher if possible... Any suggestion to get this sorted out is really appreciated! Best Regards, Filippo -- Mr. Filippo SPIGA, M.Sc. http://filippospiga.info ~ skype: filippo.spiga «Nobody will drive us out of Cantor's paradise.» ~ David Hilbert * Disclaimer: "Please note this message and any attachments are CONFIDENTIAL and may be privileged or otherwise protected from disclosure. The contents are not to be disclosed to anyone other than the addressee. Unauthorized recipients are requested to preserve this confidentiality and to advise the sender immediately of any error in transmission."
Re: [OMPI users] Deadly warning "Epoll ADD(4) on fd 2 failed." ?
Dear Ralph, On May 27, 2014, at 6:31 PM, Ralph Castain <r...@open-mpi.org> wrote: > So out of curiosity - how was this job launched? Via mpirun or directly using > srun? The job has been submitted using mpirun. However Open MPI is compiled with SLURM support (and I start to believe this is might not ideal after all !!!). I have a partial job trace dumped by the process when it died: -- mpirun noticed that process rank 8190 with PID 29319 on node sand-8-39 exited on signal 11 (Segmentation fault). -- forrtl: error (78): process killed (SIGTERM) Image PCRoutineLineSource diag_OMPI-INTEL.x 00537349 Unknown Unknown Unknown diag_OMPI-INTEL.x 00535C1E Unknown Unknown Unknown diag_OMPI-INTEL.x 0050CF52 Unknown Unknown Unknown diag_OMPI-INTEL.x 004F0BB3 Unknown Unknown Unknown diag_OMPI-INTEL.x 004BEB99 Unknown Unknown Unknown libpthread.so.07FE5B5BE5710 Unknown Unknown Unknown libmlx4-rdmav2.so 7FE5A8C0A867 Unknown Unknown Unknown mca_btl_openib.so 7FE5ADA36644 Unknown Unknown Unknown libopen-pal.so.6 7FE5B288262A Unknown Unknown Unknown mca_pml_ob1.so 7FE5AC344FAF Unknown Unknown Unknown libmpi.so.17FE5B5064E7D Unknown Unknown Unknown libmpi_mpifh.so.2 7FE5B531919B Unknown Unknown Unknown libelpa.so.0 7FE5B82EC0CE Unknown Unknown Unknown libelpa.so.0 7FE5B82EBE36 Unknown Unknown Unknown libelpa.so.0 7FE5B82EBDFD Unknown Unknown Unknown libelpa.so.0 7FE5B82EC2CD Unknown Unknown Unknown libelpa.so.0 7FE5B82EB798 Unknown Unknown Unknown libelpa.so.0 7FE5B82E571A Unknown Unknown Unknown diag_OMPI-INTEL.x 004101C2 MAIN__562 dirac_exomol_eigen.f90 diag_OMPI-INTEL.x 0040A1A6 Unknown Unknown Unknown libc.so.6 7FE5B4A89D1D Unknown Unknown Unknown diag_OMPI-INTEL.x 0040A099 Unknown Unknown Unknown (plus many other trace information like this) No more information that this unfortunately because not everything library has been built using debug flags. The computation is all concentrated in ScaLAPACK and ELPA that I recompiled by myself, I run over 8192 MPI and the memory allocated per MPI process was below 1 GByte (per MPI). My compute nodes have 64 GByte of RAM and 2 eight-core Intel Sandy Bridge. Since 512 nodes are 80% of the cluster I have available for this test, I cannot easily reschedule a repetition of the test. I wonder if this message that can be related to libevent may in principle cause this seg fault error. I am working to understand the cause on my side but so far a reduced problem size using less nodes never failed. Any help is much appreciated! Regards, F -- Mr. Filippo SPIGA, M.Sc. http://www.linkedin.com/in/filippospiga ~ skype: filippo.spiga «Nobody will drive us out of Cantor's paradise.» ~ David Hilbert * Disclaimer: "Please note this message and any attachments are CONFIDENTIAL and may be privileged or otherwise protected from disclosure. The contents are not to be disclosed to anyone other than the addressee. Unauthorized recipients are requested to preserve this confidentiality and to advise the sender immediately of any error in transmission."
[OMPI users] Deadly warning "Epoll ADD(4) on fd 2 failed." ?
Dear all, I am using Open MPI v1.8.2 night snapshot compiled with SLURM support (version 14.03pre5). These two messages below appeared during a job of 2048 MPI that died after 24 hours! [warn] Epoll ADD(1) on fd 0 failed. Old events were 0; read change was 1 (add); write change was 0 (none): Operation not permitted [warn] Epoll ADD(4) on fd 2 failed. Old events were 0; read change was 0 (none); write change was 1 (add): Operation not permitted The first one, appeared immediately at the beginning had no effect. The application started to compute and it successfully called a big parallel eigensolver. The second message appeared after 18~19 hours of non-stop computation and the application crashed without showing any other error message! Regularly I was checking that MPI processes were not stuck, after this message the processes were all aborted without dumping anything on stdout/stderr. It is quite weird. I believe these messages come from Open MPI (but correct me if I am wrong!). I am going to look at the application and the various libraries to find out if something is wrong. In the meanwhile it will be a great help if anyone can clarify the exact meaning of these warning messages. Many thanks in advance. Regards, Filippo -- Mr. Filippo SPIGA, M.Sc. http://www.linkedin.com/in/filippospiga ~ skype: filippo.spiga «Nobody will drive us out of Cantor's paradise.» ~ David Hilbert * Disclaimer: "Please note this message and any attachments are CONFIDENTIAL and may be privileged or otherwise protected from disclosure. The contents are not to be disclosed to anyone other than the addressee. Unauthorized recipients are requested to preserve this confidentiality and to advise the sender immediately of any error in transmission."
Re: [OMPI users] OpenMPI 1.8.0 + PGI 13.6 = undeclared variable __LDBL_MANT_DIG__
FYI I haven't solve this yet but I managed to move to code to be compatible woth PGI 14.3. Open MPI 1.8 compiles perfectly with the latest PGI. In parallel I will push this issue to the PGI forum. Anyway, it is not a blocking factor anymore. -- Filippo SPIGA * Sent from my iPhone, sorry if I'm brief * > On 6 Apr 2014, at 21:50, Filippo Spiga <spiga.fili...@gmail.com> wrote: > > Dear all, > I am trying to compile Open MPI 1.8 with PGI 13.6. I am using on purpose > this old version of PGI because of results and performance comparisons with > old benchmarks. I run configure in this way: > > ./configure CC=pgcc CXX=pgCC FC=pgf90 F90=pgfortran CFLAGS="-noswitcherror" > FCFLAGS="-noswitcherror" CXXFLAGS="-noswitcherror" > --prefix=/usr/local/Cluster-Users/fs395/openmpi-1.8.0/pgi-13.6 > --with-hwloc=internal --enable-mca-no-build=btl-usnic --with-verbs > > > and here where configure got stuck: > > make[2]: Entering directory > `/home/fs395/archive/openmpi-1.8/build/opal/datatype' > CCLD libdatatype_reliable.la > CC opal_convertor.lo > PGC-S-0039-Use of undeclared variable __LDBL_MANT_DIG__ > (../../../opal/util/arch.h: 268) > PGC/x86 Linux 13.6-0: compilation completed with severe errors > make[2]: *** [opal_convertor.lo] Error 1 > make[2]: Leaving directory > `/home/fs395/archive/openmpi-1.8/build/opal/datatype' > make[1]: *** [install-recursive] Error 1 > make[1]: Leaving directory `/home/fs395/archive/openmpi-1.8/build/opal' > make: *** [install-recursive] Error 1 > > > Any suggestions? Googling does not help much... > > F > > -- > Mr. Filippo SPIGA, M.Sc. > http://www.linkedin.com/in/filippospiga ~ skype: filippo.spiga > > «Nobody will drive us out of Cantor's paradise.» ~ David Hilbert > > * > Disclaimer: "Please note this message and any attachments are CONFIDENTIAL > and may be privileged or otherwise protected from disclosure. The contents > are not to be disclosed to anyone other than the addressee. Unauthorized > recipients are requested to preserve this confidentiality and to advise the > sender immediately of any error in transmission." > >
Re: [OMPI users] usNIC point-to-point messaging module
Dear Dave, your suggestion worked, the file was there ar delete the directory and rebuild everything solved the issue. I still do not understand why this keeps appearing... srun: cluster configuration lacks support for cpu binding Any clue? F On Apr 1, 2014, at 9:08 PM, Dave Goodell (dgoodell) <dgood...@cisco.com> wrote: > On Apr 1, 2014, at 12:13 PM, Filippo Spiga <spiga.fili...@gmail.com> wrote: > >> Dear Ralph, Dear Jeff, >> >> I've just recompiled the latest Open MPI 1.8. I added >> "--enable-mca-no-build=btl-usnic" to configure but the message still appear. >> Here the output of "--mca btl_base_verbose 100" (trunked immediately after >> the application starts) > > Jeff's on vacation, so I'll see if I can help here. > > Try deleting all the files in "$PREFIX/lib/openmpi/", where "$PREFIX" is the > value you passed to configure with "--prefix=". If you did not pass a value, > then it is "/usr/local". Then reinstall (with "make install" in the OMPI > build tree). > > What I think is happening is that you still have an "mca_btl_usnic.so" file > leftover from the last time you installed OMPI (before passing > "--enable-mca-no-build=btl-usnic"). So OMPI is using this shared library and > you get exactly the same problem. > > -Dave > > ___ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users -- Mr. Filippo SPIGA, M.Sc. http://www.linkedin.com/in/filippospiga ~ skype: filippo.spiga «Nobody will drive us out of Cantor's paradise.» ~ David Hilbert * Disclaimer: "Please note this message and any attachments are CONFIDENTIAL and may be privileged or otherwise protected from disclosure. The contents are not to be disclosed to anyone other than the addressee. Unauthorized recipients are requested to preserve this confidentiality and to advise the sender immediately of any error in transmission."
Re: [OMPI users] usNIC point-to-point messaging module
f concern - you can >> turn it "off" by adding >> >> -mca btl ^usnic >> >> on your command line, or configuring OMPI --enable-mca-no-build=btl-usnic >> >> >> On Mar 22, 2014, at 10:00 PM, Filippo Spiga <spiga.fili...@gmail.com> wrote: >> >>> Dear all, >>> >>> I recompiled Open MPI 1.7.5 a couple of time, I am sure I have been >>> selected openib. However I have some doubts because this message >>> >>> -- >>> [[28098,1],8]: A high-performance Open MPI point-to-point messaging module >>> was unable to find any relevant network interfaces: >>> >>> Module: usNIC >>> Host: tesla79 >>> >>> Another transport will be used instead, although this may result in >>> lower performance. >>> ------ >>> >>> keeps popping up. I am really worried there might be a degradation of >>> performance because of this warning. Any clue about where it came from and >>> how I can let it disappear? >>> >>> Thanks in advance, >>> Filippo >>> >>> -- >>> Mr. Filippo SPIGA, M.Sc. >>> http://www.linkedin.com/in/filippospiga ~ skype: filippo.spiga >>> >>> «Nobody will drive us out of Cantor's paradise.» ~ David Hilbert >>> >>> * >>> Disclaimer: "Please note this message and any attachments are CONFIDENTIAL >>> and may be privileged or otherwise protected from disclosure. The contents >>> are not to be disclosed to anyone other than the addressee. Unauthorized >>> recipients are requested to preserve this confidentiality and to advise the >>> sender immediately of any error in transmission." >>> >>> >>> ___ >>> users mailing list >>> us...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/users >> >> ___ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > > -- > Jeff Squyres > jsquy...@cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > ___ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users -- Mr. Filippo SPIGA, M.Sc. http://www.linkedin.com/in/filippospiga ~ skype: filippo.spiga «Nobody will drive us out of Cantor's paradise.» ~ David Hilbert * Disclaimer: "Please note this message and any attachments are CONFIDENTIAL and may be privileged or otherwise protected from disclosure. The contents are not to be disclosed to anyone other than the addressee. Unauthorized recipients are requested to preserve this confidentiality and to advise the sender immediately of any error in transmission."
[OMPI users] usNIC point-to-point messaging module
Dear all, I recompiled Open MPI 1.7.5 a couple of time, I am sure I have been selected openib. However I have some doubts because this message -- [[28098,1],8]: A high-performance Open MPI point-to-point messaging module was unable to find any relevant network interfaces: Module: usNIC Host: tesla79 Another transport will be used instead, although this may result in lower performance. -- keeps popping up. I am really worried there might be a degradation of performance because of this warning. Any clue about where it came from and how I can let it disappear? Thanks in advance, Filippo -- Mr. Filippo SPIGA, M.Sc. http://www.linkedin.com/in/filippospiga ~ skype: filippo.spiga «Nobody will drive us out of Cantor's paradise.» ~ David Hilbert * Disclaimer: "Please note this message and any attachments are CONFIDENTIAL and may be privileged or otherwise protected from disclosure. The contents are not to be disclosed to anyone other than the addressee. Unauthorized recipients are requested to preserve this confidentiality and to advise the sender immediately of any error in transmission."
Re: [OMPI users] Compiling Open MPI 1.7.4 using PGI 14.2 and Mellanox HCOLL enabled
Hi Jeff, Hi Ake, removing --with-slurm and keeping --with-hcoll seems to work. The error disappears at compile time, I have not yet tried to run a job. I can copy config.log and the make.log is needed. Cheers, F On Mar 11, 2014, at 4:48 PM, Jeff Squyres (jsquyres) <jsquy...@cisco.com> wrote: > On Mar 11, 2014, at 11:22 AM, Åke Sandgren <ake.sandg...@hpc2n.umu.se> wrote: > >>>> ../configure CC=pgcc CXX=pgCC FC=pgf90 F90=pgf90 >>>> --prefix=/usr/local/Cluster-Users/fs395/openmpi-1.7.4/pgi-14.2_cuda-6.0RC >>>> --enable-mpirun-prefix-by-default --with-hcoll=$HCOLL_DIR >>>> --with-fca=$FCA_DIR --with-mxm=$MXM_DIR --with-knem=$KNEM_DIR >>>> --with-slurm=/usr/local/Cluster-Apps/slurm --with-cuda=$CUDA_INSTALL_PATH >>>> >>>> >>>> At some point the compile process fails with this error: >>>> >>>> make[2]: Leaving directory >>>> `/home/fs395/archive/openmpi-1.7.4/build/ompi/mca/coll/hierarch' >>>> Making all in mca/coll/hcoll >>>> make[2]: Entering directory >>>> `/home/fs395/archive/openmpi-1.7.4/build/ompi/mca/coll/hcoll' >>>> CC coll_hcoll_module.lo >>>> CC coll_hcoll_component.lo >>>> CC coll_hcoll_rte.lo >>>> CC coll_hcoll_ops.lo >>>> CCLD mca_coll_hcoll.la >>>> pgcc-Error-Unknown switch: -pthread >> >> You have to remove the -pthread from inherited_linker_flags= >> in libpmi.la libslurm.la from your slurm build. > > With the configure line given above, I don't think he should be linking > against libslurm. > > But I wonder if the underlying issue is actually correct: perhaps the > inherited_linker_flags from libhcoll.la has -pthreads in it. -- Mr. Filippo SPIGA, M.Sc. http://www.linkedin.com/in/filippospiga ~ skype: filippo.spiga «Nobody will drive us out of Cantor's paradise.» ~ David Hilbert * Disclaimer: "Please note this message and any attachments are CONFIDENTIAL and may be privileged or otherwise protected from disclosure. The contents are not to be disclosed to anyone other than the addressee. Unauthorized recipients are requested to preserve this confidentiality and to advise the sender immediately of any error in transmission."
Re: [OMPI users] 1.7.5rc1, error "COLL-ML ml_discover_hierarchy exited with error."
Dear Rolf, your suggestion works! $ mpirun -np 4 --map-by ppr:1:socket -bind-to core --mca coll ^ml osu_alltoall # OSU MPI All-to-All Personalized Exchange Latency Test v4.2 # Size Avg Latency(us) 1 8.02 2 2.96 4 2.91 8 2.91 16 2.96 32 3.07 64 3.25 128 3.74 256 3.85 512 4.11 10244.79 20485.91 4096 15.84 8192 24.88 16384 35.35 32768 56.20 65536 66.88 131072114.89 262144209.36 524288396.12 1048576 765.65 Can you clarify exactly where the problem come from? Regards, Filippo On Mar 4, 2014, at 12:17 AM, Rolf vandeVaart <rvandeva...@nvidia.com> wrote: > Can you try running with --mca coll ^ml and see if things work? > > Rolf > >> -Original Message- >> From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Filippo Spiga >> Sent: Monday, March 03, 2014 7:14 PM >> To: Open MPI Users >> Subject: [OMPI users] 1.7.5rc1, error "COLL-ML ml_discover_hierarchy exited >> with error." >> >> Dear Open MPI developers, >> >> I hit an expected error running OSU osu_alltoall benchmark using Open MPI >> 1.7.5rc1. Here the error: >> >> $ mpirun -np 4 --map-by ppr:1:socket -bind-to core osu_alltoall In >> bcol_comm_query hmca_bcol_basesmuma_allocate_sm_ctl_memory failed >> In bcol_comm_query hmca_bcol_basesmuma_allocate_sm_ctl_memory >> failed >> [tesla50][[6927,1],1][../../../../../ompi/mca/coll/ml/coll_ml_module.c:2996:mc >> a_coll_ml_comm_query] COLL-ML ml_discover_hierarchy exited with error. >> >> [tesla50:42200] In base_bcol_masesmuma_setup_library_buffers and mpool >> was not successfully setup! >> [tesla50][[6927,1],0][../../../../../ompi/mca/coll/ml/coll_ml_module.c:2996:mc >> a_coll_ml_comm_query] COLL-ML ml_discover_hierarchy exited with error. >> >> [tesla50:42201] In base_bcol_masesmuma_setup_library_buffers and mpool >> was not successfully setup! >> # OSU MPI All-to-All Personalized Exchange Latency Test v4.2 >> # Size Avg Latency(us) >> -- >> mpirun noticed that process rank 3 with PID 4508 on node tesla51 exited on >> signal 11 (Segmentation fault). >> -- >> 2 total processes killed (some possibly by mpirun during cleanup) >> >> Any idea where this come from? >> >> I compiled Open MPI using Intel 12.1, latest Mellanox stack and CUDA 6.0RC. >> Attached outputs grabbed from configure, make and run. The configure was >> >> export MXM_DIR=/opt/mellanox/mxm >> export KNEM_DIR=$(find /opt -maxdepth 1 -type d -name "knem*" -print0) >> export FCA_DIR=/opt/mellanox/fca export HCOLL_DIR=/opt/mellanox/hcoll >> >> ../configure CC=icc CXX=icpc F77=ifort FC=ifort FFLAGS="-xSSE4.2 -axAVX -ip - >> O3 -fno-fnalias" FCFLAGS="-xSSE4.2 -axAVX -ip -O3 -fno-fnalias" >> --prefix=<...> >> --enable-mpirun-prefix-by-default --with-fca=$FCA_DIR --with- >> mxm=$MXM_DIR --with-knem=$KNEM_DIR --with- >> cuda=$CUDA_INSTALL_PATH --enable-mpi-thread-multiple --with- >> hwloc=internal --with-verbs 2>&1 | tee config.out >> >> >> Thanks in advance, >> Regards >> >> Filippo >> >> -- >> Mr. Filippo SPIGA, M.Sc. >> http://www.linkedin.com/in/filippospiga ~ skype: filippo.spiga >> >> ~ David Hilbert >> >> * >> Disclaimer: "Please note this message and any attachments are >> CONFIDENTIAL and may be privileged or otherwise protected from disclosure. >> The contents are not to be disclosed to anyone other than the addressee. >> Unauthorized recipients are requested to preserve this confidentiality and to >> advise the sender immediately of any error in transmission." > > --- > This email message is for the sole use of the intended recipient(s) and may > contain > confidential information. Any unauthorized review, use, disclosure or > distribution > is prohibited. If you are not the intended recipient, please contact the > sender by > reply email and destroy all copies of the original message. > -
[OMPI users] 1.7.5rc1, error "COLL-ML ml_discover_hierarchy exited with error."
Dear Open MPI developers, I hit an expected error running OSU osu_alltoall benchmark using Open MPI 1.7.5rc1. Here the error: $ mpirun -np 4 --map-by ppr:1:socket -bind-to core osu_alltoall In bcol_comm_query hmca_bcol_basesmuma_allocate_sm_ctl_memory failed In bcol_comm_query hmca_bcol_basesmuma_allocate_sm_ctl_memory failed [tesla50][[6927,1],1][../../../../../ompi/mca/coll/ml/coll_ml_module.c:2996:mca_coll_ml_comm_query] COLL-ML ml_discover_hierarchy exited with error. [tesla50:42200] In base_bcol_masesmuma_setup_library_buffers and mpool was not successfully setup! [tesla50][[6927,1],0][../../../../../ompi/mca/coll/ml/coll_ml_module.c:2996:mca_coll_ml_comm_query] COLL-ML ml_discover_hierarchy exited with error. [tesla50:42201] In base_bcol_masesmuma_setup_library_buffers and mpool was not successfully setup! # OSU MPI All-to-All Personalized Exchange Latency Test v4.2 # Size Avg Latency(us) -- mpirun noticed that process rank 3 with PID 4508 on node tesla51 exited on signal 11 (Segmentation fault). -- 2 total processes killed (some possibly by mpirun during cleanup) Any idea where this come from? I compiled Open MPI using Intel 12.1, latest Mellanox stack and CUDA 6.0RC. Attached outputs grabbed from configure, make and run. The configure was export MXM_DIR=/opt/mellanox/mxm export KNEM_DIR=$(find /opt -maxdepth 1 -type d -name "knem*" -print0) export FCA_DIR=/opt/mellanox/fca export HCOLL_DIR=/opt/mellanox/hcoll ../configure CC=icc CXX=icpc F77=ifort FC=ifort FFLAGS="-xSSE4.2 -axAVX -ip -O3 -fno-fnalias" FCFLAGS="-xSSE4.2 -axAVX -ip -O3 -fno-fnalias" --prefix=<...> --enable-mpirun-prefix-by-default --with-fca=$FCA_DIR --with-mxm=$MXM_DIR --with-knem=$KNEM_DIR --with-cuda=$CUDA_INSTALL_PATH --enable-mpi-thread-multiple --with-hwloc=internal --with-verbs 2>&1 | tee config.out Thanks in advance, Regards Filippo -- Mr. Filippo SPIGA, M.Sc. http://www.linkedin.com/in/filippospiga ~ skype: filippo.spiga «Nobody will drive us out of Cantor's paradise.» ~ David Hilbert * Disclaimer: "Please note this message and any attachments are CONFIDENTIAL and may be privileged or otherwise protected from disclosure. The contents are not to be disclosed to anyone other than the addressee. Unauthorized recipients are requested to preserve this confidentiality and to advise the sender immediately of any error in transmission." openmpi-1.7.5rc1_wrong.tar.gz Description: GNU Zip compressed data
Re: [OMPI users] Compiling Open MPI 1.7.4 using PGI 14.2 and Mellanox HCOLL enabled
Dear Ralph, I still need a workaround to compile using PGI and --with-hcoll. I tried a night snapshot last week I will try again the latest one and if something change I will let you know. Regards, Filippo On Feb 26, 2014, at 6:16 PM, Ralph Castain <r...@open-mpi.org> wrote: > Perhaps you could try the nightly 1.7.5 tarball? I believe some PGI fixes may > have gone in there > > > On Feb 25, 2014, at 3:22 PM, Filippo Spiga <spiga.fili...@gmail.com> wrote: > >> Dear all, >> >> I came across another small issue while I was compiling Open MPI 1.7.4 using >> PGI 14.2 and building the support for Mellanox Hierarchical Collectives >> (--with-hcoll). Here you how configure Open MPI: >> >> export MXM_DIR=/opt/mellanox/mxm >> export KNEM_DIR=$(find /opt -maxdepth 1 -type d -name "knem*" -print0) >> export FCA_DIR=/opt/mellanox/fca >> export HCOLL_DIR=/opt/mellanox/hcoll >> >> ../configure CC=pgcc CXX=pgCC FC=pgf90 F90=pgf90 >> --prefix=/usr/local/Cluster-Users/fs395/openmpi-1.7.4/pgi-14.2_cuda-6.0RC >> --enable-mpirun-prefix-by-default --with-hcoll=$HCOLL_DIR >> --with-fca=$FCA_DIR --with-mxm=$MXM_DIR --with-knem=$KNEM_DIR >> --with-slurm=/usr/local/Cluster-Apps/slurm --with-cuda=$CUDA_INSTALL_PATH >> >> >> At some point the compile process fails with this error: >> >> make[2]: Leaving directory >> `/home/fs395/archive/openmpi-1.7.4/build/ompi/mca/coll/hierarch' >> Making all in mca/coll/hcoll >> make[2]: Entering directory >> `/home/fs395/archive/openmpi-1.7.4/build/ompi/mca/coll/hcoll' >> CC coll_hcoll_module.lo >> CC coll_hcoll_component.lo >> CC coll_hcoll_rte.lo >> CC coll_hcoll_ops.lo >> CCLD mca_coll_hcoll.la >> pgcc-Error-Unknown switch: -pthread >> make[2]: *** [mca_coll_hcoll.la] Error 1 >> make[2]: Leaving directory >> `/home/fs395/archive/openmpi-1.7.4/build/ompi/mca/coll/hcoll' >> make[1]: *** [all-recursive] Error 1 >> make[1]: Leaving directory `/home/fs395/archive/openmpi-1.7.4/build/ompi' >> make: *** [all-recursive] Error 1 >> >> Attached the configure.log and the make.log collected as reported on the >> website. Using google I found an old post referring to the same problem. >> Here few relevant links: >> http://www.open-mpi.org/community/lists/users/2009/03/8687.php >> http://www.open-mpi.org/community/lists/users/2010/09/14229.php >> http://www.open-mpi.org/community/lists/users/2009/04/8911.php >> >> I have no problem to use a fake wrapper or the "-noswitcherror" compiler >> pgf90 flag. I wonder if this procedure will affect in some way the MPI built >> and I have to carry on this flag also when I compile my applications. >> >> Is there any way to fix libtool so Open MPI can build itself properly? >> >> Thanks >> Filippo >> >> -- >> Mr. Filippo SPIGA, M.Sc. >> http://www.linkedin.com/in/filippospiga ~ skype: filippo.spiga >> >> «Nobody will drive us out of Cantor's paradise.» ~ David Hilbert >> >> * >> Disclaimer: "Please note this message and any attachments are CONFIDENTIAL >> and may be privileged or otherwise protected from disclosure. The contents >> are not to be disclosed to anyone other than the addressee. Unauthorized >> recipients are requested to preserve this confidentiality and to advise the >> sender immediately of any error in transmission." >> >> >> ___ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > ___ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users -- Mr. Filippo SPIGA, M.Sc. http://www.linkedin.com/in/filippospiga ~ skype: filippo.spiga «Nobody will drive us out of Cantor's paradise.» ~ David Hilbert * Disclaimer: "Please note this message and any attachments are CONFIDENTIAL and may be privileged or otherwise protected from disclosure. The contents are not to be disclosed to anyone other than the addressee. Unauthorized recipients are requested to preserve this confidentiality and to advise the sender immediately of any error in transmission."
Re: [OMPI users] OpenMPI 1.7.5 and "--map-by" new syntax
Yes it works. Information provided by mpirun is confusing but I get the right syntax now. Thank you! F On Feb 26, 2014, at 12:34 PM, tmish...@jcity.maeda.co.jp wrote: > Hi, this help message might be just a simple mistake. > > Please try: mpirun -np 20 --map-by ppr:5:socket -bind-to core osu_alltoall > > There's no available explanation yet as far as I know, because it's still > alfa version. > > Tetsuya Mishima > >> Dear all, >> >> I am playing with Open MPI 1.7.5 and with the "--map-by" option but I am > not sure I am doing thing correctly despite I am following the instruction. > Here what I got >> >> $mpirun -np 20 --npersocket 5 -bind-to core osu_alltoall >> > -- >> The following command line options and corresponding MCA parameter have >> been deprecated and replaced as follows: >> >> Command line options: >> Deprecated: --npersocket, -npersocket >> Replacement: --map-by socket:PPR=N >> >> Equivalent MCA parameter: >> Deprecated: rmaps_base_n_persocket, rmaps_ppr_n_persocket >> Replacement: rmaps_base_mapping_policy=socket:PPR=N >> >> The deprecated forms *will* disappear in a future version of Open MPI. >> Please update to the new syntax. >> > -- >> >> >> after changing according to the instructions I see >> >> $ mpirun -np 24 --map-by socket:PPR=5 -bind-to core osu_alltoall >> >> > -- >> The mapping request contains an unrecognized modifier: >> >> Request: socket:PPR=5 >> >> Please check your request and try again. >> > -- >> [tesla49:30459] [[29390,0],0] ORTE_ERROR_LOG: Bad parameter in file > ess_hnp_module.c at line 510 >> > -- >> It looks like orte_init failed for some reason; your parallel process is >> likely to abort. There are many reasons that a parallel process can >> fail during orte_init; some of which are due to configuration or >> environment problems. This failure appears to be an internal failure; >> here's some additional information (which may only be relevant to an >> Open MPI developer): >> >> orte_rmaps_base_open failed >> --> Returned value Bad parameter (-5) instead of ORTE_SUCCESS >> > -- >> >> >> >> Is there any place where the new syntax is explained? >> >> Thanks in advance >> F >> >> -- >> Mr. Filippo SPIGA, M.Sc. - HPC Application Specialist >> High Performance Computing Service, University of Cambridge (UK) >> http://www.hpc.cam.ac.uk/ ~ http://filippospiga.me ~ skype: filippo.spiga >> >> «Nobody will drive us out of Cantor's paradise.» ~ David Hilbert >> >> * >> Disclaimer: "Please note this message and any attachments are > CONFIDENTIAL and may be privileged or otherwise protected from disclosure. > The contents are not to be disclosed to anyone other than the >> addressee. Unauthorized recipients are requested to preserve this > confidentiality and to advise the sender immediately of any error in > transmission." >> ___ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > ___ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users -- Mr. Filippo SPIGA, M.Sc. http://www.linkedin.com/in/filippospiga ~ skype: filippo.spiga «Nobody will drive us out of Cantor's paradise.» ~ David Hilbert * Disclaimer: "Please note this message and any attachments are CONFIDENTIAL and may be privileged or otherwise protected from disclosure. The contents are not to be disclosed to anyone other than the addressee. Unauthorized recipients are requested to preserve this confidentiality and to advise the sender immediately of any error in transmission."
[OMPI users] OpenMPI 1.7.5 and "--map-by" new syntax
Dear all, I am playing with Open MPI 1.7.5 and with the "--map-by" option but I am not sure I am doing thing correctly despite I am following the instruction. Here what I got $mpirun -np 20 --npersocket 5 -bind-to core osu_alltoall -- The following command line options and corresponding MCA parameter have been deprecated and replaced as follows: Command line options: Deprecated: --npersocket, -npersocket Replacement: --map-by socket:PPR=N Equivalent MCA parameter: Deprecated: rmaps_base_n_persocket, rmaps_ppr_n_persocket Replacement: rmaps_base_mapping_policy=socket:PPR=N The deprecated forms *will* disappear in a future version of Open MPI. Please update to the new syntax. -- after changing according to the instructions I see $ mpirun -np 24 --map-by socket:PPR=5 -bind-to core osu_alltoall -- The mapping request contains an unrecognized modifier: Request: socket:PPR=5 Please check your request and try again. -- [tesla49:30459] [[29390,0],0] ORTE_ERROR_LOG: Bad parameter in file ess_hnp_module.c at line 510 -- It looks like orte_init failed for some reason; your parallel process is likely to abort. There are many reasons that a parallel process can fail during orte_init; some of which are due to configuration or environment problems. This failure appears to be an internal failure; here's some additional information (which may only be relevant to an Open MPI developer): orte_rmaps_base_open failed --> Returned value Bad parameter (-5) instead of ORTE_SUCCESS -- Is there any place where the new syntax is explained? Thanks in advance F -- Mr. Filippo SPIGA, M.Sc. - HPC Application Specialist High Performance Computing Service, University of Cambridge (UK) http://www.hpc.cam.ac.uk/ ~ http://filippospiga.me ~ skype: filippo.spiga «Nobody will drive us out of Cantor's paradise.» ~ David Hilbert * Disclaimer: "Please note this message and any attachments are CONFIDENTIAL and may be privileged or otherwise protected from disclosure. The contents are not to be disclosed to anyone other than the addressee. Unauthorized recipients are requested to preserve this confidentiality and to advise the sender immediately of any error in transmission."
[OMPI users] Compiling Open MPI 1.7.4 using PGI 14.2 and Mellanox HCOLL enabled
Dear all, I came across another small issue while I was compiling Open MPI 1.7.4 using PGI 14.2 and building the support for Mellanox Hierarchical Collectives (--with-hcoll). Here you how configure Open MPI: export MXM_DIR=/opt/mellanox/mxm export KNEM_DIR=$(find /opt -maxdepth 1 -type d -name "knem*" -print0) export FCA_DIR=/opt/mellanox/fca export HCOLL_DIR=/opt/mellanox/hcoll ../configure CC=pgcc CXX=pgCC FC=pgf90 F90=pgf90 --prefix=/usr/local/Cluster-Users/fs395/openmpi-1.7.4/pgi-14.2_cuda-6.0RC --enable-mpirun-prefix-by-default --with-hcoll=$HCOLL_DIR --with-fca=$FCA_DIR --with-mxm=$MXM_DIR --with-knem=$KNEM_DIR --with-slurm=/usr/local/Cluster-Apps/slurm --with-cuda=$CUDA_INSTALL_PATH At some point the compile process fails with this error: make[2]: Leaving directory `/home/fs395/archive/openmpi-1.7.4/build/ompi/mca/coll/hierarch' Making all in mca/coll/hcoll make[2]: Entering directory `/home/fs395/archive/openmpi-1.7.4/build/ompi/mca/coll/hcoll' CC coll_hcoll_module.lo CC coll_hcoll_component.lo CC coll_hcoll_rte.lo CC coll_hcoll_ops.lo CCLD mca_coll_hcoll.la pgcc-Error-Unknown switch: -pthread make[2]: *** [mca_coll_hcoll.la] Error 1 make[2]: Leaving directory `/home/fs395/archive/openmpi-1.7.4/build/ompi/mca/coll/hcoll' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/home/fs395/archive/openmpi-1.7.4/build/ompi' make: *** [all-recursive] Error 1 Attached the configure.log and the make.log collected as reported on the website. Using google I found an old post referring to the same problem. Here few relevant links: http://www.open-mpi.org/community/lists/users/2009/03/8687.php http://www.open-mpi.org/community/lists/users/2010/09/14229.php http://www.open-mpi.org/community/lists/users/2009/04/8911.php I have no problem to use a fake wrapper or the "-noswitcherror" compiler pgf90 flag. I wonder if this procedure will affect in some way the MPI built and I have to carry on this flag also when I compile my applications. Is there any way to fix libtool so Open MPI can build itself properly? Thanks Filippo -- Mr. Filippo SPIGA, M.Sc. http://www.linkedin.com/in/filippospiga ~ skype: filippo.spiga «Nobody will drive us out of Cantor's paradise.» ~ David Hilbert * Disclaimer: "Please note this message and any attachments are CONFIDENTIAL and may be privileged or otherwise protected from disclosure. The contents are not to be disclosed to anyone other than the addressee. Unauthorized recipients are requested to preserve this confidentiality and to advise the sender immediately of any error in transmission."
[OMPI users] Configure issue with/without HWLOC when PGI used and CUDA support enabled
Dear Open MPI developers, I just want to point to a weird behavior of the configure procedure I discovered. I am compiling Open MPI 1.7.4 with CUDA support (CUDA 6.0 RC) and PGI 14.1 If I explicitly compile against a self-compiled version of HWLOC (1.8.1) using this configure line ../configure CC=pgcc CXX=pgCC FC=pgf90 F90=pgf90 --prefix=/usr/local/Cluster-Users/fs395/openmpi-1.7.4/pgi-14.1_cuda-6.0RC --enable-mpirun-prefix-by-default --with-fca=$FCA_DIR --with-mxm=$MXM_DIR --with-knem=$KNEM_DIR --with-hwloc=/usr/local/Cluster-Users/fs395/hwlock-1.8.1/gcc-4.4.7_cuda-6.0RC --with-slurm=/usr/local/Cluster-Apps/slurm --with-cuda=/usr/local/Cluster-Users/fs395/cuda/6.0-RC make fails telling me that it cannot find "-lcudart". If I compile without HWLOC using this configure line: ../configure CC=pgcc CXX=pgCC FC=pgf90 F90=pgf90 --prefix=/usr/local/Cluster-Users/fs395/openmpi-1.7.4/pgi-14.1_cuda-6.0RC --enable-mpirun-prefix-by-default --with-fca=$FCA_DIR --with-mxm=$MXM_DIR --with-knem=$KNEM_DIR --with-slurm=/usr/local/Cluster-Apps/slurm --with-cuda=/usr/local/Cluster-Users/fs395/cuda/6.0-RC make succeeds and I have Open MPI compiled properly. $ mpif90 -show pgf90 -I/usr/local/Cluster-Users/fs395/openmpi-1.7.4/pgi-14.1_cuda-6.0RC/include -I/usr/local/Cluster-Users/fs395/openmpi-1.7.4/pgi-14.1_cuda-6.0RC/lib -Wl,-rpath -Wl,/usr/local/Cluster-Users/fs395/openmpi-1.7.4/pgi-14.1_cuda-6.0RC/lib -L/usr/local/Cluster-Users/fs395/openmpi-1.7.4/pgi-14.1_cuda-6.0RC/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi $ ompi_info --all | grep btl_openib_have_cuda_gdr MCA btl: informational "btl_openib_have_cuda_gdr" (current value: "true", data source: default, level: 5 tuner/detail, type: bool) I wonder why the configure picks up lib instead of lib64. I will test the build using real codes. Cheers, Filippo -- Mr. Filippo SPIGA, M.Sc. http://www.linkedin.com/in/filippospiga ~ skype: filippo.spiga «Nobody will drive us out of Cantor's paradise.» ~ David Hilbert * Disclaimer: "Please note this message and any attachments are CONFIDENTIAL and may be privileged or otherwise protected from disclosure. The contents are not to be disclosed to anyone other than the addressee. Unauthorized recipients are requested to preserve this confidentiality and to advise the sender immediately of any error in transmission."
[OMPI users] Clarification about dual-rail capabilities (sharing)
Dear Open MPI users, in my institution a cluster with dual-rail IB has recently deployed. Each compute node has two physical single-port Mellanox Connect-IB MT27600 card (mlx5_0, mlx5_1). By running bandwidth tests (OSU 4.2 benchmark) using MVAPICH2, I can achieve from one node to another (1 MPI process per node) up to 12 GB/s using the rail in sharing by brokering small messages across both HCA devices. This is good. The I switched to Open MPI (1.7.3 and 1.7.4rc1). I tried to use both HCAs together but it seems to me that only one is used (because there is only one process per node?). In Open MPI it seems more complicated to set up such a test. This is what I did... mpirun --mca coll_fca_enable 0 --mca btl_openib_verbose 1 -host HOST1,HOST2 --mca btl_openib_if_include mlx5_0,mlx5_1 -np 1 ./osu-bin/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw : -np 1 --mca coll_fca_enable 0 --mca btl_openib_verbose 1 --mca btl_openib_if_include mlx5_0,mlx5_1 ./osu-bin/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw Max measured bandwidth is around 6.5 GB/s, basically the same if I use a single HCA. What I do wrong? Is this the correct way to exploit a multi-rail system? Many thanks in advance, Regards -- Mr. Filippo SPIGA, M.Sc. http://www.linkedin.com/in/filippospiga ~ skype: filippo.spiga «Nobody will drive us out of Cantor's paradise.» ~ David Hilbert
