Re: [OMPI users] new map-by-obj has a problem

[tmishima]

Hi Ralph, I have tested your fix - 30895. I'm afraid to say
I found a mistake.

You should include "SETTING BIND_TO_NONE" in the above if-clause
at the line 74, 256, 511, 656. Othrewise, just warning message
disappears but binding to core is still overwritten by binding
to none. Pleaes see attached patch.

(See attached file: patch_from_30895)

Tetsuya


> Hi Ralph, I understood what you meant.
>
> I often use float for our applicatoin.
> float c = (float)(unsinged int a - unsinged int b) could
> be very huge number, if a < b. So I always carefully cast to
> int from unsigned int when I subtract them. I didn't know/mind
> inc d = (unsinged int a - unsinged int b) has no problem.
> I noticed it by your suggestion, thanks.
>
> Therefore, I think my fix is not necesarry.
>
> Tetsuya
>
>
> > Yes, indeed. In future, when we will have many many cores
> > in the machine, we will have to take care of overrun of
> > num_procs.
> >
> > Tetsuya
> >
> > > Cool - easily modified. Thanks!
> > >
> > > Of course, you understand (I'm sure) that the cast does nothing to
> > protect the code from blowing up if we overrun the var. In other words,
> if
> > the unsigned var has wrapped, then casting it to int
> > > won't help - you'll still get a negative integer, and the code will
> > trash.
> > >
> > >
> > > On Feb 28, 2014, at 3:43 PM, tmish...@jcity.maeda.co.jp wrote:
> > >
> > > >
> > > >
> > > > Hi Ralph, I'm a litte bit late to your release.
> > > >
> > > > I found a minor mistake in byobj_span -integer casting problem.
> > > >
> > > > --- rmaps_rr_mappers.30892.c2014-03-01 08:31:50 +0900
> > > > +++ rmaps_rr_mappers.c  2014-03-01 08:33:22 +0900
> > > > @@ -689,7 +689,7 @@
> > > > }
> > > >
> > > > /* compute how many objs need an extra proc */
> > > > -if (0 > (nxtra_objs = app->num_procs - (navg * nobjs))) {
> > > > +if (0 > (nxtra_objs = (int)app->num_procs - (navg *
> (int)nobjs)))
> > {
> > > > nxtra_objs = 0;
> > > > }
> > > >
> > > > Tetsuya
> > > >
> > > >> Please take a look at
> https://svn.open-mpi.org/trac/ompi/ticket/4317
> > > >>
> > > >>
> > > >> On Feb 27, 2014, at 8:13 PM, tmish...@jcity.maeda.co.jp wrote:
> > > >>
> > > >>>
> > > >>>
> > > >>> Hi Ralph, I can't operate our cluster for a few days, sorry.
> > > >>>
> > > >>> But now, I'm narrowing down the cause by browsing the source
code.
> > > >>>
> > > >>> My best guess is the line 529. The
opal_hwloc_base_get_obj_by_type
> > will
> > > >>> reset the object pointer to the first one when you move on to the
> > next
> > > >>> node.
> > > >>>
> > > >>> 529if (NULL == (obj =
> > > > opal_hwloc_base_get_obj_by_type
> > > >>> (node->topology, target, cache_level, i, OPAL_HWLOC_AVAILABLE)))
{
> > > >>> 530ORTE_ERROR_LOG(ORTE_ERR_NOT_FOUND);
> > > >>> 531return ORTE_ERR_NOT_FOUND;
> > > >>> 532}
> > > >>>
> > > >>> if node->slots=1, then nprocs is set as nprocs=1 in the second
> pass:
> > > >>>
> > > >>> 495nprocs = (node->slots - node->slots_inuse) /
> > > >>> orte_rmaps_base.cpus_per_rank;
> > > >>> 496if (nprocs < 1) {
> > > >>> 497if (second_pass) {
> > > >>> 498/* already checked for oversubscription
> > > > permission,
> > > >>> so at least put
> > > >>> 499 * one proc on it
> > > >>> 500 */
> > > >>> 501nprocs = 1;
> > > >>>
> > > >>> Therefore, opal_hwloc_base_get_obj_by_type is called one by one
at
> > each
> > > >>> node, which means
> > > >>> the object we get is always first one.
> > > >>>
> > > >>> It's not elegant but I guess you need dummy calls of
> > > >>> opal_hwloc_base_get_obj_by_type to
> > > >>> move the object pointer to the right place or modify
> > > >>> opal_hwloc_base_get_obj_by_type itself.
> > > >>>
> > > >>> Tetsuya
> > > >>>
> > >  I'm having trouble seeing why it is failing, so I added some
more
> > > > debug
> > > >>> output. Could you run the failure case again with -mca
> > > > rmaps_base_verbose
> > > >>> 10?
> > > 
> > >  Thanks
> > >  Ralph
> > > 
> > >  On Feb 27, 2014, at 6:11 PM, tmish...@jcity.maeda.co.jp wrote:
> > > 
> > > >
> > > >
> > > > Just checking the difference, not so significant meaning...
> > > >
> > > > Anyway, I guess it's due to the behavior when slot counts is
> > missing
> > > > (regarded as slots=1) and it's oversubscribed unintentionally.
> > > >
> > > > I'm going out now, so I can't verify it quickly. If I provide
the
> > > > correct slot counts, it wll work, I guess. How do you think?
> > > >
> > > > Tetsuya
> > > >
> > > >> "restore" in what sense?
> > > >>
> > > >> On Feb 27, 2014, at 4:10 PM, tmish...@jcity.maeda.co.jp wrote:
> > > >>
> > > >>>
> > > >>>
> > > >>> Hi Ralph, this is just for your information.
> > > >>>
> > > >>> I tried to 

Re: [OMPI users] Compiling Open MPI 1.7.4 using PGI 14.2 and Mellanox HCOLL enabled

[Filippo Spiga]

Dear Ralph,

I still need a workaround to compile using PGI and --with-hcoll. I tried a 
night snapshot last week I will try again the latest one and if something 
change I will let you know.

Regards,
Filippo


On Feb 26, 2014, at 6:16 PM, Ralph Castain  wrote:

> Perhaps you could try the nightly 1.7.5 tarball? I believe some PGI fixes may 
> have gone in there
> 
> 
> On Feb 25, 2014, at 3:22 PM, Filippo Spiga  wrote:
> 
>> Dear all,
>> 
>> I came across another small issue while I was compiling Open MPI 1.7.4 using 
>> PGI 14.2 and building the support for Mellanox Hierarchical Collectives 
>> (--with-hcoll). Here you how configure Open MPI:
>> 
>> export MXM_DIR=/opt/mellanox/mxm
>> export KNEM_DIR=$(find /opt -maxdepth 1 -type d -name "knem*" -print0)
>> export FCA_DIR=/opt/mellanox/fca
>> export HCOLL_DIR=/opt/mellanox/hcoll
>> 
>> ../configure  CC=pgcc CXX=pgCC FC=pgf90 F90=pgf90 
>> --prefix=/usr/local/Cluster-Users/fs395/openmpi-1.7.4/pgi-14.2_cuda-6.0RC  
>> --enable-mpirun-prefix-by-default --with-hcoll=$HCOLL_DIR 
>> --with-fca=$FCA_DIR --with-mxm=$MXM_DIR --with-knem=$KNEM_DIR 
>> --with-slurm=/usr/local/Cluster-Apps/slurm  --with-cuda=$CUDA_INSTALL_PATH
>> 
>> 
>> At some point the compile process fails with this error:
>> 
>> make[2]: Leaving directory 
>> `/home/fs395/archive/openmpi-1.7.4/build/ompi/mca/coll/hierarch'
>> Making all in mca/coll/hcoll
>> make[2]: Entering directory 
>> `/home/fs395/archive/openmpi-1.7.4/build/ompi/mca/coll/hcoll'
>> CC   coll_hcoll_module.lo
>> CC   coll_hcoll_component.lo
>> CC   coll_hcoll_rte.lo
>> CC   coll_hcoll_ops.lo
>> CCLD mca_coll_hcoll.la
>> pgcc-Error-Unknown switch: -pthread
>> make[2]: *** [mca_coll_hcoll.la] Error 1
>> make[2]: Leaving directory 
>> `/home/fs395/archive/openmpi-1.7.4/build/ompi/mca/coll/hcoll'
>> make[1]: *** [all-recursive] Error 1
>> make[1]: Leaving directory `/home/fs395/archive/openmpi-1.7.4/build/ompi'
>> make: *** [all-recursive] Error 1
>> 
>> Attached the configure.log and the make.log collected as reported on the 
>> website.  Using google I found an old post referring to the same problem. 
>> Here few relevant links:
>> http://www.open-mpi.org/community/lists/users/2009/03/8687.php
>> http://www.open-mpi.org/community/lists/users/2010/09/14229.php
>> http://www.open-mpi.org/community/lists/users/2009/04/8911.php
>> 
>> I have no problem to use a fake wrapper or the "-noswitcherror" compiler 
>> pgf90 flag. I wonder if this procedure will affect in some way the MPI built 
>> and I have to carry on this flag also when I compile my applications. 
>> 
>> Is there any way to fix libtool so Open MPI can build itself properly?
>> 
>> Thanks
>> Filippo
>> 
>> --
>> Mr. Filippo SPIGA, M.Sc.
>> http://www.linkedin.com/in/filippospiga ~ skype: filippo.spiga
>> 
>> «Nobody will drive us out of Cantor's paradise.» ~ David Hilbert
>> 
>> *
>> Disclaimer: "Please note this message and any attachments are CONFIDENTIAL 
>> and may be privileged or otherwise protected from disclosure. The contents 
>> are not to be disclosed to anyone other than the addressee. Unauthorized 
>> recipients are requested to preserve this confidentiality and to advise the 
>> sender immediately of any error in transmission."
>> 
>> 
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

--
Mr. Filippo SPIGA, M.Sc.
http://www.linkedin.com/in/filippospiga ~ skype: filippo.spiga

«Nobody will drive us out of Cantor's paradise.» ~ David Hilbert

*
Disclaimer: "Please note this message and any attachments are CONFIDENTIAL and 
may be privileged or otherwise protected from disclosure. The contents are not 
to be disclosed to anyone other than the addressee. Unauthorized recipients are 
requested to preserve this confidentiality and to advise the sender immediately 
of any error in transmission."

Re: [OMPI users] Heterogeneous cluster problem - mixing AMD and Intel nodes

[Victor]

Thanks for your reply. There are some updates, but it was too late last
night to post it.

I now have the AMD/Intel heterogeneous cluster up and running. The initial
problem was that when I installed OpenMPI on the AMD nodes, the library
paths were set to a different location than on the Intel nodes. I am not
sure why.

In any case I then followed the suggestion from the FAQ and instead shared
the same OpenMPI install directory with all the nodes via NFS. Now the job
is running so I can confirm that it is indeed possible to run the same job
on a heterogeneous cluster comprised of AMD and Intel nodes.

I am using OpenMPI 1.7.4 now.

There is a related problem though. I am sharing /opt/openmpi-1.7.4 via NFS
but there does not seem to be a way to tell the nodes where OpenMPI is
located when using non-interactive SSH (using secure key login). SSH does
not seem to parse .bash_profile so I do not know how to tell the jobs on
the nodes where to find OpenMPI except by starting the job with
/opt/openmpi-1.7.4/bin/mpirun.

Regarding open-mx, yes I will look into that next to see if the job is
indeed using it. My msa flag is --mca mx self

Re: [OMPI users] OpenMPI job initializing problem

[Jeff Squyres (jsquyres)]

Both 1.6.x and 1.7.x/1.8.x will need verbs.h to use the native verbs network 
stack.

You can use emulated TCP over IB (e.g., using the OMPI TCP BTL), but it's 
nowhere near as fast/efficient the native verbs network stack.



On Mar 2, 2014, at 10:47 AM, Beichuan Yan  wrote:

> Thanks. Yes I am asking sysadmin to install the libibverbs-devel, which has 
> not been installed. 
> 
> It appears that Open MPI 1.6.5 does not need the verbs.h, but 1.7.4 does need 
> it?
> 
> Beichuan
> 
> -Original Message-
> From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Jeff Squyres 
> (jsquyres)
> Sent: Sunday, March 02, 2014 11:44
> To: Open MPI Users
> Subject: Re: [OMPI users] OpenMPI job initializing problem
> 
> On Mar 2, 2014, at 10:18 AM, Gustavo Correa  wrote:
> 
>> Make sure you have any ofed/openib "devel" packages installed, in case 
>> they exist and yum lists them.
>> This may be a possible reason for missing header files.
> 
> +1
> 
> Look for libibverbs-devel.
> 
> --
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to: 
> http://www.cisco.com/web/about/doing_business/legal/cri/
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users


-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] OpenMPI job initializing problem

[Jeff Squyres (jsquyres)]

On Mar 2, 2014, at 10:18 AM, Gustavo Correa  wrote:

> Make sure you have any ofed/openib "devel" packages installed,
> in case they exist and yum lists them.  
> This may be a possible reason for missing header files.

+1

Look for libibverbs-devel.

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] OpenMPI job initializing problem

[Gustavo Correa]

FWIW, I never added any verbs or openib switches to the OMPI configure command
line, and configure found them and built with Infiniband support.  
Up to OMPI 1.6.5 (I didn't try the 1.7 series).
Unless you are installing your ofed/IB packages in non-standard places,
I guess configure will find them.

When it comes to the tm/Torque/PBS Pro, since it is installed in a non-standard
location, most likely you do need the configure switch 
--with-tm=/opt/pbs/default.
[Unless OMPI configure became much more clever lately ... :)]

Along the lines of Jeff's recommendation, 
to avoid headaches, here I use just the minimal set of configure clauses, 
i.e., only what I really know configure won't be able to find by itself
(mostly commercial compilers' names and location and Torque/tm location).
You can always check what was found/built with, by grepping config.log,
or with the ompi-info command.

Make sure you have any ofed/openib "devel" packages installed,
in case they exist and yum lists them.  
This may be a possible reason for missing header files.

I hope this helps,
Gus Correa


On Mar 2, 2014, at 12:44 PM, Jeff Squyres (jsquyres) wrote:

> FWIW: /usr/include/infiniband/verbs.h is the normal location for verbs.h.  
> Don't add --with-verbs=/usr/include/infinband; it won't work.
> 
> Please send all the information listed here and we can have a look at your 
> logs:
> 
>http://www.open-mpi.org/community/help/
> 
> 
> On Mar 2, 2014, at 7:16 AM, Ralph Castain  wrote:
> 
>> It should have been looking in the same place - check to see where you 
>> installed the inifiniband support. Is "verbs.h" under your /usr/include?
>> 
>> In looking at the code, the 1.6 series searched for verbs.h in 
>> /usr/include/infiniband. The 1.7 series also does (though it doesn't look 
>> quite right to me), but it wouldn't hurt to add it yourself
>> 
>> --with-verbs=/usr/include/infiniband 
>> --with-verbs-libdir=/usr/lib64/infiniband
>> 
>> or something like that
>> 
>> 
>> On Mar 1, 2014, at 11:56 PM, Beichuan Yan  wrote:
>> 
>>> Ralph and Gus,
>>> 
>>> 1. Thank you for your suggestion. I built Open MPI 1.6.5 with the following 
>>> command: 
>>> ./configure 
>>> --prefix=/work4/projects/openmpi/openmpi-1.6.5-gcc-compilers-4.7.3 
>>> --with-tm=/opt/pbs/default --with-openib=  --with-openib-libdir=/usr/lib64
>>> 
>>> In my job script, I need to specify the IB subnet like this:
>>> TCP="--mca btl_tcp_if_include 10.148.0.0/16"
>>> mpirun $TCP -np 64 -hostfile $PBS_NODEFILE ./paraEllip3d input.txt
>>> 
>>> Then my job can get initialized and run correctly each time!
>>> 
>>> 2. However, to build Open MPI 1.7.4 with another command (in order to 
>>> test/compare shared-memory performance of Open MPI):
>>> ./configure 
>>> --prefix=/work4/projects/openmpi/openmpi-1.7.4-gcc-compilers-4.7.3 
>>> --with-tm=/opt/pbs/default --with-verbs=  --with-verbs-libdir=/usr/lib64
>>> 
>>> It gets error as follows:
>>> 
>>> == Modular Component Architecture (MCA) setup
>>> 
>>> checking for subdir args...  
>>> '--prefix=/work4/projects/openmpi/openmpi-1.7.4-gcc-compilers-4.7.3' 
>>> '--with-tm=/opt/pbs/default' '--with-verbs=' 
>>> '--with-verbs-libdir=/usr/lib64' 'CC=gcc' 'CXX=g++'
>>> checking --with-verbs value... simple ok (unspecified)
>>> checking --with-verbs-libdir value... sanity check ok (/usr/lib64)
>>> configure: WARNING: Could not find verbs.h in the usual locations under
>>> configure: error: Cannot continue
>>> 
>>> Our system is Red Hat 6.4. Do we need to install more packages of 
>>> Infiniband? Can you please advise?
>>> 
>>> Thanks,
>>> Beichuan Yan
>>> 
>>> 
>>> -Original Message-
>>> From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Gus Correa
>>> Sent: Friday, February 28, 2014 15:59
>>> To: Open MPI Users
>>> Subject: Re: [OMPI users] OpenMPI job initializing problem
>>> 
>>> HI Beichuan
>>> 
>>> To add to what Ralph said,
>>> the RHEL OpenMPI package probably wasn't built with with PBS Pro support 
>>> either.
>>> Besides, OMPI 1.5.4 (RHEL version) is old.
>>> 
>>> **
>>> 
>>> You will save yourself time and grief if you read the installation FAQs, 
>>> before you install from the source tarball:
>>> 
>>> http://www.open-mpi.org/faq/?category=building
>>> 
>>> However, as Ralph said, that is your best bet, and it is quite easy to get 
>>> right.
>>> 
>>> 
>>> See this FAQ on how to build with PBS Pro support:
>>> 
>>> http://www.open-mpi.org/faq/?category=building#build-rte-tm
>>> 
>>> And this one on how to build with Infiniband support:
>>> 
>>> http://www.open-mpi.org/faq/?category=building#build-p2p
>>> 
>>> Here is how to select the installation directory (--prefix):
>>> 
>>> http://www.open-mpi.org/faq/?category=building#easy-build
>>> 
>>> Here is how to select the compilers (gcc,g++, and gfortran are 

Re: [OMPI users] OpenMPI job initializing problem

[Jeff Squyres (jsquyres)]

FWIW: /usr/include/infiniband/verbs.h is the normal location for verbs.h.  
Don't add --with-verbs=/usr/include/infinband; it won't work.

Please send all the information listed here and we can have a look at your logs:

http://www.open-mpi.org/community/help/


On Mar 2, 2014, at 7:16 AM, Ralph Castain  wrote:

> It should have been looking in the same place - check to see where you 
> installed the inifiniband support. Is "verbs.h" under your /usr/include?
> 
> In looking at the code, the 1.6 series searched for verbs.h in 
> /usr/include/infiniband. The 1.7 series also does (though it doesn't look 
> quite right to me), but it wouldn't hurt to add it yourself
> 
> --with-verbs=/usr/include/infiniband --with-verbs-libdir=/usr/lib64/infiniband
> 
> or something like that
> 
> 
> On Mar 1, 2014, at 11:56 PM, Beichuan Yan  wrote:
> 
>> Ralph and Gus,
>> 
>> 1. Thank you for your suggestion. I built Open MPI 1.6.5 with the following 
>> command: 
>> ./configure 
>> --prefix=/work4/projects/openmpi/openmpi-1.6.5-gcc-compilers-4.7.3 
>> --with-tm=/opt/pbs/default --with-openib=  --with-openib-libdir=/usr/lib64
>> 
>> In my job script, I need to specify the IB subnet like this:
>> TCP="--mca btl_tcp_if_include 10.148.0.0/16"
>> mpirun $TCP -np 64 -hostfile $PBS_NODEFILE ./paraEllip3d input.txt
>> 
>> Then my job can get initialized and run correctly each time!
>> 
>> 2. However, to build Open MPI 1.7.4 with another command (in order to 
>> test/compare shared-memory performance of Open MPI):
>> ./configure 
>> --prefix=/work4/projects/openmpi/openmpi-1.7.4-gcc-compilers-4.7.3 
>> --with-tm=/opt/pbs/default --with-verbs=  --with-verbs-libdir=/usr/lib64
>> 
>> It gets error as follows:
>> 
>> == Modular Component Architecture (MCA) setup
>> 
>> checking for subdir args...  
>> '--prefix=/work4/projects/openmpi/openmpi-1.7.4-gcc-compilers-4.7.3' 
>> '--with-tm=/opt/pbs/default' '--with-verbs=' 
>> '--with-verbs-libdir=/usr/lib64' 'CC=gcc' 'CXX=g++'
>> checking --with-verbs value... simple ok (unspecified)
>> checking --with-verbs-libdir value... sanity check ok (/usr/lib64)
>> configure: WARNING: Could not find verbs.h in the usual locations under
>> configure: error: Cannot continue
>> 
>> Our system is Red Hat 6.4. Do we need to install more packages of 
>> Infiniband? Can you please advise?
>> 
>> Thanks,
>> Beichuan Yan
>> 
>> 
>> -Original Message-
>> From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Gus Correa
>> Sent: Friday, February 28, 2014 15:59
>> To: Open MPI Users
>> Subject: Re: [OMPI users] OpenMPI job initializing problem
>> 
>> HI Beichuan
>> 
>> To add to what Ralph said,
>> the RHEL OpenMPI package probably wasn't built with with PBS Pro support 
>> either.
>> Besides, OMPI 1.5.4 (RHEL version) is old.
>> 
>> **
>> 
>> You will save yourself time and grief if you read the installation FAQs, 
>> before you install from the source tarball:
>> 
>> http://www.open-mpi.org/faq/?category=building
>> 
>> However, as Ralph said, that is your best bet, and it is quite easy to get 
>> right.
>> 
>> 
>> See this FAQ on how to build with PBS Pro support:
>> 
>> http://www.open-mpi.org/faq/?category=building#build-rte-tm
>> 
>> And this one on how to build with Infiniband support:
>> 
>> http://www.open-mpi.org/faq/?category=building#build-p2p
>> 
>> Here is how to select the installation directory (--prefix):
>> 
>> http://www.open-mpi.org/faq/?category=building#easy-build
>> 
>> Here is how to select the compilers (gcc,g++, and gfortran are fine):
>> 
>> http://www.open-mpi.org/faq/?category=building#build-compilers
>> 
>> I hope this helps,
>> Gus Correa
>> 
>> On 02/28/2014 12:36 PM, Ralph Castain wrote:
>>> Almost certainly, the redhat package wasn't built with matching 
>>> infiniband support and so we aren't picking it up. I'd suggest 
>>> downloading the latest 1.7.4 or 1.7.5 nightly tarball, or even the 
>>> latest 1.6 tarball if you want the stable release, and build it 
>>> yourself so you *know* it was built for your system.
>>> 
>>> 
>>> On Feb 28, 2014, at 9:20 AM, Beichuan Yan >> > wrote:
>>> 
 Hi there,
 I am running jobs on clusters with Infiniband connection. They 
 installed OpenMPI v1.5.4 via REDHAT 6 yum package). My problem is 
 that although my jobs gets queued and started by PBS PRO quickly, 
 most of the time they don't really run (occasionally they really run) 
 and give error info like this (even though there are a lot of CPU/IB 
 resource
 available):
 [r2i6n7][[25564,1],0][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_com
 plete_connect]
 connect() to 192.168.159.156 failed: Connection refused (111) And 
 even though when a job gets started and runs 

Re: [OMPI users] OpenMPI job initializing problem

[Ralph Castain]

It should have been looking in the same place - check to see where you 
installed the inifiniband support. Is "verbs.h" under your /usr/include?

In looking at the code, the 1.6 series searched for verbs.h in 
/usr/include/infiniband. The 1.7 series also does (though it doesn't look quite 
right to me), but it wouldn't hurt to add it yourself

--with-verbs=/usr/include/infiniband --with-verbs-libdir=/usr/lib64/infiniband

or something like that


On Mar 1, 2014, at 11:56 PM, Beichuan Yan  wrote:

> Ralph and Gus,
> 
> 1. Thank you for your suggestion. I built Open MPI 1.6.5 with the following 
> command: 
> ./configure 
> --prefix=/work4/projects/openmpi/openmpi-1.6.5-gcc-compilers-4.7.3 
> --with-tm=/opt/pbs/default --with-openib=  --with-openib-libdir=/usr/lib64
> 
> In my job script, I need to specify the IB subnet like this:
> TCP="--mca btl_tcp_if_include 10.148.0.0/16"
> mpirun $TCP -np 64 -hostfile $PBS_NODEFILE ./paraEllip3d input.txt
> 
> Then my job can get initialized and run correctly each time!
> 
> 2. However, to build Open MPI 1.7.4 with another command (in order to 
> test/compare shared-memory performance of Open MPI):
> ./configure 
> --prefix=/work4/projects/openmpi/openmpi-1.7.4-gcc-compilers-4.7.3 
> --with-tm=/opt/pbs/default --with-verbs=  --with-verbs-libdir=/usr/lib64
> 
> It gets error as follows:
> 
> == Modular Component Architecture (MCA) setup
> 
> checking for subdir args...  
> '--prefix=/work4/projects/openmpi/openmpi-1.7.4-gcc-compilers-4.7.3' 
> '--with-tm=/opt/pbs/default' '--with-verbs=' '--with-verbs-libdir=/usr/lib64' 
> 'CC=gcc' 'CXX=g++'
> checking --with-verbs value... simple ok (unspecified)
> checking --with-verbs-libdir value... sanity check ok (/usr/lib64)
> configure: WARNING: Could not find verbs.h in the usual locations under
> configure: error: Cannot continue
> 
> Our system is Red Hat 6.4. Do we need to install more packages of Infiniband? 
> Can you please advise?
> 
> Thanks,
> Beichuan Yan
> 
> 
> -Original Message-
> From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Gus Correa
> Sent: Friday, February 28, 2014 15:59
> To: Open MPI Users
> Subject: Re: [OMPI users] OpenMPI job initializing problem
> 
> HI Beichuan
> 
> To add to what Ralph said,
> the RHEL OpenMPI package probably wasn't built with with PBS Pro support 
> either.
> Besides, OMPI 1.5.4 (RHEL version) is old.
> 
> **
> 
> You will save yourself time and grief if you read the installation FAQs, 
> before you install from the source tarball:
> 
> http://www.open-mpi.org/faq/?category=building
> 
> However, as Ralph said, that is your best bet, and it is quite easy to get 
> right.
> 
> 
> See this FAQ on how to build with PBS Pro support:
> 
> http://www.open-mpi.org/faq/?category=building#build-rte-tm
> 
> And this one on how to build with Infiniband support:
> 
> http://www.open-mpi.org/faq/?category=building#build-p2p
> 
> Here is how to select the installation directory (--prefix):
> 
> http://www.open-mpi.org/faq/?category=building#easy-build
> 
> Here is how to select the compilers (gcc,g++, and gfortran are fine):
> 
> http://www.open-mpi.org/faq/?category=building#build-compilers
> 
> I hope this helps,
> Gus Correa
> 
> On 02/28/2014 12:36 PM, Ralph Castain wrote:
>> Almost certainly, the redhat package wasn't built with matching 
>> infiniband support and so we aren't picking it up. I'd suggest 
>> downloading the latest 1.7.4 or 1.7.5 nightly tarball, or even the 
>> latest 1.6 tarball if you want the stable release, and build it 
>> yourself so you *know* it was built for your system.
>> 
>> 
>> On Feb 28, 2014, at 9:20 AM, Beichuan Yan > > wrote:
>> 
>>> Hi there,
>>> I am running jobs on clusters with Infiniband connection. They 
>>> installed OpenMPI v1.5.4 via REDHAT 6 yum package). My problem is 
>>> that although my jobs gets queued and started by PBS PRO quickly, 
>>> most of the time they don't really run (occasionally they really run) 
>>> and give error info like this (even though there are a lot of CPU/IB 
>>> resource
>>> available):
>>> [r2i6n7][[25564,1],0][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_com
>>> plete_connect]
>>> connect() to 192.168.159.156 failed: Connection refused (111) And 
>>> even though when a job gets started and runs well, it prompts this
>>> error:
>>> -
>>> -
>>> WARNING: There was an error initializing an OpenFabrics device.
>>> Local host: r1i2n6
>>> Local device: mlx4_0
>>> -
>>> - 1. Here is the info from one of the compute nodes:
>>> -bash-4.1$ /sbin/ifconfig
>>> eth0 Link encap:Ethernet HWaddr 8C:89:A5:E3:D2:96 inet 
>>> 

Re: [OMPI users] OpenMPI job initializing problem

[Beichuan Yan]

Ralph and Gus,

1. Thank you for your suggestion. I built Open MPI 1.6.5 with the following 
command: 
./configure --prefix=/work4/projects/openmpi/openmpi-1.6.5-gcc-compilers-4.7.3 
--with-tm=/opt/pbs/default --with-openib=  --with-openib-libdir=/usr/lib64

In my job script, I need to specify the IB subnet like this:
TCP="--mca btl_tcp_if_include 10.148.0.0/16"
mpirun $TCP -np 64 -hostfile $PBS_NODEFILE ./paraEllip3d input.txt

Then my job can get initialized and run correctly each time!

2. However, to build Open MPI 1.7.4 with another command (in order to 
test/compare shared-memory performance of Open MPI):
./configure --prefix=/work4/projects/openmpi/openmpi-1.7.4-gcc-compilers-4.7.3 
--with-tm=/opt/pbs/default --with-verbs=  --with-verbs-libdir=/usr/lib64

It gets error as follows:

== Modular Component Architecture (MCA) setup

checking for subdir args...  
'--prefix=/work4/projects/openmpi/openmpi-1.7.4-gcc-compilers-4.7.3' 
'--with-tm=/opt/pbs/default' '--with-verbs=' '--with-verbs-libdir=/usr/lib64' 
'CC=gcc' 'CXX=g++'
checking --with-verbs value... simple ok (unspecified)
checking --with-verbs-libdir value... sanity check ok (/usr/lib64)
configure: WARNING: Could not find verbs.h in the usual locations under
configure: error: Cannot continue

Our system is Red Hat 6.4. Do we need to install more packages of Infiniband? 
Can you please advise?

Thanks,
Beichuan Yan


-Original Message-
From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Gus Correa
Sent: Friday, February 28, 2014 15:59
To: Open MPI Users
Subject: Re: [OMPI users] OpenMPI job initializing problem

HI Beichuan

To add to what Ralph said,
the RHEL OpenMPI package probably wasn't built with with PBS Pro support either.
Besides, OMPI 1.5.4 (RHEL version) is old.

**

You will save yourself time and grief if you read the installation FAQs, before 
you install from the source tarball:

http://www.open-mpi.org/faq/?category=building

However, as Ralph said, that is your best bet, and it is quite easy to get 
right.


See this FAQ on how to build with PBS Pro support:

http://www.open-mpi.org/faq/?category=building#build-rte-tm

And this one on how to build with Infiniband support:

http://www.open-mpi.org/faq/?category=building#build-p2p

Here is how to select the installation directory (--prefix):

http://www.open-mpi.org/faq/?category=building#easy-build

Here is how to select the compilers (gcc,g++, and gfortran are fine):

http://www.open-mpi.org/faq/?category=building#build-compilers

I hope this helps,
Gus Correa

On 02/28/2014 12:36 PM, Ralph Castain wrote:
> Almost certainly, the redhat package wasn't built with matching 
> infiniband support and so we aren't picking it up. I'd suggest 
> downloading the latest 1.7.4 or 1.7.5 nightly tarball, or even the 
> latest 1.6 tarball if you want the stable release, and build it 
> yourself so you *know* it was built for your system.
>
>
> On Feb 28, 2014, at 9:20 AM, Beichuan Yan  > wrote:
>
>> Hi there,
>> I am running jobs on clusters with Infiniband connection. They 
>> installed OpenMPI v1.5.4 via REDHAT 6 yum package). My problem is 
>> that although my jobs gets queued and started by PBS PRO quickly, 
>> most of the time they don't really run (occasionally they really run) 
>> and give error info like this (even though there are a lot of CPU/IB 
>> resource
>> available):
>> [r2i6n7][[25564,1],0][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_com
>> plete_connect]
>> connect() to 192.168.159.156 failed: Connection refused (111) And 
>> even though when a job gets started and runs well, it prompts this
>> error:
>> -
>> -
>> WARNING: There was an error initializing an OpenFabrics device.
>> Local host: r1i2n6
>> Local device: mlx4_0
>> -
>> - 1. Here is the info from one of the compute nodes:
>> -bash-4.1$ /sbin/ifconfig
>> eth0 Link encap:Ethernet HWaddr 8C:89:A5:E3:D2:96 inet 
>> addr:192.168.159.205 Bcast:192.168.159.255 Mask:255.255.255.0
>> inet6 addr: fe80::8e89:a5ff:fee3:d296/64 Scope:Link UP BROADCAST 
>> RUNNING MULTICAST MTU:1500 Metric:1 RX packets:48879864 errors:0 
>> dropped:0 overruns:17 frame:0 TX packets:39286060 errors:0 dropped:0 
>> overruns:0 carrier:0
>> collisions:0 txqueuelen:1000
>> RX bytes:54771093645 (51.0 GiB) TX bytes:37512462596 (34.9 GiB)
>> Memory:dfc0-dfc2
>> Ifconfig uses the ioctl access method to get the full address 
>> information, which limits hardware addresses to 8 bytes.
>> Because Infiniband address has 20 bytes, only the first 8 bytes are 
>> displayed correctly.
>> Ifconfig is obsolete! For replacement check ip.
>> ib0 Link encap:InfiniBand HWaddr
>> 

Re: [OMPI users] Heterogeneous cluster problem - mixing AMD and Intel nodes

[Brice Goglin]

What's your mpirun or mpiexec command-line?
The error "BTLs attempted: self sm tcp" says that it didn't even try the
MX BTL (for Open-MX). Did you use the MX MTL instead?
Are you sure that you actually use Open-MX when not mixing AMD and Intel
nodes?

Brice



Le 02/03/2014 08:06, Victor a écrit :
> I got 4 x AMD A-10 6800K nodes on loan for a few months and added them
> to my existing Intel nodes.
>
> All nodes share the relevant directories via NFS. I have OpenMPI 1.6.5
> which was build with Open-MX 1.5.3 support networked via GbE.
>
> All nodes run Ubuntu 12.04.
>
> Problem:
>
> I can run a job EITHER on 4 x AMD nodes OR on 2 x Intel nodes, but I
> cannot run a job on any combination of an AMD and Intel node, ie. 1 x
> AMD node + 1 x Intel node = error below.
>
> The error that I get during job setup is:
>
>
> --
> At least one pair of MPI processes are unable to reach each other for
> MPI communications.  This means that no Open MPI device has indicated
> that it can be used to communicate between these processes.  This is
> an error; Open MPI requires that all MPI processes be able to reach
> each other.  This error can sometimes be the result of forgetting to
> specify the "self" BTL.
>   Process 1 ([[2229,1],1]) is on host: AMD-Node-1
>   Process 2 ([[2229,1],8]) is on host: Intel-Node-1
>   BTLs attempted: self sm tcp
> Your MPI job is now going to abort; sorry.
> --
> --
> MPI_INIT has failed because at least one MPI process is unreachable
> from another.  This *usually* means that an underlying communication
> plugin -- such as a BTL or an MTL -- has either not loaded or not
> allowed itself to be used.  Your MPI job will now abort.
> You may wish to try to narrow down the problem;
>  * Check the output of ompi_info to see which BTL/MTL plugins are
>available.
>  * Run your application with MPI_THREAD_SINGLE.
>  * Set the MCA parameter btl_base_verbose to 100 (or mtl_base_verbose,
>if using MTL-based communications) to see exactly which
>communication plugins were considered and/or discarded.
> --
> [AMD-Node-1:3932] *** An error occurred in MPI_Init
> [AMD-Node-1:3932] *** on a NULL communicator
> [AMD-Node-1:3932] *** Unknown error
> [AMD-Node-1:3932] *** MPI_ERRORS_ARE_FATAL: your MPI job will now
> abort
> --
> An MPI process is aborting at a time when it cannot guarantee that all
> of its peer processes in the job will be killed properly.  You should
> double check that everything has shut down cleanly.
>   Reason: Before MPI_INIT completed
>   Local host: AMD-Node-1
>   PID:3932
> --
>
>
>
> What I would like to know is, is it actually difficult (impossible) to
> mix AMD and Intel machines in the same cluster and have them run the
> same job, or am I missing something obvious, or not so obvious when it
> comes to the communication stack on the Intel nodes for example. 
>
> I set up the AMD nodes just yesterday, but I used the same OpenMPI and
> Open-MX versions, however I may have inadvertently done something
> different, so I am thinking (hoping) that it is possible to run such a
> heterogeneous cluster, and that all I need to do is ensure that all
> OpenMPI modules are correctly installed on all nodes.
>
> I need the extra 32 Gb RAM and the AMD nodes bring as I need to
> validate our CFD application, and our additional Intel nodes are still
> not here (ETA 2 weeks).
>
> Thank you,
>
> Victor
>
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

[OMPI users] Heterogeneous cluster problem - mixing AMD and Intel nodes

[Victor]

I got 4 x AMD A-10 6800K nodes on loan for a few months and added them to
my existing Intel nodes.

All nodes share the relevant directories via NFS. I have OpenMPI 1.6.5
which was build with Open-MX 1.5.3 support networked via GbE.

All nodes run Ubuntu 12.04.

Problem:

I can run a job EITHER on 4 x AMD nodes OR on 2 x Intel nodes, but I cannot
run a job on any combination of an AMD and Intel node, ie. 1 x AMD node + 1
x Intel node = error below.

The error that I get during job setup is:

>
> --
> At least one pair of MPI processes are unable to reach each other for
> MPI communications.  This means that no Open MPI device has indicated
> that it can be used to communicate between these processes.  This is
> an error; Open MPI requires that all MPI processes be able to reach
> each other.  This error can sometimes be the result of forgetting to
> specify the "self" BTL.
>   Process 1 ([[2229,1],1]) is on host: AMD-Node-1
>   Process 2 ([[2229,1],8]) is on host: Intel-Node-1
>   BTLs attempted: self sm tcp
> Your MPI job is now going to abort; sorry.
> --
> --
> MPI_INIT has failed because at least one MPI process is unreachable
> from another.  This *usually* means that an underlying communication
> plugin -- such as a BTL or an MTL -- has either not loaded or not
> allowed itself to be used.  Your MPI job will now abort.
> You may wish to try to narrow down the problem;
>  * Check the output of ompi_info to see which BTL/MTL plugins are
>available.
>  * Run your application with MPI_THREAD_SINGLE.
>  * Set the MCA parameter btl_base_verbose to 100 (or mtl_base_verbose,
>if using MTL-based communications) to see exactly which
>communication plugins were considered and/or discarded.
> --
> [AMD-Node-1:3932] *** An error occurred in MPI_Init
> [AMD-Node-1:3932] *** on a NULL communicator
> [AMD-Node-1:3932] *** Unknown error
> [AMD-Node-1:3932] *** MPI_ERRORS_ARE_FATAL: your MPI job will now abort
> --
> An MPI process is aborting at a time when it cannot guarantee that all
> of its peer processes in the job will be killed properly.  You should
> double check that everything has shut down cleanly.
>   Reason: Before MPI_INIT completed
>   Local host: AMD-Node-1
>   PID:3932
> --



What I would like to know is, is it actually difficult (impossible) to mix
AMD and Intel machines in the same cluster and have them run the same job,
or am I missing something obvious, or not so obvious when it comes to the
communication stack on the Intel nodes for example.

I set up the AMD nodes just yesterday, but I used the same OpenMPI and
Open-MX versions, however I may have inadvertently done something
different, so I am thinking (hoping) that it is possible to run such a
heterogeneous cluster, and that all I need to do is ensure that all OpenMPI
modules are correctly installed on all nodes.

I need the extra 32 Gb RAM and the AMD nodes bring as I need to validate
our CFD application, and our additional Intel nodes are still not here (ETA
2 weeks).

Thank you,

Victor