Re: [OMPI users] -host vs -hostfile

[Elken, Tom]

“4 threads”   In MPI, we refer to this as 4 ranks or 4 processes.

So what is your question?   Are you getting errors with PBS?

-Tom

From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of Mahmood 
Naderan
Sent: Monday, July 31, 2017 9:27 AM
To: Open MPI Users 
Subject: Re: [OMPI users] -host vs -hostfile

Excuse me, my fault.. I meant
nodes=2:ppn=2
is 4 threads.

Regards,
Mahmood


On Mon, Jul 31, 2017 at 8:49 PM, r...@open-mpi.org<mailto:r...@open-mpi.org> 
mailto:r...@open-mpi.org>> wrote:
?? Doesn't that tell pbs to allocate 1 node with 2 slots on it? I don't see 
where you get 4

Sent from my iPad

On Jul 31, 2017, at 10:00 AM, Mahmood Naderan 
mailto:mahmood...@gmail.com>> wrote:
OK. The next question is how touse it with torque (PBS)? currently we write 
this directive

Nodes=1:ppn=2

which means 4 threads. Then we omit -np and -hostfile in the mpirun command.

On 31 Jul 2017 20:24, "Elken, Tom" 
mailto:tom.el...@intel.com>> wrote:
Hi Mahmood,

With the -hostfile case, Open MPI is trying to helpfully run things faster by 
keeping both processes on one host.  Ways to avoid this…

On the mpirun command line add:

-pernode  (runs 1 process per node), oe
-npernode 1 ,   but these two has been deprecated in favor of the wonderful 
syntax:
--map-by ppr:1:node

Or you could change your hostfile to:

cluster slots=1

compute-0-0 slots=1


-Tom

From: users 
[mailto:users-boun...@lists.open-mpi.org<mailto:users-boun...@lists.open-mpi.org>]
 On Behalf Of Mahmood Naderan
Sent: Monday, July 31, 2017 6:47 AM
To: Open MPI Users mailto:users@lists.open-mpi.org>>
Subject: [OMPI users] -host vs -hostfile

Hi,

I have stuck at a problem which I don't remember that on previous versions. 
when I run a test program with -host, it works. I mean, the process spans to 
the hosts I specified. However, when I specify -hostfile, it doesn't work!!

mahmood@cluster:mpitest$ /share/apps/computer/openmpi-2.0.1/bin/mpirun -host 
compute-0-0,cluster -np 2 a.out



* hwloc 1.11.2 has encountered what looks like an error from the operating 
system.

*

* Package (P#1 cpuset 0x) intersects with NUMANode (P#1 cpuset 
0xff00) without inclusion!

* Error occurred in topology.c line 1048

*

* The following FAQ entry in the hwloc documentation may help:

*   What should I do when hwloc reports "operating system" warnings?

* Otherwise please report this error message to the hwloc user's mailing list,

* along with the output+tarball generated by the hwloc-gather-topology script.



Hello world from processor cluster.hpc.org<http://cluster.hpc.org>, rank 1 out 
of 2 processors

Hello world from processor compute-0-0.local, rank 0 out of 2 processors

mahmood@cluster:mpitest$ cat hosts

cluster

compute-0-0



mahmood@cluster:mpitest$ /share/apps/computer/openmpi-2.0.1/bin/mpirun 
-hostfile hosts -np 2 a.out



* hwloc 1.11.2 has encountered what looks like an error from the operating 
system.

*

* Package (P#1 cpuset 0x) intersects with NUMANode (P#1 cpuset 
0xff00) without inclusion!

* Error occurred in topology.c line 1048

*

* The following FAQ entry in the hwloc documentation may help:

*   What should I do when hwloc reports "operating system" warnings?

* Otherwise please report this error message to the hwloc user's mailing list,

* along with the output+tarball generated by the hwloc-gather-topology script.



Hello world from processor cluster.hpc.org<http://cluster.hpc.org>, rank 0 out 
of 2 processors

Hello world from processor cluster.hpc.org<http://cluster.hpc.org>, rank 1 out 
of 2 processors


how can I resolve that?
Regards,
Mahmood

___
users mailing list
users@lists.open-mpi.org<mailto:users@lists.open-mpi.org>
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org<mailto:users@lists.open-mpi.org>
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org<mailto:users@lists.open-mpi.org>
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] -host vs -hostfile

[Elken, Tom]

Hi Mahmood,

With the -hostfile case, Open MPI is trying to helpfully run things faster by 
keeping both processes on one host.  Ways to avoid this…

On the mpirun command line add:

-pernode  (runs 1 process per node), oe
-npernode 1 ,   but these two has been deprecated in favor of the wonderful 
syntax:
--map-by ppr:1:node

Or you could change your hostfile to:

cluster slots=1

compute-0-0 slots=1


-Tom

From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of Mahmood 
Naderan
Sent: Monday, July 31, 2017 6:47 AM
To: Open MPI Users 
Subject: [OMPI users] -host vs -hostfile

Hi,

I have stuck at a problem which I don't remember that on previous versions. 
when I run a test program with -host, it works. I mean, the process spans to 
the hosts I specified. However, when I specify -hostfile, it doesn't work!!

mahmood@cluster:mpitest$ /share/apps/computer/openmpi-2.0.1/bin/mpirun -host 
compute-0-0,cluster -np 2 a.out



* hwloc 1.11.2 has encountered what looks like an error from the operating 
system.

*

* Package (P#1 cpuset 0x) intersects with NUMANode (P#1 cpuset 
0xff00) without inclusion!

* Error occurred in topology.c line 1048

*

* The following FAQ entry in the hwloc documentation may help:

*   What should I do when hwloc reports "operating system" warnings?

* Otherwise please report this error message to the hwloc user's mailing list,

* along with the output+tarball generated by the hwloc-gather-topology script.



Hello world from processor cluster.hpc.org, rank 1 out 
of 2 processors

Hello world from processor compute-0-0.local, rank 0 out of 2 processors

mahmood@cluster:mpitest$ cat hosts

cluster

compute-0-0



mahmood@cluster:mpitest$ /share/apps/computer/openmpi-2.0.1/bin/mpirun 
-hostfile hosts -np 2 a.out



* hwloc 1.11.2 has encountered what looks like an error from the operating 
system.

*

* Package (P#1 cpuset 0x) intersects with NUMANode (P#1 cpuset 
0xff00) without inclusion!

* Error occurred in topology.c line 1048

*

* The following FAQ entry in the hwloc documentation may help:

*   What should I do when hwloc reports "operating system" warnings?

* Otherwise please report this error message to the hwloc user's mailing list,

* along with the output+tarball generated by the hwloc-gather-topology script.



Hello world from processor cluster.hpc.org, rank 0 out 
of 2 processors

Hello world from processor cluster.hpc.org, rank 1 out 
of 2 processors


how can I resolve that?
Regards,
Mahmood

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] Many different errors with ompi version 2.1.1

[Elken, Tom]

" i do not think btl/openib can be used with QLogic cards
(please someone correct me if i am wrong)"

You are wrong :) .  The openib BTL is the best one to use for interoperability 
between QLogic and Mellanox IB cards.
The Intel True Scale (the continuation of the QLogic IB product line)  Host SW 
User Guide 
http://www.intel.com/content/dam/support/us/en/documents/network-and-i-o/fabric-products/OFED_Host_Software_UserGuide_G91902_06.pdf
 
says, (I paraphrase):

To run over IB verbs ... for example:

$ mpirun -np 4 -hostfile mpihosts --mca btl self,sm,openib --mca mtl ^psm 
./mpi_app_name


But, as some have suggested, you may make your life simpler and get ~ the same 
or better performance (depending on the workload) if you use the Mlx node as a 
head node and run the job on the 5 QLogic HCA nodes using mtl psm.

-Tom

-Original Message-
From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of Gilles 
Gouaillardet
Sent: Friday, May 19, 2017 12:16 AM
To: Open MPI Users 
Subject: Re: [OMPI users] Many different errors with ompi version 2.1.1

Allan,


i just noted smd has a Mellanox card, while other nodes have QLogic cards.

mtl/psm works best for QLogic while btl/openib (or mtl/mxm) work best for 
Mellanox,

but these are not interoperable. also, i do not think btl/openib can be used 
with QLogic cards

(please someone correct me if i am wrong)


from the logs, i can see that smd (Mellanox) is not even able to use the 
infiniband port.

if you run with 2 MPI tasks, both run on smd and hence btl/vader is used, that 
is why it works

if you run with more than 2 MPI tasks, then smd and other nodes are used, and 
every MPI task fall back to btl/tcp

for inter node communication.

[smd][[41971,1],1][btl_tcp_endpoint.c:803:mca_btl_tcp_endpoint_complete_connect]
connect() to 192.168.1.196 failed: No route to host (113)

this usually indicates a firewall, but since both ssh and oob/tcp are fine, 
this puzzles me.


what if you

mpirun -np 2 --hostfile nodes --mca oob_tcp_if_include 192.168.1.0/24 
--mca btl_tcp_if_include 192.168.1.0/24 --mca pml ob1 --mca btl 
tcp,sm,vader,self  ring

that should work with no error messages, and then you can try with 12 
MPI tasks

(note internode MPI communications will use tcp only)


if you want optimal performance, i am afraid you cannot run any MPI task 
on smd (so mtl/psm can be used )

(btw, make sure PSM support was built in Open MPI)

a suboptimal option is to force MPI communications on IPoIB with

/* make sure all nodes can ping each other via IPoIB first */

mpirun --mca oob_tcp_if_include 192.168.1.0/24 --mca btl_tcp_if_include 
10.1.0.0/24 --mca pml ob1 --mca btl tcp,sm,vader,self



Cheers,


Gilles


On 5/19/2017 3:50 PM, Allan Overstreet wrote:
> Gilles,
>
> On which node is mpirun invoked ?
>
> The mpirun command was involed on node smd.
>
> Are you running from a batch manager?
>
> No.
>
> Is there any firewall running on your nodes ?
>
> No CentOS minimal does not have a firewall installed and Ubuntu 
> Mate's firewall is disabled.
>
> All three of your commands have appeared to run successfully. The 
> outputs of the three commands are attached.
>
> mpirun -np 2 --hostfile nodes --mca oob_tcp_if_include 192.168.1.0/24 
> --mca oob_base_verbose 100 true &> cmd1
>
> mpirun -np 12 --hostfile nodes --mca oob_tcp_if_include 192.168.1.0/24 
> --mca oob_base_verbose 100 true &> cmd2
>
> mpirun -np 2 --hostfile nodes --mca oob_tcp_if_include 192.168.1.0/24 
> --mca oob_base_verbose 100 ring &> cmd3
>
> If I increase the number of processors in the ring program, mpirun 
> will not succeed.
>
> mpirun -np 12 --hostfile nodes --mca oob_tcp_if_include 192.168.1.0/24 
> --mca oob_base_verbose 100 ring &> cmd4
>
>
> On 05/19/2017 02:18 AM, Gilles Gouaillardet wrote:
>> Allan,
>>
>>
>> - on which node is mpirun invoked ?
>>
>> - are you running from a batch manager ?
>>
>> - is there any firewall running on your nodes ?
>>
>>
>> the error is likely occuring when wiring-up mpirun/orted
>>
>> what if you
>>
>> mpirun -np 2 --hostfile nodes --mca oob_tcp_if_include 192.168.1.0/24 
>> --mca oob_base_verbose 100 true
>>
>> then (if the previous command worked)
>>
>> mpirun -np 12 --hostfile nodes --mca oob_tcp_if_include 
>> 192.168.1.0/24 --mca oob_base_verbose 100 true
>>
>> and finally (if both previous commands worked)
>>
>> mpirun -np 2 --hostfile nodes --mca oob_tcp_if_include 192.168.1.0/24 
>> --mca oob_base_verbose 100 ring
>>
>>
>> Cheers,
>>
>> Gilles
>>
>> On 5/19/2017 3:07 PM, Allan Overstreet wrote:
>>> I experiencing many different errors with openmpi version 2.1.1. I 
>>> have had a suspicion that this might be related to the way the 
>>> servers were connected and configured. Regardless below is a diagram 
>>> of how the server are configured.
>>>
>>> __  _
>>>[__]|=|
>>>/::/|_|
>>>HO

Re: [OMPI users] Performance Issues on SMP Workstation

[Elken, Tom]

tatus
and see if you note any discrepancy between your systems

you might also consider upgrading to the latest Open MPI 2.0.2, and see how 
things gi

Cheers,

Gilles

On Thursday, February 2, 2017, mailto:nietham...@hlrs.de>> 
wrote:
Hello Andy,

You can also use the --report-bindings option of mpirun to check which cores
your program will use and to which cores the processes are bound to.


Are you using the same backend compiler on both systems?

Do you have performance tools available on the systems where you can see in
which part of the Program the time is lost? Common tools would be Score-P/
Vampir/CUBE, TAU, extrae/Paraver.

Best
Christoph

On Wednesday, 1 February 2017 21:09:28 CET Andy Witzig wrote:
> Thank you, Bennet.  From my testing, I?ve seen that the application usually
> performs better at much smaller ranks on the workstation.  I?ve tested on
> the cluster and do not see the same response (i.e. see better performance
> with ranks of -np 15 or 20).   The workstation is not shared and is not
> doing any other work.  I ran the application on the Workstation with top
> and confirmed that 20 procs were fully loaded.
>
> I?ll look into the diagnostics you mentioned and get back with you.
>
> Best regards,
> Andy
>
> On Feb 1, 2017, at 6:15 PM, Bennet Fauber > 
> wrote:
>
> How do they compare if you run a much smaller number of ranks, say -np 2 or
> 4?
>
> Is the workstation shared and doing any other work?
>
> You could insert some diagnostics into your script, for example,
> uptime and free, both before and after running your MPI program and
> compare.
>
> You could also run top in batch mode in the background for your own
> username, then run your MPI program, and compare the results from top.
> We've seen instances where the MPI ranks only get distributed to a
> small number of processors, which you see if they all have small
> percentages of CPU.
>
> Just flailing in the dark...
>
> -- bennet
>
> On Wed, Feb 1, 2017 at 6:36 PM, Andy Witzig 
> > wrote:
> > Thank for the idea.  I did the test and only get a single host.
> >
> > Thanks,
> > Andy
> >
> > On Feb 1, 2017, at 5:04 PM, r...@open-mpi.org wrote:
> >
> > Simple test: replace your executable with ?hostname?. If you see multiple
> > hosts come out on your cluster, then you know why the performance is
> > different.
> >
> > On Feb 1, 2017, at 2:46 PM, Andy Witzig > 
> > wrote:
> >
> > Honestly, I?m not exactly sure what scheme is being used.  I am using the
> > default template from Penguin Computing for job submission.  It looks
> > like:
> >
> > #PBS -S /bin/bash
> > #PBS -q T30
> > #PBS -l walltime=24:00:00,nodes=1:ppn=20
> > #PBS -j oe
> > #PBS -N test
> > #PBS -r n
> >
> > mpirun $EXECUTABLE $INPUT_FILE
> >
> > I?m not configuring OpenMPI anywhere else. It is possible the Penguin
> > Computing folks have pre-configured my MPI environment.  I?ll see what I
> > can find.
> >
> > Best regards,
> > Andy
> >
> > On Feb 1, 2017, at 4:32 PM, Douglas L Reeder 
> > > wrote:
> >
> > Andy,
> >
> > What allocation scheme are you using on the cluster. For some codes we see
> > noticeable differences using fillup vs round robin, not 4x though. Fillup
> > is more shared memory use while round robin uses more infinniband.
> >
> > Doug
> >
> > On Feb 1, 2017, at 3:25 PM, Andy Witzig > 
> > wrote:
> >
> > Hi Tom,
> >
> > The cluster uses an Infiniband interconnect.  On the cluster I?m
> > requesting: #PBS -l walltime=24:00:00,nodes=1:ppn=20.  So technically,
> > the run on the cluster should be SMP on the node, since there are 20
> > cores/node.  On the workstation I?m just using the command: mpirun -np 20
> > ?. I haven?t finished setting Torque/PBS up yet.
> >
> > Best regards,
> > Andy
> >
> > On Feb 1, 2017, at 4:10 PM, Elken, Tom > 
> > wrote:
> >
> > For this case:  " a cluster system with 2.6GHz Intel Haswell with 20 cores
> > / node and 128GB RAM/node.  "
> >
> > are you running 5 ranks per node on 4 nodes?
> > What interconnect are you using for the cluster?
> >
> > -Tom
> >
> > -Original Message-
> > From: users [mailto:users-boun...@lists.open-mpi.org] On 
> > Behalf Of Andrew
> > Witzig
> > Sent: Wednesday, February 01, 2017 1:37 PM
> > To: Open MPI Users
> > Subject: Re: [OMPI users] Performance Issues on SMP Workstation
> >
> > By the way, the workstation has a total of 36 cores / 72 threads, so using
> > mpiru

Re: [OMPI users] Performance Issues on SMP Workstation

[Elken, Tom]

For this case:  " a cluster system with 2.6GHz Intel Haswell with 20 cores / 
node and 128GB RAM/node.  "

are you running 5 ranks per node on 4 nodes?
What interconnect are you using for the cluster?

-Tom

> -Original Message-
> From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of Andrew
> Witzig
> Sent: Wednesday, February 01, 2017 1:37 PM
> To: Open MPI Users
> Subject: Re: [OMPI users] Performance Issues on SMP Workstation
> 
> By the way, the workstation has a total of 36 cores / 72 threads, so using 
> mpirun
> -np 20 is possible (and should be equivalent) on both platforms.
> 
> Thanks,
> cap79
> 
> > On Feb 1, 2017, at 2:52 PM, Andy Witzig  wrote:
> >
> > Hi all,
> >
> > I’m testing my application on a SMP workstation (dual Intel Xeon E5-2697 V4
> 2.3 GHz Intel Broadwell (boost 2.8-3.1GHz) processors 128GB RAM) and am
> seeing a 4x performance drop compared to a cluster system with 2.6GHz Intel
> Haswell with 20 cores / node and 128GB RAM/node.  Both applications have
> been compiled using OpenMPI 1.6.4.  I have tried running:
> >
> > mpirun -np 20 $EXECUTABLE $INPUT_FILE
> > mpirun -np 20 --mca btl self,sm $EXECUTABLE $INPUT_FILE
> >
> > and others, but cannot achieve the same performance on the workstation as is
> seen on the cluster.  The workstation outperforms on other non-MPI but multi-
> threaded applications, so I don’t think it’s a hardware issue.
> >
> > Any help you can provide would be appreciated.
> >
> > Thanks,
> > cap79
> > ___
> > users mailing list
> > users@lists.open-mpi.org
> > https://rfd.newmexicoconsortium.org/mailman/listinfo/users
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] locked memory and queue pairs

[Elken, Tom]

Hi Mike,

In this file, 
$ cat /etc/security/limits.conf
...
< do you see at the end ... >

* hard memlock unlimited
* soft memlock unlimited
# -- All InfiniBand Settings End here --
?

-Tom

> -Original Message-
> From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Michael Di
> Domenico
> Sent: Thursday, March 10, 2016 8:55 AM
> To: Open MPI Users
> Subject: [OMPI users] locked memory and queue pairs
> 
> when i try to run an openmpi job with >128 ranks (16 ranks per node)
> using alltoall or alltoallv, i'm getting an error that the process was
> unable to get a queue pair.
> 
> i've checked the max locked memory settings across my machines;
> 
> using ulimit -l in and outside of mpirun and they're all set to unlimited
> pam modules to ensure pam_limits.so is loaded and working
> the /etc/security/limits.conf is set for soft/hard mem to unlimited
> 
> i tried a couple of quick mpi config settings i could think of;
> 
> -mca mtl ^psm no affect
> -mca btl_openib_flags 1 no affect
> 
> the openmpi faq says to tweak some mtt values in /sys, but since i'm
> not on mellanox that doesn't apply to me
> 
> the machines are rhel 6.7, kernel 2.6.32-573.12.1(with bundled ofed),
> running on qlogic single-port infiniband cards, psm is enabled
> 
> other collectives seem to run okay, it seems to only be alltoall comms
> that fail and only at scale
> 
> i believe (but can't prove) that this worked at one point, but i can't
> recall when i last tested it.  so it's reasonable to assume that some
> change to the system is preventing this.
> 
> the question is, where should i start poking to find it?
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: http://www.open-
> mpi.org/community/lists/users/2016/03/28673.php

Re: [OMPI users] OpenMPI build and OFED versions - rebuild necessary?

[Elken, Tom]

When you change major RHEL release # and OFED version #, it's a pretty safe bet 
that you would need to rebuild Open MPI.

-Tom
P.S.  There was no Open Fabrics OFED 2.4   (Release #s jumped from 1.5 to 3.5) 
, so I guess this is a Mellanox OFED 2.4.

From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Sasso, John (GE 
Power & Water, Non-GE)
Sent: Thursday, January 22, 2015 6:59 AM
To: us...@open-mpi.org
Subject: [OMPI users] OpenMPI build and OFED versions - rebuild necessary?

On RHEL 5 hosts which have OFED 1.5 installed, we have builds of OpenMPI 1.4.5 
and 1.6.5 in place.

On RHEL 6 hosts we have OFED 2.4 installed.  Will we need to rebuild OpenMPI 
1.4.5 and 1.6.5, or can our existing builds of such still work on the RHEL 6 
hosts?

--john

Re: [OMPI users] Running OSU benchmark latency test with Open MPI version 1.6.5 and version 1.8.1

[Elken, Tom]

Hi Na Zhang,

It seems likely that on your Open MPI 1.8.1 run that you have the 2 ranks 
running on one host whereas on the 1.6.5 results they are running on 2 hosts.  
You should be able to verify that by running top on one of the nodes during the 
1.8.1 runs and see if you have 2 or 0 osu_latency processes running on that 
node.

Also, it would be useful to see your mpirun or other launch command for your 
job for both MPI versions.

Thanks,
Tom


From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Na Zhang
Sent: Tuesday, July 15, 2014 11:01 AM
To: us...@open-mpi.org
Subject: Re: [OMPI users] Running OSU benchmark latency test with Open MPI 
version 1.6.5 and version 1.8.1

Forgot to add the config info about the tests in previous post.

We have FDR IB, two nodes (HP DL 380 G8 sever) communicating through a Mellanox 
Switch, RDMA mode, hyperthreading enabled.
Thanks,
Na Zhang

On Tue, Jul 15, 2014 at 12:00 PM, Na Zhang 
mailto:nkuzhan...@gmail.com>> wrote:
Dear developers and users,

I am trying to run OSU benchmark tests (like osu_latency, osu_bw, etc)  with 
Open MPI. While I was able to run the tests with both versions (1.6.5, 1.8.1, 
same default build), I got disparate performance results. Please see data 
below. I wonder what reason would cause the performance difference.

And also, I was trying to run the tests with latest release 1.8.2, got the 
warning like below, and got the results similar to version 1.8.1. How can I 
avoid this warning?
--
WARNING: a request was made to bind a process. While the system
supports binding the process itself, at least one node does NOT
support binding memory to the process location.

  Node:  hostname

This is a warning only; your job will continue, though performance may
be degraded.
--

Thanks,
Na Zhang

Version 1.6.5
# OSU MPI Latency Test v4.3
# Size  Latency (us)
0   1.83
1   1.11
2   1.11
4   1.11
8   1.13
16  1.15
32  1.17
64  1.22
128 1.54
256 1.73
512 1.90
10242.24
20482.93
40963.59
81924.96
16384   7.81
32768  11.22
65536  18.18
131072 32.10
262144 59.91
524288115.56
1048576   226.84
2097152   449.71
4194304   894.75

Version 1.8.1
# OSU MPI Latency Test v4.3
# Size  Latency (us)
0   0.22
1   0.27
2   0.27
4   0.27
8   0.27
16  0.28
32  0.28
64  0.31
128 0.44
256 0.47
512 0.48
10240.60
20480.83
40961.30
81921.95
16384   3.45
32768   5.79
65536   8.88
131072 15.79
262144 28.78
524288 53.50
1048576   104.09
2097152   202.54
4194304   398.43

Re: [OMPI users] Question about scheduler support (or is this about cmake now?)

[Elken, Tom]

Martin Siegert wrote:
> Just set LDFLAGS='-Wl,-rpath,/usr/local/xyz/lib64' with autotools.
> With cmake? Really complicated.

John Cary wrote:
> For cmake,
> 
> -DCMAKE_SHARED_LINKER_FLAGS:STRING=-Wl,-rpath,'$HDF5_SERSH_DIR/lib'
> or
> -DCMAKE_EXE_LINKER_FLAGS:STRING=-Wl,-rpath,'$HDF5_SERSH_DIR/lib'
[Tom] 
OK, so you verified the "really complicated" comment.

It seems clear to me that the OpenMPI developers are not going to switch to 
Cmake.
So why is this discussion continuing?

-Tom

> 
> I don't have a dog in this, but I will say that we have found supporting
> Windows
> to be much easier with cmake.  If that is not an issue, then autotools is
> is just fine too.  We do both and are happy with either.
> 
> Yes, one must build cmake to use it.  Does not seem to be a critical
> issue to me if one wants to support Windows, as you have to install
> something (e.g., cygwin) to use autotools.
> 
> We looked into cmake for openmpi a while ago, but only because we wondered
> whether there was much interest in supporting Windows. There wasn't.
> 
> As to compiler support, we build our codes on all of
> 
> Clang, OS X native (which is variants of GNU and Clang),
> PGI, Intel, Cray, Microsoft Visual, IBM BlueGene (xl).
> 
> Have not tried  Absoft, HP-UX, Oracle Solaris (Linux and Solaris), Tru64.
> Only rarely are we seeing the last three OS's anymore.  No requests.
> But I am confident cmake could do these.
> 
> ..John
> 
> 
> 
> 
> 
> 
> On 5/16/2014 3:00 PM, Martin Siegert wrote:
> > +1 even if cmake would make life easier for the developpers, you may
> > want to consider those sysadmins/users who actually need to compile
> > and install the software. And for those cmake is a nightmare. Everytime
> > I run into a software package that uses cmake it makes me cringe.
> > gromacs is the perfect example - it has become orders of magnitudes
> > more complicated to compile just because it now uses cmake. I still
> > have not succeeded cross compiling (compiling on a machine with a
> > different processor than the code will later run on) gromacs. This was
> > trivial before they switched to cmake.
> > Another example: want to add RPATH to the executables/libraries?
> > Just set LDFLAGS='-Wl,-rpath,/usr/local/xyz/lib64' with autotools.
> > With cmake? Really complicated.
> >
> > Please, just say no.
> >
> > Cheers,
> > Martin
> >
> > On Fri, May 16, 2014 at 08:33:15PM +, Hjelm, Nathan T wrote:
> >> +1 the bootstrapping issue is 50% of the reason I will never use CMake for
> any production code.
> >>
> >> vygr:~ hjelmn$ type -p cmake
> >> vygr:~ hjelmn$
> >>
> >> Nada, zilch, nothing on standard OS X install. I do not want to put an 
> >> extra
> requirement on my users. Nor do I want something as simple-minded as CMake.
> autotools works great for me.
> >>
> >> -Nathan
> >>
> >> 
> >> From: users [users-boun...@open-mpi.org] on behalf of Ralph Castain
> [r...@open-mpi.org]
> >> Sent: Friday, May 16, 2014 2:07 PM
> >> To: Open MPI Users
> >> Subject: Re: [OMPI users] Question about scheduler support
> >>
> >> On May 16, 2014, at 1:03 PM, Fabricio Cannini
> mailto:fcann...@gmail.com>> wrote:
> >>
> >> Em 16-05-2014 10:06, Jeff Squyres (jsquyres) escreveu:
> >> On May 15, 2014, at 8:00 PM, Fabricio Cannini
> mailto:fcann...@gmail.com>>
> >> wrote:
> >>
> >> Nobody is disagreeing that one could find a way to make CMake
> >> work - all we are saying is that (a) CMake has issues too, just
> >> like autotools, and (b) we have yet to see a compelling reason to
> >> undertake the transition...which would have to be a *very*
> >> compelling one.
> >>
> >> I was simply agreeing with Maxime about why it could work. ;)
> >>
> >> But if you and the other devels are fine with it, i'm fine too.
> >>
> >> FWIW, simply for my own curiosity's sake, if someone could confirm
> >> deny whether cmake:
> >>
> >> 1. Supports the following compiler suites: GNU (that's a given, I
> >> assume), Clang, OS X native (which is variants of GNU and Clang),
> >> Absoft, PGI, Intel, Cray, HP-UX, Oracle Solaris (Linux and Solaris),
> >> Tru64, Microsoft Visual, IBM BlueGene (I think that's gcc, but am
> >> not entirely sure).  (some of these matter mainly to hwloc, not
> >> necessarily OMPI)
> >>
> >> Not 100% confirmed, but this is good evidence that cmake does indeed
> supports all these suites. See the file list:
> >>
> >> http://fr2.rpmfind.net//linux/RPM/centos/6.5/x86_64/Packages/cmake-
> 2.6.4-5.el6.x86_64.html
> >>
> >> http://fr2.rpmfind.net//linux/RPM/dag/redhat/el6/x86_64/extras/cmake-
> 2.8.8-1.el6.rfx.x86_64.html
> >>
> >>
> http://fr2.rpmfind.net//linux/RPM/opensuse/factory/aarch64/aarch64/cmake-
> 3.0.0~rc4-2.1.aarch64.html
> >>
> >> 2. Bootstrap a tarball such that an end user does not need to have
> >> cmake installed.
> >>
> >> What do you mean by 'bootstrapping a tarball' ?
> >>
> >> If someone doesn't have cmake installed and downlo

Re: [OMPI users] mpirun runs in serial even I set np to several processors

[Elken, Tom]

That’s OK.  Many of us make that mistake, though often as a typo.
One thing that helps is that the correct spelling of Open MPI has a space in 
it, but OpenMP does not.
If not aware what OpenMP is, here is a link: http://openmp.org/wp/

What makes it more confusing is that more and more apps. offer the option of 
running in a hybrid mode, such as WRF, with OpenMP threads running over MPI 
ranks with the same executable.  And sometimes that MPI is Open MPI.

Cheers,
-Tom

From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Djordje Romanic
Sent: Monday, April 14, 2014 1:28 PM
To: Open MPI Users
Subject: Re: [OMPI users] mpirun runs in serial even I set np to several 
processors

OK guys... Thanks for all this info. Frankly, I didn't know these diferences 
between OpenMP and OpenMPI. The commands:
which mpirun
which mpif90
which mpicc
give,
/usr/bin/mpirun
/usr/bin/mpif90
/usr/bin/mpicc
respectively.
A tutorial on how to compile WRF 
(http://www.mmm.ucar.edu/wrf/OnLineTutorial/compilation_tutorial.php) provides 
a test program to test MPI. I ran the program and it gave me the output of 
successful run, which is:
-
C function called by Fortran
Values are xx = 2.00 and ii = 1
status = 2
SUCCESS test 2 fortran + c + netcdf + mpi
-
It uses mpif90 and mpicc for compiling. Below is the output of 'ldd ./wrf.exe':

linux-vdso.so.1 =>  (0x7fff584e7000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 
(0x7f4d160ab000)
libgfortran.so.3 => /usr/lib/x86_64-linux-gnu/libgfortran.so.3 
(0x7f4d15d94000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x7f4d15a97000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x7f4d15881000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x7f4d154c1000)
/lib64/ld-linux-x86-64.so.2 (0x7f4d162e8000)
libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0 
(0x7f4d1528a000)

On Mon, Apr 14, 2014 at 4:09 PM, Gus Correa 
mailto:g...@ldeo.columbia.edu>> wrote:
Djordje

Your WRF configure file seems to use mpif90 and mpicc (line 115 & following).
In addition, it also seems to have DISABLED OpenMP (NO TRAILING "I")
(lines 109-111, where OpenMP stuff is commented out).
So, it looks like to me your intent was to compile with MPI.

Whether it is THIS MPI (OpenMPI) or another MPI (say MPICH, or MVAPICH,
or Intel MPI, or Cray, or ...) only your environment can tell.

What do you get from these commands:

which mpirun
which mpif90
which mpicc

I never built WRF here (but other people here use it).
Which input do you provide to the command that generates the configure
script that you sent before?
Maybe the full command line will shed some light on the problem.


I hope this helps,
Gus Correa

On 04/14/2014 03:11 PM, Djordje Romanic wrote:
to get help :)



On Mon, Apr 14, 2014 at 3:11 PM, Djordje Romanic 
mailto:djord...@gmail.com>
>> wrote:

Yes, but I was hoping to get. :)


On Mon, Apr 14, 2014 at 3:02 PM, Jeff Squyres (jsquyres)
mailto:jsquy...@cisco.com> 
>> wrote:

If you didn't use Open MPI, then this is the wrong mailing list
for you.  :-)

(this is the Open MPI users' support mailing list)


On Apr 14, 2014, at 2:58 PM, Djordje Romanic 
mailto:djord...@gmail.com>
>> wrote:

 > I didn't use OpenMPI.
 >
 >
 > On Mon, Apr 14, 2014 at 2:37 PM, Jeff Squyres (jsquyres)
mailto:jsquy...@cisco.com> 
>> wrote:
 > This can also happen when you compile your application with
one MPI implementation (e.g., Open MPI), but then mistakenly use
the "mpirun" (or "mpiexec") from a different MPI implementation
(e.g., MPICH).
 >
 >
 > On Apr 14, 2014, at 2:32 PM, Djordje Romanic
mailto:djord...@gmail.com> 
>> wrote:
 >
 > > I compiled it with: x86_64 Linux, gfortran compiler with
gcc   (dmpar). dmpar - distributed memory option.
 > >
 > > Attached is the self-generated configuration file. The
architecture specification settings start at line 107. I didn't
use Open MPI (shared memory option).
 > >
 > >
 > > On Mon, Apr 14, 2014 at 1:23 PM, Dave Goodell (dgoodell)
mailto:dgood...@cisco.com> 
>> wrote:
 > > On Apr 14, 2014, at 12:15 PM, Djordje Romanic
mailto:djord...@gmail.com> 
>> wrote:
 > >
 > > > When I start wrf with mpirun -np 4 ./wrf.exe, I get this:
 > > > 

Re: [OMPI users] ofed installation

[Elken, Tom]

Just to give a quick pointer...  RHEL 6.4 is pretty new, and OFED 1.5.3.2 is 
pretty old, so that is likely the root of your issue.

I believe the first OFED that supported RHEL 6.4 , which is roughly = CentOS 
6.4, is OFED 3.5-1:
http://www.openfabrics.org/downloads/OFED/ofed-3.5-1/

What also might work for you (with newer packages and more bug fixes is 3.5-2 
RC2 from
http://www.openfabrics.org/downloads/OFED/ofed-3.5-2/

-Tom

From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Robo Beans
Sent: Wednesday, October 30, 2013 1:30 PM
To: Open MPI Users
Subject: Re: [OMPI users] ofed installation

I did try ofed forum: 
https://www.openfabrics.org/forum/7-installation/882-ofed-1532.html#882

but was wondering if group members faced similar issue as well while installing 
ofed and what steps they followed to resolve it? Thanks!


Robo

On Wed, Oct 30, 2013 at 1:22 PM, Ralph Castain 
mailto:r...@open-mpi.org>> wrote:
Looks to me like that's an error from the OFED installer, not something from 
OMPI. Have you tried their mailing list?


On Oct 30, 2013, at 1:05 PM, Robo Beans 
mailto:robobe...@gmail.com>> wrote:


Hello everyone,

I am trying to install ofed-1.5.3.2 on centos 6.4 using 
install.pl provided but getting following error:

/lib/modules/2.6.32-358.el6.x86_64/build/scripts is required to build kernel-ib 
RPM.

// info. about current kernel

$ uname -a

Linux scc-10-2-xx-xx-xyz.com 
2.6.32-358.el6.x86_64 #1 SMP Fri Feb 22 00:31:26 UTC 2013 x86_64 x86_64 x86_64 
GNU/Linux



If possible, could someone from the group please direct me what needs to be 
done to resolve this issue? Thanks!



Robo

___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users


___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] Debugging Runtime/Ethernet Problems

[Elken, Tom]

> The trouble is when I try to add some "--mca" parameters to force it to
> use TCP/Ethernet, the program seems to hang.  I get the headers of the
> "osu_bw" output, but no results, even on the first case (1 byte payload
> per packet).  This is occurring on both the IB-enabled nodes, and on the
> Ethernet-only nodes.  The specific syntax I was using was:  "mpirun
> --mca btl ^openib --mca btl_tcp_if_exclude ib0 ./osu_bw"
 
When we want to run over TCP and IPoIB on an IB/PSM equipped cluster, we use:
--mca btl sm --mca btl tcp,self --mca btl_tcp_if_exclude eth0 --mca 
btl_tcp_if_include ib0 --mca mtl ^psm

based on this, it looks like the following might work for you:
--mca btl sm,tcp,self --mca btl_tcp_if_exclude ib0 --mca btl_tcp_if_include 
eth0 --mca btl ^openib

If you don't have ib0 ports configured on the IB nodes, probably you don't need 
the" --mca btl_tcp_if_exclude ib0."

-Tom

> 
> The problem occurs at least with OpenMPI 1.6.3 compiled with GNU 4.4
> compilers, with 1.6.3 compiled with Intel 13.0.1 compilers, and with
> 1.6.5 compiled with Intel 13.0.1 compilers.  I haven't tested any other
> combinations yet.
> 
> Any ideas here?  It's very possible this is a system configuration
> problem, but I don't know where to look.  At this point, any ideas would
> be welcome, either about the specific situation, or general pointers on
> mpirun debugging flags to use.  I can't find much in the docs yet on
> run-time debugging for OpenMPI, as opposed to debugging the application.
>  Maybe I'm just looking in the wrong place.
> 
> 
> Thanks,
> 
> --
> Lloyd Brown
> Systems Administrator
> Fulton Supercomputing Lab
> Brigham Young University
> http://marylou.byu.edu
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] ERROR: At least one pair of MPI processes are unable to reach each other for MPI communications.

[Elken, Tom]

On 8/3/13 7:09 PM, RoboBeans wrote:
On first 7 nodes:

[mpidemo@SERVER-3 ~]$ ofed_info | head -n 1
OFED-1.5.3.2:

On last 4 nodes:

[mpidemo@sv-2 ~]$ ofed_info | head -n 1
-bash: ofed_info: command not found
[Tom]
This is a pretty good clue that OFED is not installed on the last 4 nodes.  You 
should fix that by installing OFED 1.5.3.2 on the last 4 nodes, OR better (but 
more work) install a newer OFED such as 1.5.4.1 or 3.5 on ALL the nodes (You 
need to look at the OFED release notes to see if your OS is supported by these 
OFEDs).

BTW, since you are using QLogic HCAs, they typically work with the best 
performance when using the PSM API to the HCA.  PSM is part of OFED.  To use 
this by default with Open MPI, you can build Open MPI as follows:
./configure --with-psm --prefix=
make
make install

With an Open MPI that is already built, you can try to use PSM as follows:
mpirun ... --mca mtl psm --mca btl ^openib ...

-Tom

[mpidemo@sv-2 ~]$ which ofed_info
/usr/bin/which: no ofed_info in 
(/usr/OPENMPI/openmpi-1.7.2/bin:/usr/OPENMPI/openmpi-1.7.2/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/bin/:/usr/lib/:/usr/lib:/usr:/usr/:/bin/:/usr/lib/:/usr/lib:/usr:/usr/)


Are there some specific locations where I should look for ofed_info? How can I 
make sure that ofed was installed on a node or not?

Thanks again!!!


On 8/3/13 5:52 PM, Ralph Castain wrote:
Are the ofed versions the same across all the machines? I would suspect that 
might be the problem.


On Aug 3, 2013, at 4:06 PM, RoboBeans 
mailto:robobe...@gmail.com>> wrote:


Hi Ralph, I tried using 1.5.4, 1.6.5 and 1.7.2 (compiled from source code) with 
no configuration arguments but I am facing the same issue. When I run a job 
using 1.5.4 (installed using yum), I get warnings but it doesn't affect my 
output.

Example of warning that I get:

sv-2.7960ipath_userinit: Mismatched user minor version (12) and driver minor 
version (11) while context sharing. Ensure that driver and library are from the 
same release.

Each system has a QLogic card ("QLE7342-CK dual port IB card"), with the same 
OS but different kernel revision no. (e.g. 2.6.32-358.2.1.el6.x86_64, 
2.6.32-358.el6.x86_64).

Thank you for your time.

On 8/3/13 2:05 PM, Ralph Castain wrote:
Hmmm...strange indeed. I would remove those four configure options and give it 
a try. That will eliminate all the obvious things, I would think, though they 
aren't generally involved in the issue shown here. Still, worth taking out 
potential trouble sources.

What is the connectivity between SERVER-2 and node 100? Should I assume that 
the first seven nodes are connected via one type of interconnect, and the other 
four are connected to those seven by another type?


On Aug 3, 2013, at 1:30 PM, RoboBeans 
mailto:robobe...@gmail.com>> wrote:


Thanks for looking into in Ralph. I modified the hosts file but I am still 
getting the same error. Any other pointers you can think of? The difference 
between this 1.7.2 installation and 1.5.4 is that I installed 1.5.4 using yum 
and for 1.7.2, I used the source code and configured with 
--enable-event-thread-support --enable-opal-multi-threads 
--enable-orte-progress-threads --enable-mpi-thread-multiple
. Am I missing something here?

//**

$ cat mpi_hostfile

x.x.x.22 slots=15 max-slots=15
x.x.x.24 slots=2 max-slots=2
x.x.x.26 slots=14 max-slots=14
x.x.x.28 slots=16 max-slots=16
x.x.x.29 slots=14 max-slots=14
x.x.x.30 slots=16 max-slots=16
x.x.x.41 slots=46 max-slots=46
x.x.x.101 slots=46 max-slots=46
x.x.x.100 slots=46 max-slots=46
x.x.x.102 slots=22 max-slots=22
x.x.x.103 slots=22 max-slots=22

//**
$ mpirun -d --display-map -np 10 --hostfile mpi_hostfile --bynode ./test

[SERVER-2:08907] procdir: /tmp/openmpi-sessions-mpidemo@SERVER-2_0/62216/0/0
[SERVER-2:08907] jobdir: /tmp/openmpi-sessions-mpidemo@SERVER-2_0/62216/0
[SERVER-2:08907] top: openmpi-sessions-mpidemo@SERVER-2_0
[SERVER-2:08907] tmp: /tmp
CentOS release 6.4 (Final)
Kernel \r on an \m
CentOS release 6.4 (Final)
Kernel \r on an \m
CentOS release 6.4 (Final)
Kernel \r on an \m
[SERVER-3:32517] procdir: /tmp/openmpi-sessions-mpidemo@SERVER-3_0/62216/0/1
[SERVER-3:32517] jobdir: /tmp/openmpi-sessions-mpidemo@SERVER-3_0/62216/0
[SERVER-3:32517] top: openmpi-sessions-mpidemo@SERVER-3_0
[SERVER-3:32517] tmp: /tmp
CentOS release 6.4 (Final)
Kernel \r on an \m
CentOS release 6.4 (Final)
Kernel \r on an \m
[SERVER-6:11595] procdir: /tmp/openmpi-sessions-mpidemo@SERVER-6_0/62216/0/4
[SERVER-6:11595] jobdir: /tmp/openmpi-sessions-mpidemo@SERVER-6_0/62216/0
[SERVER-6:11595] top: openmpi-sessions-mpidemo@SERVER-6_0
[SERVER-6:11595] tmp: /tmp
[SERVER-4:27445] procdir: /tmp/openmpi-sessions-mpidemo@SERVER-4_0/62216/0/2
[SERVER-4:27445] jobdir: /tmp/openmpi-sessions-mpidemo@SERVER-4_0/62216/0
[SERVER-4:27445] top: openmpi-sessions-mpidemo@SERVER-4_0
[SERVER-4:27445

Re: [OMPI users] knem/openmpi performance?

[Elken, Tom]

> "Elken, Tom"  writes:
> > there is a kcopy module
> > that assists shared memory MPI bandwidth in a way similar to knem.
> 
> Somewhat OT, but is it documented?  I went looking some time ago, and
> couldn't find anything more useful than the source.
[Tom] 
Strangely no.  Though there were some marketing/sales foils some time ago that 
played it up.

I have on my To-Do list to get more environment variables documented, including 
these two relevant to kcopy:

PSM_SHM_KCOPY=none =put, =get :   'put' is the default.

PSM_KASSIST_MODE =none =kcopy-put, =kcopy-get :   'kcopy-put' is the default.

So either of these could be used to turn off kcopy (using =none)  so you could 
measure performance with and without kcopy.
The second of these is general enough to handle knem too.  But we need testing 
before we can say that knem is supported.

-Tom


> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] knem/openmpi performance?

[Elken, Tom]

> I was hoping that someone might have some examples of real application
> behaviour rather than micro benchmarks. It can be crazy hard to get that
> information from users.
[Tom] 
I don't have direct performance information on knem, but with Intel's (formerly 
QLogic's) PSM layer as delivered in our software stack (Intel True Scale Fabric 
Suite) known as IFS, there is a kcopy module that assists shared memory MPI 
bandwidth in a way similar to knem.

We ran SPEC MPI2007 benchmarks quite a while ago and kcopy showed about a 2% 
advantage on average over the 13 applications that make up the suite. -- There 
were codes which did not benefit, but no downside.  This was run over 16 nodes 
at 8 cores per node, so not very fat nodes.

More interestingly, on one of our software revs. a few years ago, a bug crept 
in which disabled kcopy.  A customer filed an issue that one of their apps 
slowed down by 30%.  Fixing that bug restored the previous performance.  The 
application was proprietary, so I don't even know what it did in general.  It 
was run over multiple nodes, so this was not a single-node performance 
comparison.

More recently, some customers with large memory nodes, and > 40 cores per node 
found kcopy was important to the performance of their most important app, a 
finite element code (I don't have a percentage figure).

kcopy works with Open MPI over PSM , so using knem instead of kcopy is not 
likely to speed up that configuration much (unless you get your PSM from OFED 
or a Linux distro, then that won't include kcopy; we weren't able to get kcopy 
accepted upstream).  Recent PSM (from OFED 3.5 say) can be built to use knem 
for kernel-assisted copies. kcopy also works with the other MPIs that 
support PSM.

Hope these anecdotes are relevant to Open MPI users considering knem.

-Tom Elken  





> 
> Unusually for us, we're putting in a second cluster with the same
> architecture, CPUs, memory and OS as the last one. I might be able to use
> this as a bigger stick to get some better feedback. If so, I'll pass it
> on.
> 
> > Darius Buntinas, Brice Goglin, et al. wrote an excellent paper about
> > exactly this set of issues; see http://runtime.bordeaux.inria.fr/knem/.
> ...
> 
> I'll definitely take a look - thanks again.
> 
> All the best,
> 
> Mark
> --
> -
> Mark Dixon   Email: m.c.di...@leeds.ac.uk
> HPC/Grid Systems Support Tel (int): 35429
> Information Systems Services Tel (ext): +44(0)113 343 5429
> University of Leeds, LS2 9JT, UK
> -
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] Support for CUDA and GPU-direct with OpenMPI 1.6.5 an 1.7.2

[Elken, Tom]

Hi Tim,


Well, in general and not on MIC I usually build the MPI stacks using the Intel 
compiler set. Have you ran into s/w that requires GCC instead of Intel 
compilers (beside Nvidia Cuda)? Did you try to use Intel compiler to produce 
MIC native code (the OpenMPI stack for that matter)?
[Tom]
Good idea Michael,  With the Intel Compiler, I would use the -mmic flag to 
build MIC code.

Tim wrote: "My first pass at doing a cross-compile with the GNU compilers 
failed to produce something with OFED support (not surprising)

export PATH=/usr/linux-k1om-4.7/bin:$PATH
./configure --build=x86_64-unknown-linux-gnu --host=x86_64-k1om-linux \
--disable-mpi-f77

checking if MCA component btl:openib can compile... no

Regarding a Gnu cross compiler, this worked for one of our engineers building 
for True Scale HCAs and PSM/infinipath.  But it may give useful tips for 
building for btl:openib as well:

3. How to configure/compile OpenMPI:
   a). untar the openmpi tarball.
   b). edit configure in top directory, add '-linfinipath' after 
'-lpsm_infinipath "
   but not necessary for messages, only for command lines.

   c). run the following script,
#!/bin/sh

./configure \
--host=x86_64-k1om-linux \
--enable-mpi-f77=no --enable-mpi-f90=no \
--with-psm=/.../psm-7.6 \
--prefix=/.../openmpi \
CC=x86_64-k1om-linux-gcc  CXX=x86_64-k1om-linux-g++ \
AR=x86_64-k1om-linux-ar RANLIB=x86_64-k1om-linux-ranlib

   d). run 'make' and 'make install'

OK, I see that they did not configure for mpi-f77 & mpif90, but perhaps this is 
still helpful, if the AR and RANLIB flags are important.
-Tom



regards
Michael

On Mon, Jul 8, 2013 at 4:30 PM, Tim Carlson 
mailto:tim.carl...@pnl.gov>> wrote:
On Mon, 8 Jul 2013, Elken, Tom wrote:

It isn't quite so easy.

Out of the box, there is no gcc on the Phi card. You can use the cross compiler 
on the host, but you don't get gcc on the Phi by default.

See this post http://software.intel.com/en-us/forums/topic/382057

I really think you would need to build and install gcc on the Phi first.

My first pass at doing a cross-compile with the GNU compilers failed to produce 
something with OFED support (not surprising)

export PATH=/usr/linux-k1om-4.7/bin:$PATH
./configure --build=x86_64-unknown-linux-gnu --host=x86_64-k1om-linux \
--disable-mpi-f77

checking if MCA component btl:openib can compile... no


Tim




Thanks Tom, that sounds good. I will give it a try as soon as our Phi host
here host gets installed.



I assume that all the prerequisite libs and bins on the Phi side are
available when we download the Phi s/w stack from Intel's site, right ?

[Tom]

Right.  When you install Intel's MPSS (Manycore Platform Software Stack),
including following the section on "OFED Support" in the readme file, you
should have all the prerequisite libs and bins.  Note that I have not built
Open MPI for Xeon Phi for your interconnect, but it seems to me that it
should work.



-Tom



Cheers

Michael







On Mon, Jul 8, 2013 at 12:10 PM, Elken, Tom 
mailto:tom.el...@intel.com>> wrote:

Do you guys have any plan to support Intel Phi in the future? That is,
running MPI code on the Phi cards or across the multicore and Phi, as Intel
MPI does?

[Tom]

Hi Michael,

Because a Xeon Phi card acts a lot like a Linux host with an x86
architecture, you can build your own Open MPI libraries to serve this
purpose.

Our team has used existing (an older 1.4.3 version of) Open MPI source to
build an Open MPI for running MPI code on Intel Xeon Phi cards over Intel's
(formerly QLogic's) True Scale InfiniBand fabric, and it works quite well.
We have not released a pre-built Open MPI as part of any Intel software
release.   But I think if you have a compiler for Xeon Phi (Intel Compiler
or GCC) and an interconnect for it, you should be able to build an Open MPI
that works on Xeon Phi.

Cheers,
Tom Elken

thanks...

Michael



On Sat, Jul 6, 2013 at 2:36 PM, Ralph Castain 
mailto:r...@open-mpi.org>> wrote:

Rolf will have to answer the question on level of support. The CUDA code is
not in the 1.6 series as it was developed after that series went "stable".
It is in the 1.7 series, although the level of support will likely be
incrementally increasing as that "feature" series continues to evolve.



On Jul 6, 2013, at 12:06 PM, Michael Thomadakis 
mailto:drmichaelt7...@gmail.com>>
wrote:

> Hello OpenMPI,
>
> I am wondering what level of support is there for CUDA and GPUdirect on
OpenMPI 1.6.5 and 1.7.2.
>
> I saw the ./configure --with-cuda=CUDA_DIR option in the FAQ. However, it
seems that with configure v1.6.5 it was ignored.
>
> Can you identify GPU memory and send messages from it directly without
copying to host memory first?
>
>
> Or in general, what level of CUDA support is there on 1.6.5 and 1.7.2 ? Do
you support SDK 5.0 and above?

Re: [OMPI users] Support for CUDA and GPU-direct with OpenMPI 1.6.5 an 1.7.2

[Elken, Tom]

Thanks Tom, that sounds good. I will give it a try as soon as our Phi host here 
host gets installed.

I assume that all the prerequisite libs and bins on the Phi side are available 
when we download the Phi s/w stack from Intel's site, right ?
[Tom]
Right.  When you install Intel's MPSS (Manycore Platform Software Stack), 
including following the section on "OFED Support" in the readme file, you 
should have all the prerequisite libs and bins.  Note that I have not built 
Open MPI for Xeon Phi for your interconnect, but it seems to me that it should 
work.

-Tom

Cheers
Michael



On Mon, Jul 8, 2013 at 12:10 PM, Elken, Tom 
mailto:tom.el...@intel.com>> wrote:
Do you guys have any plan to support Intel Phi in the future? That is, running 
MPI code on the Phi cards or across the multicore and Phi, as Intel MPI does?
[Tom]
Hi Michael,
Because a Xeon Phi card acts a lot like a Linux host with an x86 architecture, 
you can build your own Open MPI libraries to serve this purpose.

Our team has used existing (an older 1.4.3 version of) Open MPI source to build 
an Open MPI for running MPI code on Intel Xeon Phi cards over Intel's (formerly 
QLogic's) True Scale InfiniBand fabric, and it works quite well.  We have not 
released a pre-built Open MPI as part of any Intel software release.   But I 
think if you have a compiler for Xeon Phi (Intel Compiler or GCC) and an 
interconnect for it, you should be able to build an Open MPI that works on Xeon 
Phi.
Cheers,
Tom Elken
thanks...
Michael

On Sat, Jul 6, 2013 at 2:36 PM, Ralph Castain 
mailto:r...@open-mpi.org>> wrote:
Rolf will have to answer the question on level of support. The CUDA code is not 
in the 1.6 series as it was developed after that series went "stable". It is in 
the 1.7 series, although the level of support will likely be incrementally 
increasing as that "feature" series continues to evolve.


On Jul 6, 2013, at 12:06 PM, Michael Thomadakis 
mailto:drmichaelt7...@gmail.com>> wrote:

> Hello OpenMPI,
>
> I am wondering what level of support is there for CUDA and GPUdirect on 
> OpenMPI 1.6.5 and 1.7.2.
>
> I saw the ./configure --with-cuda=CUDA_DIR option in the FAQ. However, it 
> seems that with configure v1.6.5 it was ignored.
>
> Can you identify GPU memory and send messages from it directly without 
> copying to host memory first?
>
>
> Or in general, what level of CUDA support is there on 1.6.5 and 1.7.2 ? Do 
> you support SDK 5.0 and above?
>
> Cheers ...
> Michael
> ___
> users mailing list
> us...@open-mpi.org<mailto:us...@open-mpi.org>
> http://www.open-mpi.org/mailman/listinfo.cgi/users


___
users mailing list
us...@open-mpi.org<mailto:us...@open-mpi.org>
http://www.open-mpi.org/mailman/listinfo.cgi/users


___
users mailing list
us...@open-mpi.org<mailto:us...@open-mpi.org>
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] Support for CUDA and GPU-direct with OpenMPI 1.6.5 an 1.7.2

[Elken, Tom]

Do you guys have any plan to support Intel Phi in the future? That is, running 
MPI code on the Phi cards or across the multicore and Phi, as Intel MPI does?
[Tom]
Hi Michael,
Because a Xeon Phi card acts a lot like a Linux host with an x86 architecture, 
you can build your own Open MPI libraries to serve this purpose.

Our team has used existing (an older 1.4.3 version of) Open MPI source to build 
an Open MPI for running MPI code on Intel Xeon Phi cards over Intel's (formerly 
QLogic's) True Scale InfiniBand fabric, and it works quite well.  We have not 
released a pre-built Open MPI as part of any Intel software release.   But I 
think if you have a compiler for Xeon Phi (Intel Compiler or GCC) and an 
interconnect for it, you should be able to build an Open MPI that works on Xeon 
Phi.
Cheers,
Tom Elken
thanks...
Michael

On Sat, Jul 6, 2013 at 2:36 PM, Ralph Castain 
mailto:r...@open-mpi.org>> wrote:
Rolf will have to answer the question on level of support. The CUDA code is not 
in the 1.6 series as it was developed after that series went "stable". It is in 
the 1.7 series, although the level of support will likely be incrementally 
increasing as that "feature" series continues to evolve.


On Jul 6, 2013, at 12:06 PM, Michael Thomadakis 
mailto:drmichaelt7...@gmail.com>> wrote:

> Hello OpenMPI,
>
> I am wondering what level of support is there for CUDA and GPUdirect on 
> OpenMPI 1.6.5 and 1.7.2.
>
> I saw the ./configure --with-cuda=CUDA_DIR option in the FAQ. However, it 
> seems that with configure v1.6.5 it was ignored.
>
> Can you identify GPU memory and send messages from it directly without 
> copying to host memory first?
>
>
> Or in general, what level of CUDA support is there on 1.6.5 and 1.7.2 ? Do 
> you support SDK 5.0 and above?
>
> Cheers ...
> Michael
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users


___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] Intel compiled undefined reference errors (i.e. mpi_reduce_)

[Elken, Tom]

I saw a couple of issues noted below ...


Now I am still getting some undefined reference errors. Namely:
[root@odie 060911_rmc_eam_gr_vk_t4]# make
ifort -g -debug all -check all -implicitnone -warn all -o rmc_test ran2.o 
globals.o read_inputs.o model.o rmc_functions.o scattering_factors.o fem1.o 
gr.o eam.o rmc_v_1_0.o  -L /export/apps/openmpi_intel_20130618/lib -lmpi
[Tom]
>From your makefile below, this compile command is generated by your DEBUG 
>target.  Fixes to the FC_DEBUG command noted below in your makefile...

ifort: warning #10182: disabling optimization; runtime debug checks enabled
fem1.o: In function `fem':
/state/partition1/home/jjmaldonis/Jinwoo/2011/060911_rmc_eam_gr_vk_t4/fem1.f90:583:
 undefined reference to `mpi_reduce_'
/state/partition1/home/jjmaldonis/Jinwoo/2011/060911_rmc_eam_gr_vk_t4/fem1.f90:584:
 undefined reference to `mpi_reduce_'
fem1.o: In function `fem_update':
[Tom]  

The above undefined reference errors are my problem.

Below are my PATH and LD_LIBRARY_PATH variables, if they are of help.
echo $PATH
/usr/lib64/qt-3.3/bin:/state/partition1/apps/intel_20130618/composer_xe_2013.3.163/bin/intel64:/state/partition1/apps/intel_20130618/composer_xe_2013.3.163/mpirt/bin/intel64:/state/partition1/apps/intel_20130618/composer_xe_2013.3.163/bin/intel64:/state/partition1/apps/intel_20130618/composer_xe_2013.3.163/bin/intel64_mic:/state/partition1/apps/intel_20130618/composer_xe_2013.3.163/debugger/gui/intel64:/export/apps/intel_20130618/advisor_xe_2013/bin64:/export/apps/intel_20130618/inspector_xe_2013/bin64:/export/apps/intel_20130618/vtune_amplifier_xe_2013/bin64:/usr/local/bin:/bin:/usr/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/eclipse:/opt/ganglia/bin:/opt/ganglia/sbin:/usr/java/latest/bin:/opt/maven/bin:/export/apps/openmpi_intel_20130618/:/opt/pdsh/bin:/opt/rocks/bin:/opt/rocks/sbin:/opt/condor/bin:/opt/condor/sbin:/usr/local/samba/sbin:/opt/gridengine/bin/linux-x64:/export/apps/openmpi_intel_20130618/bin/:/usr/local/samba/sbin:/export/apps/openmpi_intel_20130618/bin/
[Tom]
You should try to get
/export/apps/openmpi_intel_20130618/bin
earlier in your path, ahead of this part:
/state/partition1/apps/intel_20130618/composer_xe_2013.3.163/mpirt/bin/intel64
which comes from the intel compilers 'compilervars.[c]sh' script and points to 
the Intel MPI run-time libraries (which you don't want, since you are using 
Open MPI).
Either that, or use a full path to your OpenMPI/Intel mpirun when you are ready 
to run the app., such as:
/export/apps/openmpi_intel_20130618/bin/mpirun  ./rmc_test

[Tom] 

Below I will paste my makefile, and attached are the compressed "./configure 
..." and "make all install" command logs. Please let me know anything else that 
you need.


#
# default makefile for ifort compiler with more or less
# appropriate options for debugging and high performance
#

# application name
APP = rmc_test

# list of source files
SRC = ran2.f90 globals.f90 read_inputs.f90 model.f90 rmc_functions.f90 
scattering_factors.f90 fem1.f90 gr.f90 eam.f90 rmc_v_1_0.f90

# list of object files
OBJ = ran2.o globals.o read_inputs.o model.o rmc_functions.o 
scattering_factors.o fem1.o gr.o eam.o rmc_v_1_0.o

# define libraries needed by the linker
LIBS = -L /export/apps/openmpi_intel_20130618/lib -lmpi
[Tom]
Make this blank, so you will depend on 'mpif90' to provide the linking 
information:
LIBS =


# -rpath 
/state/partition1/apps/intel_20130618/composer_xe_2013.3.163/compiler/lib/

# compiler options for debugging
FC_DEBUG = ifort -g -debug all -check all -implicitnone -warn all
[Tom] You should make this more like your FC_OPT definition, using mpif90 
rather than ifort, such as:
FC_DEBUG = mpif90 -g -debug all -check all -implicitnone -warn all

[Tom] Cheers, Tom

# compiler options for optmized running
#FC_OPT = ifort -O3 -xO -ipo -no-prec-div -static
#FC_OPT = mpif90 -O3 -ipo -static
FC_OPT = mpif90 -O3

# build rules

.SUFFIXES: .f90 .o
.f90.o:
${FC_DEBUG} -c $<

debug: ${OBJ}
${FC_DEBUG} -o ${APP} ${OBJ} ${LIBS}

opt: ${SRC}
${FC_OPT} -o ${APP} ${SRC} ${LIBS}

clean:
rm -f *.mod *.o ${APP}
ln -s /export/apps/openmpi_intel_20130618/lib/mpi.mod ./mpi.mod

Re: [OMPI users] OpenMPI installation

[Elken, Tom]

> As a guess: I suggest looking for a package named openmpi-devel, or something
> like that.
[Tom] 
Yes, you want  "-devel" in addition to the RPM you listed.  Going to the URL 
below, I see listed:

openmpi-1.5.4-1.el6.x86_64.rpm - Open Message Passing Interface
openmpi-devel-1.5.4-1.el6.x86_64.rpm - Development files for openmpi
openmpi-psm-1.5.3-3.el6.x86_64.rpm - Open Message Passing Interface using 
InfiniPath
openmpi-psm-devel-1.5.3-3.el6.x86_64.rpm - Development files for openmpi using 
InfiniPath

Also, If you have QLogic or Intel True Scale InfiniBand adapters, download the 
last two RPMs listed above to get an Open MPI built to use PSM (Performance 
Scaled Messaging) as a default for the best performance and stability on these 
adapters.

-Tom

> 
> 
> 
> On May 29, 2013, at 7:41 AM, alankrutha reddy
> 
>  wrote:
> 
> > Hi,
> >
> > I have downloaded openmpi from
> >
> > http://pkgs.org/centos-6-rhel-6/centos-rhel-x86_64/openmpi-1.5.4-
> 1.el6.x86_64.rpm.html
> >
> > And I have installed using rpm -i openmpi-1.5.4-1.el6.x86_64.rpm.
> >
> > mpicc & mpif77 are not generated. How to generate those binaries.
> >
> > Can anyone help me.
> >
> > Thank you
> >
> >
> >
> > --
> > Regards
> > Alankrutha
> > ___
> > users mailing list
> > us...@open-mpi.org
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
> 
> 
> --
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
> 
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] mpirun error

[Elken, Tom]

> It looks like your PATH is pointing to the Intel MPI mpirun,
> not to the Open MPI mpirun/mpiexec.
[Tom] 
Just to expand a little on this, it looks like your path is pointing to the 
Intel MPI run-time version (mpirt) that is included with the Intel Compiler and 
it's PATH/LD_LIBRARY_PATH is set up by compilervars.[c]sh.
A common way to set up OpenMPI variables, is by using OFED is mpi-selector.

The combination of the above two runs into this problem:
> >> /opt/intel/composer_xe_2013.1.117/mpirt/bin/intel64/mpirun:  
is in the PATH before Open MPI's mpirun.


> Check the OpenMPI FAQ:
> http://www.open-mpi.org/faq/?category=running#run-prereqs
[Tom] 
Good FAQ.  One solution it points to is the use of the --prefix option, or
"note that specifying the absolute pathname to mpirun is equivalent to using 
the --prefix argument. For example, the following is equivalent to the above 
command line that uses --prefix: 

shell$ /opt/openmpi-1.6.4/bin/mpirun -np 4 a.out"

Another method is to use  OpenMPI's mpivars.sh or set the  OpenMPI parts of 
your PATH/LD_LIBRARY_PATH in your shell's startup file AFTER you use the Intel 
Compiler's  compilervars.sh.

-Tom



> 
> 
> I hope this helps,
> Gus Correa
> 
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] QLogic HCA random crash after prolonged use

[Elken, Tom]

> > Intel has acquired the InfiniBand assets of QLogic
> > about a year ago.  These SDR HCAs are no longer supported, but should
> > still work.
[Tom] 
I guess the more important part of what I wrote is that " These SDR HCAs are no 
longer supported" :)

> 
> Do you mean they should work with the latest infinipath libraries
> (despite what it said or implied in the notes for last version I got
> from QLogic?) or possibly what's in RHEL?  I thought I'd actually tried
> and failed with later stuff, but may just have gone by the release notes.
[Tom] 
Some testing from an Intel group who had these QLE7140 HCAs revealed to me that 
they do _not_ work with our recent software stack such as IFS 7.1.1 (which 
includes OFED 1.5.4.1) .

They were able to get them to work with the QLogic OFED+ 6.0.2 stack.  That 
corresponds to OFED 1.5.2 -- that was the first OFED to include PSM.  
I am providing this info as a courtesy, but not making any guarantees that it 
will work.

> 
> > You can get the driver (ib_qib) and PSM library from OFED 1.5.4.1 or
> > the current release OFED 3.5.
> 
> I wonder if there's a version of the driver that's known to work in a
> current RHEL5 system with QLE7140.  We get frequent qib-related kernel
> panics from a vanilla RHEL5.9 kernel
[Tom] 
The older QLogic and OFED stacks mentioned above were not ported to nor tested 
with RHEL 5.9, which did not exist at the time.  Sorry.

 -- after running OK under test for
> a few weeks, and nothing relevant appearing to have changed to cause
> it...  (There's a trace on the redhat bugzilla with qib in the issue
> title, for what it's worth.)  I'm currently reverting to old stuff.
> 
> It's good if Infinipath-land is taking an interest in OMPI again, and
> that the libraries are now under a free licence.
[Tom] 
Thanks,
Tom

> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] QLogic HCA random crash after prolonged use

[Elken, Tom]

> I have seen it recommended to use psm instead of openib for QLogic cards.
[Tom] 
Yes.  PSM will perform better and be more stable when running OpenMPI than 
using verbs.  Intel has acquired the InfiniBand assets of QLogic about a year 
ago.  These SDR HCAs are no longer supported, but should still work.  You can 
get the driver (ib_qib) and PSM library from OFED 1.5.4.1 or the current 
release OFED 3.5.

With the current OFED 3.5 release there are included psm-release notes which 
start out this way (read down to the OpenMPI build instructions for PSM):
"
  Open Fabrics Enterprise Distribution (OFED)
PSM in OFED 3.5 Release Notes

  December 2012

==
1. Overview
==

The Performance Scaled Messaging (PSM) API is Intel's low-level user-level
communications interface for the Intel True Scale Fabric family of products.

The PSM libraries are included in the infinipath-psm-3.1-364.1140_open.src.rpm
source RPM and get built and installed as part of a default OFED
install process.

The primary way to use PSM is by compiling applications with an MPI that has
been built to use the PSM layer as its interface to QLogic HCAs.
PSM is the high-performance interface to the QLogic TrueScale HCAs.

Minimal instructions* for building two MPIs tested with OFED 
with PSM support are as follows:


Open MPI:

- Download a recent Open MPI tar ball from 
   http://www.open-mpi.org/software/ompi/v1.6/ .
  Versions 1.6.1 & 1.6.3 have been tested with PSM from this OFED release.
- Untar the file and cd to the Open MPI directory.  Then
  ./configure --with-psm --prefix=
  make
  make install
"

-Tom Elken

> Could this be part of the problem? After a lot of experimentation I am at a
> complete loss as to how to get psm up and running. If possible, could someone
> also help me understand which out of this list (ibibverbs, libipathverbs, 
> libmthca,
> librdmacm, ib_mad, ib_umad, rdma_ucm, ib_uverbs, ib_qib) is the actual driver
> for my card and is there any way to configure the driver? This blog
> posthttp://swik.net/Debian/Planet+Debian/Julien+Blache%3A+QLogic+QLE73xx
> +InfiniBand+adapters,+QDR,+ib_qib,+OFED+1.5.2+and+Debian+Squeeze/e56if
> 
> seems to suggest that I will need to download the complete QLogic OFED stack
> and configure the driver which I've tried to do and failed on many
> counts.
> 
> I would be very grateful for any advice at this stage.
> 
> Best regards,
> 
> Vanja
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] mpirun error

[Elken, Tom]

> The Intel Fortran 2013 compiler comes with support for Intel's MPI runtime and
> you are getting that instead of OpenMPI.   You need to fix your path for all 
> the
> shells you use.
[Tom] 
Agree with Michael, but thought I would note something additional.
If you are using OFED's mpi-selector to select Open MPI, it will set up the 
path to Open MPI before a startup script like  .bashrc gets processed.
So if you source the Intel Compiler's compilervars.sh, you will get Intel's 
mpirt in your path before the OpenMPI's bin directory.

One workaround is to source the following _after_ you source the Intel 
Compiler's compilervars.sh in your start-up scripts:
. /var/mpi-selector/data/openmpi_...sh

-Tom

> 
> On Apr 1, 2013, at 5:12 AM, Pradeep Jha wrote:
> 
> > /opt/intel/composer_xe_2013.1.117/mpirt/bin/intel64/mpirun: line 96:
> /opt/intel/composer_xe_2013.1.117/mpirt/bin/intel64/mpivars.sh: No such file
> or directory
> 
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] configuration openMPI problem

[Elken, Tom]

Now I would like to test it with a simple hello project.  Ralph Castain suggest 
me the following web site:
https://wiki.mst.edu/nic/examples/openmpi-intel-fortran90-example

This is the results of my simulation:
 Hello World! I am0  of1

How ever I have a quad core processor, I belive (I run a  cat /proc/cpuinfo)
[Tom]
What was your mpirun command?
Did it have a '-np 4'  in it to tell mpirun that you want 4 processes to be 
spawned?

Don't be afraid to read the FAQ on running MPI programs:
http://www.open-mpi.org/faq/?category=running

-Tom

Thanks a lot

Diego



On 26 November 2012 13:49, Gus Correa 
mailto:g...@ldeo.columbia.edu>> wrote:
Hi Diego


> deal all, dear Gustavo,
>
> This is my bash.bashrc in ubuntu 12.04:
>
> ##
> /PATH="/opt/intel/bin/compilervars.sh intel64$PATH"/
> /source /opt/intel/bin/compilervars.sh intel64/
> /source /opt/intel/mkl/bin/mklvars.sh intel64 mod lp64/
> /export PATH/
> ##

This is not an OpenMPI problem, but about Linux environment setup.

Anyway, my guess is that all you
need in your .bashrc are these two lines (2 and 3):


source /opt/intel/bin/compilervars.sh intel64
source /opt/intel/mkl/bin/mklvars.sh intel64 mod lp64
The first line is probably messing up your PATH, the fourth line
may be just redundant with the the Intel compilervars.sh script.
Try commenting out lines 1 and 4 (with a leading # character),
and leave only lines 2 and 3.

(Note, no '/' in the beginning or at the end of the lines, not sure
if the '/'s are part of your .bashrc or just part of your email.)

After you make the change, then login again, or open
a new terminal/shell window and try these commands:

which icc
which icpc
which ifort
printenv

to make sure your environment is pointing
to the correct Intel compilers.

I hope this helps,
Gus Correa


On 11/26/2012 09:42 AM, Diego Avesani wrote:
I think that is correct according to your mail, so I do not think that
this is this problem.
I check the config.log file. It says:
  checking for gcc
##
/configure:5133: result: icc/
/configure:5362: checking for C compiler version/
/configure:5371: icc --version >&5/
/./configure: line 5373: icc: command not found/
/configure:5382: $? = 127/
/configure:5371: icc -v >&5/

##
When I write the simple project inside the config.log file in new file .c
##
  int
  main ()
  {
;
return 0;
}
##

it works when I compile it with icc

Do I probably need to change also the .csh?
My current intel version is 13.0, When I compile it they told me to set
/compilervars.sh /moreover check iccvars.sh,  ifortvars.sh and
/compilervars.sh, /they are the same.


I do not know what to do, could I compile open mpi with gcc,
gcpc,gnufort and then use it with intel fortran?
do you think that is a OpenMpi problem? Has someone compile it with
intel linux icc? which distro have you used?

Thank all

Diego




On 25 November 2012 22:21, Gustavo Correa 
mailto:g...@ldeo.columbia.edu>
>> wrote:

urce compilervars.sh intel64



___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] Communication Round-trip time

[Elken, Tom]

I'll agree with Jeff that what you propose sounds right for avg. round-trip 
time.

Just thought I'd mention that when people talk about the ping-pong latency or 
MPI latency benchmarks, they are usually referring to 1/2 the round-trip time.  
So you compute everything the same as you did, and then divide by 2.  

But if your professor clearly said round-trip, I'd agree that you need to get a 
clearer definition of what he's looking for.  Maybe it's units.  If he asked 
for round-trip time in microseconds, you have to adjust your formula.

-Tom

> -Original Message-
> From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] On
> Behalf Of Jeff Squyres
> Sent: Tuesday, October 09, 2012 2:58 AM
> To: Open MPI Users
> Subject: Re: [OMPI users] Communication Round-trip time
> 
> In general, what you said is "right"... for some definition of "right".  :-)
> 
> Usually, benchmarking programs start a timer, do the round trip sends N times,
> stop the timer, and then divide the total time by N (to get a smoother 
> definition
> of "average").
> 
> But keep in mind that there are many, many factors involved here.  For 
> example,
> it is also common to do "warmup" round trip communications -- do, say, 100
> round-trip communications before you start the timer.  This allows MPI to
> establish connections and do any other startup processing that isn't a factor
> during steady-state communications.
> 
> More specifically, the definitions of "right" and "wrong" very much depend on
> what is in your professor's head.  Every benchmark is different; there are 
> many
> similar-but-slightly-different benchmark definitions out there.  You'll need 
> to
> figure out exactly what he is looking for; sorry.
> 
> 
> 
> On Oct 8, 2012, at 10:59 PM, huydanlin wrote:
> 
> >I'm currently doing MPI project in NUS University. My professor require 
> > me
> make a monitoring tool to check the connection among processes in Cluster.
> And also measure communication round-trip time.
> >My solution is :
> > 1. At the source process : start MPI_Send
> > 2. At the destination process : start MPI_Recv to receive the message from
> source
> > 3. At the destination process : start MPI_Send to send immediately the same
> message to source
> > 4. At the source process : start MPI_Recv to receive the message from
> destination
> >I start timer( before (1) - t1=MPI_Wtime & after (4) - t2=MPI_Wtime )
> >then t2 - t1 is time for communication. I also do (1) to (4) for N 
> > times. then i
> calculate the avg round-trip time by (t2-t1)/N.
> >So is it right? Because he tells it 's wrong. And what exactly is the
> communication round-trip time ?
> >   Regards and hope to see your reply soon.
> > ___
> > users mailing list
> > us...@open-mpi.org
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
> 
> 
> --
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
> 
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] Regarding hyperthreding

[Elken, Tom]

Hi Seshendra,

If you have implemented hyperthreading by Enabling it in the BIOS, then when 
you look at
cat /proc/cpuinfo output, you will see 2x the # of CPUs than the number of 
cores on your system.
 If that number of CPUs showing on a node = H, and the number of nodes in your 
cluster with this same configuration is M,
Then you can run your program on up to N = H x M  MPI ranks,
and your Open MPI  mpirun command can use the
'-np N -npernode H'   options to indicate this.

Note that with some applications, you may see very little benefit to 
hyperthreading.  In any case, you should also run on "-np N/2" ranks and add 
the "-npernode H/2" setting to not use more ranks per node than you have cores 
per node, and compare performance with N ranks (with H per node).

-Tom


From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] On Behalf 
Of seshendra seshu
Sent: Monday, August 13, 2012 9:27 AM
To: Open MPI Users
Subject: [OMPI users] Regarding hyperthreding

Hi,
I have written an code mpi and an iam trying to implement hyperthreading  
environment. So could any one tell me how to implement and run the program in 
open mpi.

Kindly help me.

Thanking you,



--
 WITH REGARDS
M.L.N.Seshendra

Re: [OMPI users] openmpi tar.gz for 1.6.1 or 1.6.2

[Elken, Tom]

gb rdtscp lm 
3dnowext 3dnow constant_tsc rep_good nonstop_tsc extd_apicid pni monitor cx16 
popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 
3dnowprefetch osvw ibs skinit wdt npt lbrv svm_lock nrip_save
bogomips : 4623.16
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 48 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate

processor : 5
vendor_id : AuthenticAMD
cpu family : 16
model : 4
model name : Quad-Core AMD Opteron(tm) Processor 2376
stepping : 2
cpu MHz : 2311.694
cache size : 512 KB
physical id : 1
siblings : 4
core id : 1
cpu cores : 4
apicid : 5
initial apicid : 5
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 
clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 
3dnowext 3dnow constant_tsc rep_good nonstop_tsc extd_apicid pni monitor cx16 
popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 
3dnowprefetch osvw ibs skinit wdt npt lbrv svm_lock nrip_save
bogomips : 4623.16
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 48 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate

processor : 6
vendor_id : AuthenticAMD
cpu family : 16
model : 4
model name : Quad-Core AMD Opteron(tm) Processor 2376
stepping : 2
cpu MHz : 2311.694
cache size : 512 KB
physical id : 1
siblings : 4
core id : 2
cpu cores : 4
apicid : 6
initial apicid : 6
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 
clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 
3dnowext 3dnow constant_tsc rep_good nonstop_tsc extd_apicid pni monitor cx16 
popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 
3dnowprefetch osvw ibs skinit wdt npt lbrv svm_lock nrip_save
bogomips : 4623.17
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 48 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate

processor : 7
vendor_id : AuthenticAMD
cpu family : 16
model : 4
model name : Quad-Core AMD Opteron(tm) Processor 2376
stepping : 2
cpu MHz : 2311.694
cache size : 512 KB
physical id : 1
siblings : 4
core id : 3
cpu cores : 4
apicid : 7
initial apicid : 7
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 
clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 
3dnowext 3dnow constant_tsc rep_good nonstop_tsc extd_apicid pni monitor cx16 
popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 
3dnowprefetch osvw ibs skinit wdt npt lbrv svm_lock nrip_save
bogomips : 4623.18
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 48 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate

On Jul 16, 2012, at 4:09 PM, Elken, Tom wrote:


Anne,

output from "cat /proc/cpuinfo" on your node "hostname"  may help those trying 
to answer.

-Tom


-Original Message-
From: users-boun...@open-mpi.org<mailto:users-boun...@open-mpi.org> 
[mailto:users-boun...@open-mpi.org] On
Behalf Of Ralph Castain
Sent: Monday, July 16, 2012 2:47 PM
To: Open MPI Users
Subject: Re: [OMPI users] openmpi tar.gz for 1.6.1 or 1.6.2

I gather there are two sockets on this node? So the second cmd line is 
equivalent
to leaving "num-sockets" off of the cmd line?

I haven't tried what you are doing, so it is quite possible this is a bug.


On Jul 16, 2012, at 1:49 PM, Anne M. Hammond wrote:

Thanks!

Built the latest snapshot.  Still getting an error when trying to run
on only one socket (see below):  Is there a workaround?

[hammond@node65 bin]$ ./mpirun -np 4 --num-sockets 1 --npersocket 4
hostname
--
 An invalid physical processor ID was returned when attempting to
bind an MPI process to a unique processor.

This usually means that you requested binding to more processors than
exist (e.g., trying to bind N MPI processes to M processors, where N >
M).  Double check that you have enough unique processors for all the
MPI processes that you are launching on this host.

You job will now abort.
--

--
 mpirun was unable to start the specified application as it
encountered an error:

Error name: Fatal
Node: node65.cl.corp.com<http://node65.cl.corp.com/>

when attempting to start process rank 0.
--

4 total processes failed to start


[hammond@node65 bin]$ ./mpirun -np 4 --num-sockets 2 --npersocket 4
hostname node65.cl.corp.com<http://node65.cl.c

Re: [OMPI users] openmpi tar.gz for 1.6.1 or 1.6.2

[Elken, Tom]

Anne,

output from "cat /proc/cpuinfo" on your node "hostname"  may help those trying 
to answer.

-Tom

> -Original Message-
> From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] On
> Behalf Of Ralph Castain
> Sent: Monday, July 16, 2012 2:47 PM
> To: Open MPI Users
> Subject: Re: [OMPI users] openmpi tar.gz for 1.6.1 or 1.6.2
> 
> I gather there are two sockets on this node? So the second cmd line is 
> equivalent
> to leaving "num-sockets" off of the cmd line?
> 
> I haven't tried what you are doing, so it is quite possible this is a bug.
> 
> 
> On Jul 16, 2012, at 1:49 PM, Anne M. Hammond wrote:
> 
> > Thanks!
> >
> > Built the latest snapshot.  Still getting an error when trying to run
> > on only one socket (see below):  Is there a workaround?
> >
> > [hammond@node65 bin]$ ./mpirun -np 4 --num-sockets 1 --npersocket 4
> > hostname
> > --
> >  An invalid physical processor ID was returned when attempting to
> > bind an MPI process to a unique processor.
> >
> > This usually means that you requested binding to more processors than
> > exist (e.g., trying to bind N MPI processes to M processors, where N >
> > M).  Double check that you have enough unique processors for all the
> > MPI processes that you are launching on this host.
> >
> > You job will now abort.
> > --
> > 
> > --
> >  mpirun was unable to start the specified application as it
> > encountered an error:
> >
> > Error name: Fatal
> > Node: node65.cl.corp.com
> >
> > when attempting to start process rank 0.
> > --
> > 
> > 4 total processes failed to start
> >
> >
> > [hammond@node65 bin]$ ./mpirun -np 4 --num-sockets 2 --npersocket 4
> > hostname node65.cl.corp.com node65.cl.corp.com node65.cl.corp.com
> > node65.cl.corp.com
> > [hammond@node65 bin]$
> >
> >
> >
> >
> > On Jul 16, 2012, at 12:56 PM, Ralph Castain wrote:
> >
> >> Jeff is at the MPI Forum this week, so his answers will be delayed. Last I
> heard, it was close, but no specific date has been set.
> >>
> >>
> >> On Jul 16, 2012, at 11:49 AM, Michael E. Thomadakis wrote:
> >>
> >>> When is the expected date for the official 1.6.1 (or 1.6.2 ?) to be 
> >>> available ?
> >>>
> >>> mike
> >>>
> >>> On 07/16/2012 01:44 PM, Ralph Castain wrote:
>  You can get it here:
> 
>  http://www.open-mpi.org/nightly/v1.6/
> 
>  On Jul 16, 2012, at 10:22 AM, Anne M. Hammond wrote:
> 
> > Hi,
> >
> > For benchmarking, we would like to use openmpi with
> > --num-sockets 1
> >
> > This fails in 1.6, but Bug Report #3119 indicates it is changed in
> > 1.6.1.
> >
> > Is 1.6.1 or 1.6.2 available in tar.gz form?
> >
> > Thanks!
> > Anne
> >
> >
> >
> > ___
> > users mailing list
> > us...@open-mpi.org
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
>  ___
>  users mailing list
>  us...@open-mpi.org
>  http://www.open-mpi.org/mailman/listinfo.cgi/users
> >>>
> >>>
> >>> ___
> >>> users mailing list
> >>> us...@open-mpi.org
> >>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> >>
> >>
> >> ___
> >> users mailing list
> >> us...@open-mpi.org
> >> http://www.open-mpi.org/mailman/listinfo.cgi/users
> >>
> >
> > Anne M. Hammond - Systems / Network Administration - Tech-X Corp
> >  hammond_at_txcorp.com 720-974-1840
> >
> >
> >
> >
> >
> > ___
> > users mailing list
> > us...@open-mpi.org
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
> 
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] Performance scaled messaging and random crashes

[Elken, Tom]

Hi Sebastien,

The Infinipath / PSM software that was developed by PathScale/QLogic is now 
part of Intel.

I'll advise you off-list about how to contact our customer support so we can 
gather information about your software installation and work to resolve your 
issue.

The 20 microseconds latency you are getting with Open MPI / PSM is still way 
too high, so there may be some network issue which needs to be solved first.

-Tom 

> -Original Message-
> From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] On
> Behalf Of Sébastien Boisvert
> Sent: Friday, June 29, 2012 10:56 AM
> To: Open MPI Users
> Subject: Re: [OMPI users] Performance scaled messaging and random crashes
> 
> Hi,
> 
> Thank you for the direction.
> 
> I installed Open-MPI 1.6 and the program is also crashing with 1.6.
> 
> Could there be a bug in my code ?
> 
> I don't see how disabling PSM would make the bug go away if the bug is in my
> code.
> 
> 
> Open-MPI configure command
> 
> module load gcc/4.5.3
> 
> ./configure \
> --prefix=/sb/project/nne-790-ab/software/Open-MPI/1.6/Build \ --with-openib
> \ --with-psm \ --with-tm=/software/tools/torque/ \
> | tee configure.log
> 
> 
> Versions
> 
> module load gcc/4.5.3
> module load /sb/project/nne-790-ab/software/modulefiles/mpi/Open-MPI/1.6
> module load /sb/project/nne-790-ab/software/modulefiles/apps/ray/2.0.0
> 
> 
> 
> PSM parameters
> 
> guillimin> ompi_info -a|grep psm
>   MCA mtl: psm (MCA v2.0, API v2.0, Component v1.6)
>   MCA mtl: parameter "mtl_psm_connect_timeout" (current
> value: <180>, data source: default value)
>   MCA mtl: parameter "mtl_psm_debug" (current value:
> <1>, data source: default value)
>   MCA mtl: parameter "mtl_psm_ib_unit" (current value:
> <-1>, data source: default value)
>   MCA mtl: parameter "mtl_psm_ib_port" (current value:
> <0>, data source: default value)
>   MCA mtl: parameter "mtl_psm_ib_service_level" (current
> value: <0>, data source: default value)
>   MCA mtl: parameter "mtl_psm_ib_pkey" (current value:
> <32767>, data source: default value)
>   MCA mtl: parameter "mtl_psm_ib_service_id" (current
> value: <0x10001175>, data source: default value)
>   MCA mtl: parameter "mtl_psm_path_query" (current
> value: , data source: default value)
>   MCA mtl: parameter "mtl_psm_priority" (current value:
> <0>, data source: default value)
> 
> 
> Thank you.
> 
> 
> Sébastien Boisvert
> 
> 
> Jeff Squyres a écrit :
> > The Open MPI 1.4 series is now deprecated.  Can you upgrade to Open MPI
> 1.6?
> >
> >
> > On Jun 29, 2012, at 9:02 AM, Sébastien Boisvert wrote:
> >
> >> I am using Open-MPI 1.4.3 compiled with gcc 4.5.3.
> >>
> >> The library:
> >>
> >> /usr/lib64/libpsm_infinipath.so.1.14: ELF 64-bit LSB shared object,
> >> AMD x86-64, version 1 (SYSV), not stripped
> >>
> >>
> >>
> >> Jeff Squyres a écrit :
> >>> Yes, PSM is the native transport for InfiniPath.  It is faster than the
> InfiniBand verbs support on the same hardware.
> >>>
> >>> What version of Open MPI are you using?
> >>>
> >>>
> >>> On Jun 28, 2012, at 10:03 PM, Sébastien Boisvert wrote:
> >>>
>  Hello,
> 
>  I am getting random crashes (segmentation faults) on a super
>  computer (guillimin) using 3 nodes with 12 cores per node. The same
>  program (Ray) runs without any problem on the other super computers I
> use.
> 
>  The interconnect is "InfiniBand: QLogic Corp. InfiniPath QME7342
>  QDR HCA" and the messages transit using "performance scaled
>  messaging" (PSM) which I think is some sort of replacement to Infiniband
> verbs although I am not sure.
> 
>  Adding '--mca mtl ^psm' to the Open-MPI mpiexec program options
>  solves the problem, but increases the latency from 20 microseconds to 55
> microseconds.
> 
>  There seems to be some sort of message corruption during the
>  transit, but I can not rule out other explanations.
> 
> 
>  I have no idea what is going on and why disabling PSM solves the problem.
> 
> 
>  Versions
> 
>  module load gcc/4.5.3
>  module load openmpi/1.4.3-gcc
> 
> 
>  Command that randomly crashes
> 
>  mpiexec -n 36 -output-filename MiSeq-bug-2012-06-28.1 \ Ray -k 31 \
>  -o MiSeq-bug-2012-06-28.1 \ -p \
>  data-for-system-tests/ecoli-MiSeq/MiSeq_Ecoli_MG1655_110527_R1.fast
>  q \
>  data-for-system-tests/ecoli-MiSeq/MiSeq_Ecoli_MG1655_110527_R2.fast
>  q
> 
> 
>  Command that completes successfully
> 
>  mpiexec -n 36 -output-filename  psm-bug-2012-06-26-hotfix.1 \ --mca
>  mtl ^psm \ Ray -k 31 \ -o psm-bug-2012-06-26-hotfix.1 \ -p \
>  data-for-system-tests/ecoli-MiSeq/MiSeq_Ecoli_MG1655_110527_R1.fast
>  q \
>  data-for-system-tests/ecoli-MiSeq/MiSeq_Ecoli_MG1655_110527_R2.fast
>  q
>