Re: [OMPI users] OpenMPI vs Intel Efficiency question
On 7/12/2011 7:45 PM, Mohan, Ashwin wrote: Hi, I noticed that the exact same code took 50% more time to run on OpenMPI than Intel. I use the following syntax to compile and run: Intel MPI Compiler: (Redhat Fedora Core release 3 (Heidelberg), Kernel version: Linux 2.6.9-1.667smp x86_64** mpiicpc -o .cpp -lmpi OpenMPI 1.4.3: (Centos 5.5 w/ python 2.4.3, Kernel version: Linux 2.6.18-194.el5 x86_64)** mpiCC .cpp -o **Other hardware specs** processor : 0 vendor_id : GenuineIntel cpu family : 15 model : 3 model name : Intel(R) Xeon(TM) CPU 3.60GHz stepping: 4 cpu MHz : 3591.062 cache size : 1024 KB physical id : 0 siblings: 2 core id : 0 cpu cores : 1 apicid : 0 fpu : yes fpu_exception : yes cpuid level : 5 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall lmconstant_tsc pni monitor ds_cpl est tm2 cid xtpr bogomips: 7182.12 clflush size: 64 cache_alignment : 128 address sizes : 36 bits physical, 48 bits virtual power management: Can the issue of efficiency be deciphered from the above info? Does the compiler flags have an effect on the efficiency of the simulation. If so, what flags maybe useful to check to be included for Open MPI. The default options for icpc are roughly equivalent to the quite aggressive choice g++ -fno-strict-aliasing -ffast-math -fnocx-limited-range -O3 -funroll-loops --param max-unroll-times=2 while you apparently used default -O0 for your mpiCC (if it is g++), neither of which is a very good initial choice for performance analysis. So, if you want a sane comparison but aren't willing to study the compiler manuals, you might use (if your source code doesn't violate the aliasing rules) mpiicpc -prec-div -prec-sqrt -ansi-alias and at least (if your linux compiler is g++) mpiCC -O2 possibly with some of the other options I mentioned earlier. If you have as ancient a g++ as your indication of FC3 implies, it really isn't fair to compare it with a currently supported compiler. Then, Intel MPI, by default, would avoid using HyperThreading, even though you have it enabled on your CPU, so, I suppose, if you are running on a single core, it will be rotating among your 4 MPI processes 1 at a time. The early Intel HyperThread CPUs typically took 15% longer to run MPI jobs when running 2 processes per core. Will including MPICH2 increase efficiency in running simulations using OpenMPI? You have to choose a single MPI. Having MPICH2 installed shouldn't affect performance of OpenMPI or Intel MPI, except to break your installation if you don't keep things sorted out. OpenMPI and Intel MPI normally perform very close, if using equivalent settings, when working within the environments for which both are suited. -- Tim Prince
Re: [OMPI users] a question about network connection of open-mpi
I believe we responded to this before...you might check your spam or inbox. On Jul 12, 2011, at 7:39 PM, zhuangchao wrote: > hello all : > > >I run the following command : > > /data1/cluster/openmpi/bin/mpirun -d -machinefile /tmp/nodes.10515.txt > -np 3 /data1/cluster/mpiblast-pio-1.6/bin/mpiblast -p blastn -i > /data1/cluster/sequences/seq_4.txt -d Baculo_Nucleotide -o > /data1/cluster/blast.out/blast.out.10515 -g T -m 0 -F F > > Then I get the following error from openmpi: > > [node7][[3812,1],2][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] > connect() to 192.168.0.5 failed: No route to host (113) > [node7][[3812,1],2][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] > connect() to 159.226.126.15 failed: No route to host (113) > > The machinefile is defined as following : > > node1 > node5 > node7 > > 192.168.0.5 is not the ip of hosts in the machinefile . > 159.226.126.15 is the another ip of node1 . But hostname node1 > > corresponds to 11.11.11.1 in the /etc/hosts . > > why do I get the error ? Can you help me ? > >Thank you ! > > > > ___ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
Re: [OMPI users] OpenMPI vs Intel Efficiency question
On 7/12/2011 4:45 PM, Mohan, Ashwin wrote:
I noticed that the exact same code took 50% more time to run on OpenMPI
than Intel.
It would be good to know if that extra time is spent inside MPI calls or
not. There is a discussion of how you might do this here:
http://www.open-mpi.org/faq/?category=perftools You should probably
start here and narrow down your investigation.
If the difference is the time spent inside MPI calls... um, that would
be interesting.
If the difference is time spent outside MPI calls, how you are compiling
(which serial compiler is being used, which optimization flags, etc.)
could be the issue. Or possibly how processes are placed on a node
("paffinity" or "binding" issues).
Does the compiler flags have an effect on the efficiency of the
simulation.
Sure. Ideally, most of the time is spent in parallel computation and
very little in MPI. For performance in such an "ideal" case, any
"decent" MPI implementation (OMPI and Intel hopefully among them) should
do just fine.
Will including MPICH2 increase efficiency in running simulations using
OpenMPI?
MPICH2 and OMPI are MPI implementations. You choose one or the other
(or other options... e.g., Intel).
[OMPI users] a question about network connection of open-mpi
hello all : I run the following command : /data1/cluster/openmpi/bin/mpirun -d -machinefile /tmp/nodes.10515.txt -np 3 /data1/cluster/mpiblast-pio-1.6/bin/mpiblast -p blastn -i /data1/cluster/sequences/seq_4.txt -d Baculo_Nucleotide -o /data1/cluster/blast.out/blast.out.10515 -g T -m 0 -F F Then I get the following error from openmpi: [node7][[3812,1],2][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.0.5 failed: No route to host (113) [node7][[3812,1],2][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 159.226.126.15 failed: No route to host (113) The machinefile is defined as following : node1 node5 node7 192.168.0.5 is not the ip of hosts in the machinefile . 159.226.126.15 is the another ip of node1 . But hostname node1 corresponds to 11.11.11.1 in the /etc/hosts . why do I get the error ? Can you help me ? Thank you !
[OMPI users] OpenMPI vs Intel Efficiency question
Hi, I noticed that the exact same code took 50% more time to run on OpenMPI than Intel. I use the following syntax to compile and run: Intel MPI Compiler: (Redhat Fedora Core release 3 (Heidelberg), Kernel version: Linux 2.6.9-1.667smp x86_64** mpiicpc -o .cpp -lmpi OpenMPI 1.4.3: (Centos 5.5 w/ python 2.4.3, Kernel version: Linux 2.6.18-194.el5 x86_64)** mpiCC .cpp -o **Other hardware specs** processor : 0 vendor_id : GenuineIntel cpu family : 15 model : 3 model name : Intel(R) Xeon(TM) CPU 3.60GHz stepping: 4 cpu MHz : 3591.062 cache size : 1024 KB physical id : 0 siblings: 2 core id : 0 cpu cores : 1 apicid : 0 fpu : yes fpu_exception : yes cpuid level : 5 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall lmconstant_tsc pni monitor ds_cpl est tm2 cid xtpr bogomips: 7182.12 clflush size: 64 cache_alignment : 128 address sizes : 36 bits physical, 48 bits virtual power management: Can the issue of efficiency be deciphered from the above info? Does the compiler flags have an effect on the efficiency of the simulation. If so, what flags maybe useful to check to be included for Open MPI. Will including MPICH2 increase efficiency in running simulations using OpenMPI? Thanks, Ashwin.
[OMPI users] How to use a wrapper for ssh?
Hi OpenMPI folks,
Using the version 1.4.3 of OpenMPI, I wanna to wrap the 'ssh' calls,
which are made from the OpenMPIs 'mpiexec'. For this purpose, at least
two ways seem to be possible for me:
1. let the wrapper have the name 'ssh' and paste the path where it is
into the PATH envvar *before* the path to real ssh
Q1: Would this work?
2. use MCA parameters described in
http://www.open-mpi.org/faq/?category=rsh#rsh-not-ssh
to bend the call to my wrapper, e.g.
export OMPI_MCA_plm_rsh_agent=WrapPer
export OMPI_MCA_orte_rsh_agent=WrapPer
the oddly thing is, that the OMPI_MCA_orte_rsh_agent envvar seem not to
have any effect, whereas OMPI_MCA_plm_rsh_agent works.
Why I believe so?
Because "strace -f mpiexec ..." says still trying for opening 'ssh' if
OMPI_MCA_orte_rsh_agent is set, and correctly trying to open the
'WrapPer' iff OMPI_MCA_plm_rsh_agent is set.
Q2: Is the supposed non-functionality of OMPI_MCA_orte_rsh_agent a bug,
or do I have just misunderstood something?
Best wishes,
Paul
P.S. reproducing: just set the envvars and do 'strace -f mpiexec ...'
example:
export OMPI_MCA_plm_rsh_agent=WrapPer
---> look'n for 'WrapPer';
stat64("/opt/lsf/8.0/linux2.6-glibc2.3-x86_64/bin/WrapPer", 0x8324)
= -1 ENOENT (No such file or directory)
export OMPI_MCA_orte_rsh_agent=WrapPer
(do not forget to unset OMPI_MCA_plm_rsh_agent :o)
---> still looking for 'ssh'
stat64("/opt/lsf/8.0/linux2.6-glibc2.3-x86_64/bin/ssh", 0x8324) = -1
ENOENT (No such file or directory)
===> OMPI_MCA_orte_rsh_agent does not work?!
--
Dipl.-Inform. Paul Kapinos - High Performance Computing,
RWTH Aachen University, Center for Computing and Communication
Seffenter Weg 23, D 52074 Aachen (Germany)
Tel: +49 241/80-24915
smime.p7s
Description: S/MIME Cryptographic Signature
Re: [OMPI users] tcp communication problems with 1.4.3 and 1.4.4 rc2 on FreeBSD
On Tue, Jul 12, 2011 at 11:03:42AM -0700, Steve Kargl wrote: > On Tue, Jul 12, 2011 at 10:37:14AM -0700, Steve Kargl wrote: > > On Fri, Jul 08, 2011 at 07:03:13PM -0400, Jeff Squyres wrote: > > > Sorry -- I got distracted all afternoon... > > > > > > In addition to what Ralph said (i.e., I'm not sure if the CIDR > > > notation stuff made it over to the v1.5 branch or not, but it > > > is available from the nightly SVN trunk tarballs: > > > http://www.open-mpi.org/nightly/trunk/), here's a few points > > > from other mails in this thread... > > > > > > > trunk does not appear to be an option. :-( > > > > % svn co http://svn.open-mpi.org/svn/ompi/trunk ompi > > % cd ompi > > % ./autogen.pl > > % ./configure --enable-mpirun-prefix-by-default --prefix=/usr/local/ompi \ > >--disable-shared --enable-static > > > > (many lines removed) > > checking prefix for function in .type... @ > > checking if .size is needed... yes > > checking if .align directive takes logarithmic value... no > > configure: error: No atomic primitives available for > > amd64-unknown-freebsd9.0 > > It seems the configure script does not recognize amd64. If I add > --build='x86_64-*-freebsd' to the configure line, then everything > appears to work. > > I'll report back after I've had a chance to work with ompi built > from trunk. > There's good news and some bad news. I got trunk to build and install. I compile the netpipe code with % /usr/local/ompi/bin/mpicc -o z -O GetOpt.c netmpi.c Bad news: I can then run % /usr/local/ompi/bin/mpiexec -machinefile mf --mca btl self,tcp \ --mca btl_base_verbose 30 ./z with mf containing node11 slots=1 (node11 contains a single bge0=168.192.0.11) node16 slots=1 (node16 contains a single bge0=168.192.0.16) or node11 slots=2 (communication on memory bus) However, if mf contains node10 slots=1 (node10 contains bge0=10.208.xx and bge1=192.168.0.10) node16 slots=1 (node16 contains a single bge0=192.168.0.16) I see the same problem where node10 cannot communicate with node16. Good News: Adding 'btl_tcp_if_include=192.168.0.0/16' to my ~/.openmpi/mca-params.conf file seems to cure the communication problem. Thanks for the help. If I run into any other problems with trunk, I'll report those here. -- Steve
Re: [OMPI users] tcp communication problems with 1.4.3 and 1.4.4 rc2 on FreeBSD
On Tue, Jul 12, 2011 at 10:37:14AM -0700, Steve Kargl wrote: > On Fri, Jul 08, 2011 at 07:03:13PM -0400, Jeff Squyres wrote: > > Sorry -- I got distracted all afternoon... > > > > In addition to what Ralph said (i.e., I'm not sure if the CIDR > > notation stuff made it over to the v1.5 branch or not, but it > > is available from the nightly SVN trunk tarballs: > > http://www.open-mpi.org/nightly/trunk/), here's a few points > > from other mails in this thread... > > > > trunk does not appear to be an option. :-( > > % svn co http://svn.open-mpi.org/svn/ompi/trunk ompi > % cd ompi > % ./autogen.pl > % ./configure --enable-mpirun-prefix-by-default --prefix=/usr/local/ompi \ >--disable-shared --enable-static > > (many lines removed) > checking prefix for function in .type... @ > checking if .size is needed... yes > checking if .align directive takes logarithmic value... no > configure: error: No atomic primitives available for amd64-unknown-freebsd9.0 It seems the configure script does not recognize amd64. If I add --build='x86_64-*-freebsd' to the configure line, then everything appears to work. I'll report back after I've had a chance to work with ompi built from trunk. -- Steve
Re: [OMPI users] tcp communication problems with 1.4.3 and 1.4.4 rc2 on FreeBSD
On Fri, Jul 08, 2011 at 07:03:13PM -0400, Jeff Squyres wrote: > Sorry -- I got distracted all afternoon... > > In addition to what Ralph said (i.e., I'm not sure if the CIDR > notation stuff made it over to the v1.5 branch or not, but it > is available from the nightly SVN trunk tarballs: > http://www.open-mpi.org/nightly/trunk/), here's a few points > from other mails in this thread... > trunk does not appear to be an option. :-( % svn co http://svn.open-mpi.org/svn/ompi/trunk ompi % cd ompi % ./autogen.pl % ./configure --enable-mpirun-prefix-by-default --prefix=/usr/local/ompi \ --disable-shared --enable-static (many lines removed) checking prefix for function in .type... @ checking if .size is needed... yes checking if .align directive takes logarithmic value... no configure: error: No atomic primitives available for amd64-unknown-freebsd9.0 -- Steve
Re: [OMPI users] icc erroring when trying to use a directory as a source file (Catastrophic error: cannot open source file "paddress.c": is a directory )
I can't quite parse your output. paddress.c should be a sym link in ompi/mpi/c/profile back to ompi/mpi/c/address.c. I'm not sure why "ls paddress.c" shows a whole directory of files...? You might want to whack your OMPI source tree, re-expand the tarball, and try again. If it still fails, please send all the output listed here: http://www.open-mpi.org/community/help/ On Jul 12, 2011, at 3:23 AM, Denis wrote: > Hello, buddies, i am trying to build openmpi with icc and it s not > working. I ve tried versions 1.4.3 and 1.4.2 the error is the same > but for different source files(directories) > > I am using the latest icc version and have compiled same version of > openmpi with gnu before. > Is there any switch that need to be passed to icc in order to be able > to compile a directory as a source code as needed to build openmpi? > > > Mac-$ icc -DHAVE_CONFIG_H -I. -I../../../../opal/include > -I../../../../orte/include -I../../../../ompi/include > -I../../../../opal/mca/paffinity/linux/plpa/src/libplpa > -DOMPI_PROFILING_DEFINES=1 -I../../../.. -D_REENTRANT -O3 -DNDEBUG > -finline-functions -fno-strict-aliasing -restrict -fvisibility=hidden > -MT paddress.lo -MD -MP -MF .deps/paddress.Tpo -c paddress.c > -fno-common -DPIC -o .libs/paddress.o > Catastrophic error: cannot open source file "paddress.c": is a directory > > > $ ls paddress.c > Makefile comm_call_errhandler.c file_c2f.c > file_set_atomicity.cgroup_difference.c > op_create.c status_set_elements.c > type_set_name.c > Makefile.am comm_compare.c > file_call_errhandler.c file_set_errhandler.c group_excl.c > op_f2c.ctest.c > type_size.c > Makefile.in comm_connect.c file_close.c > file_set_info.c group_f2c.c > op_free.c test_cancelled.c > type_struct.c > abort.c comm_create.c > file_create_errhandler.cfile_set_size.c group_free.c > open_port.c testall.c > type_ub.c > accumulate.c comm_create_errhandler.cfile_delete.c > file_set_view.c group_incl.c > pack.c testany.c > type_vector.c > add_error_class.c comm_create_keyval.cfile_f2c.c > > > Thank you in advance! > Denis. > ___ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [hwloc-users] hwloc 1.2 compilation problems
Carl Smith, le Tue 12 Jul 2011 02:46:27 +0200, a écrit : > > is it perhaps the presence of -L/usr/local/lib which makes the linking > > fail? I've commited something that might help. > > Perhaps. Your latest change does work on this AIX system. Thanks > for persisting. Great! I've backported to the 1.2 branch. Samuel
Re: [OMPI users] Mpirun only works when n< 3
On Jul 11, 2011, at 11:31 AM, Randolph Pullen wrote: > There are no firewalls by default. I can ssh between both nodes without a > password so I assumed that all is good with the comms. FWIW, ssh'ing is different than "comms" (which I assume you mean opening random TCP sockets between two servers). > I can also get both nodes to participate in the ring program at the same time. > Its just that I am limited to inly 2 processes if they are split between the > nodes > ie: > mpirun -H A,B ring (works) > mpirun -H A,A,A,A,A,A,A ring (works) > mpirun -H B,B,B,B ring (works) > mpirun -H A,B,A ring(hangs) It is odd that A,B works and A,B,A does not. > I have discovered slightly more information: > When I replace node 'B' from the new cluster with node 'C' from the old > cluster > I get the similar behavior but with an error message: > mpirun -H A,A,A,A,A,A,A ring (works from either node) > mpirun -H C,C,C ring (works from either node) > mpirun -H A,C ring (Fails from either node:) > Process 0 sending 10 to 1, tag 201 (3 processes in ring) > [C:23465] *** An error occurred in MPI_Recv > [C:23465] *** on communicator MPI_COMM_WORLD > [C:23465] *** MPI_ERRORS_ARE FATAL (your job will now abort) > Process 0 sent to 1 > -- > Running this on either node A or C produces the same result > Node C runs openMPI 1.4.1 and is an ordinary dual core on FC10 , not an i5 > 2400 like the others. > all the binaries are compiled on FC10 with gcc 4.3.2 Are you sure that all the versions of Open MPI being used on all nodes are exactly the same? I.e., are you finding/using Open MPI v1.4.1 on all nodes? Are the nodes homogeneous in terms of software? If they're heterogeneous in terms of hardware, you *might* need to have separate OMPI installations on each machine (vs., for example, a network-filesystem-based install shared to all 3) because the compiler's optimizer may produce code tailored for one of the machines, and it may therefore fail in unexpected ways on the other(s). The same is true for your executable. See this FAQ entry about heterogeneous setups: http://www.open-mpi.org/faq/?category=building#where-to-install ...hmm. I could have sworn we had more on the FAQ about heterogeneity, but perhaps not. The old LAM/MPI FAQ on heterogeneity is somewhat outdated, but most of its concepts are directly relevant to Open MPI as well: http://www.lam-mpi.org/faq/category11.php3 I should probably copy most of that LAM/MPI heterogeneous FAQ to the Open MPI FAQ, but it'll be waaay down on my priority list. :-( If anyone could help out here, I'd be happy to point them in the right direction to convert the LAM/MPI FAQ PHP to Open MPI FAQ PHP... To be clear: the PHP conversion will be pretty trivial; I stole heavily from the LAM/MPI FAQ PHP to create the Open MPI FAQ PHP -- but there are points where the LAM/MPI heterogeneity text needs to be updated; that'll take an hour or two to update all that content. -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
