Re: [OMPI users] openMPI on Xgrid

[Klymak Jody]

I have an environment a few trusted users could use to test.  However,  
I have neither the expertise or time to do the debugging myself.


Cheers,  Jody

On 2010-03-29, at 1:27 PM, Jeff Squyres wrote:


On Mar 29, 2010, at 4:11 PM, Cristobal Navarro wrote:


i realized that xcode dev tools include openMPI 1.2.x
should i keep trying??
or do you recommend to completly abandon xgrid and go for another  
tool like Torque with openMPI?


FWIW, Open MPI v1.2.x is fairly ancient -- the v1.4 series includes  
a few years worth of improvements and bug fixes since the 1.2 series.


It would be great (hint hint) if someone could fix the xgrid support  
for us...  We simply no longer have anyone in the active development  
group who has the expertise or test environment to make our xgrid  
work.  :-(


--
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/


___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] openmpi with xgrid

[Klymak Jody]

On 15-Aug-09, at 1:03 AM, Alan wrote:


Thanks Warner,

This is frustrating... I read the ticket. 6 months already and 2  
releases postponed... Frankly, I am very skeptical that this will be  
fixed for 1.3.4. I really hope so, but when 1.3.4 will be released?


I have to think about going with 1.2.x and possible disruptions in  
my configuration (I use Fink) or wait.


And I offered myself to test any nightly snapshot claiming this bug  
is fixed.


Hi Alan,

Its not too hard to get PBS/torque up and running.

The OS/X-specific (?) issues I had:

1) /etc/hosts had to have the server explicitly listed on each of the  
nodes.

2) $usecp had to be set in mom_config
3) $restricted had to be set on the nodes in mom_priv/config to accept  
calls from the server.


I think that is it.  Of course if you are already using xgrid on these  
machines for other uses it won't play well with PBS, but otherwise all  
you are missing is the cute tachometer display.


Cheers,  Jody


Cheers,
Alan

On Fri, Aug 14, 2009 at 17:20, Warner Yuen  wrote:
Hi Alan,

Xgrid support for Open MPI is currently broken in the latest version  
of Open MPI. See the ticket below. However, I believe that Xgrid  
still works with one of the earlier 1.2  versions of Open MPI. I  
don't recall for sure, but I think that it's Open MPI 1.2.3.


#1777: Xgrid support is broken in the v1.3 series
- 
+--

Reporter:  jsquyres  |Owner:  brbarret
  Type:  defect|   Status:  accepted
Priority:  major |Milestone:  Open MPI 1.3.4
Version:  trunk |   Resolution:
Keywords:|
- 
+--

Changes (by bbenton):

 * milestone:  Open MPI 1.3.3 => Open MPI 1.3.4


Warner Yuen
Scientific Computing
Consulting Engineer
Apple, Inc.
email: wy...@apple.com
Tel: 408.718.2859




On Aug 14, 2009, at 6:21 AM, users-requ...@open-mpi.org wrote:


Message: 1
Date: Fri, 14 Aug 2009 14:21:30 +0100
From: Alan 
Subject: [OMPI users] openmpi with xgrid
To: us...@open-mpi.org
Message-ID:
   
Content-Type: text/plain; charset="utf-8"


Hi there,
I saw that http://www.open-mpi.org/community/lists/users/2007/08/3900.php 
.


I use fink, and so I changed the openmpi.info file in order to get  
openmpi

with xgrid support.

As you can see:
amadeus[2081]:~/Downloads% /sw/bin/ompi_info
   Package: Open MPI root@amadeus.local Distribution
  Open MPI: 1.3.3
 Open MPI SVN revision: r21666
 Open MPI release date: Jul 14, 2009
  Open RTE: 1.3.3
 Open RTE SVN revision: r21666
 Open RTE release date: Jul 14, 2009
  OPAL: 1.3.3
 OPAL SVN revision: r21666
 OPAL release date: Jul 14, 2009
  Ident string: 1.3.3
Prefix: /sw
Configured architecture: x86_64-apple-darwin9
Configure host: amadeus.local
 Configured by: root
 Configured on: Fri Aug 14 12:58:12 BST 2009
Configure host: amadeus.local
  Built by:
  Built on: Fri Aug 14 13:07:46 BST 2009
Built host: amadeus.local
C bindings: yes
  C++ bindings: yes
Fortran77 bindings: yes (single underscore)
Fortran90 bindings: yes
Fortran90 bindings size: small
C compiler: gcc
   C compiler absolute: /sw/var/lib/fink/path-prefix-10.6/gcc
  C++ compiler: g++
 C++ compiler absolute: /sw/var/lib/fink/path-prefix-10.6/g++
Fortran77 compiler: gfortran
 Fortran77 compiler abs: /sw/bin/gfortran
Fortran90 compiler: gfortran
 Fortran90 compiler abs: /sw/bin/gfortran
   C profiling: yes
 C++ profiling: yes
   Fortran77 profiling: yes
   Fortran90 profiling: yes
C++ exceptions: no
Thread support: posix (mpi: no, progress: no)
 Sparse Groups: no
 Internal debug support: no
   MPI parameter check: runtime
Memory profiling support: no
Memory debugging support: no
   libltdl support: yes
 Heterogeneous support: no
mpirun default --prefix: no
   MPI I/O support: yes
 MPI_WTIME support: gettimeofday
Symbol visibility support: yes
 FT Checkpoint support: no  (checkpoint thread: no)
 MCA backtrace: execinfo (MCA v2.0, API v2.0, Component  
v1.3.3)

 MCA paffinity: darwin (MCA v2.0, API v2.0, Component v1.3.3)
 MCA carto: auto_detect (MCA v2.0, API v2.0, Component  
v1.3.3)

 MCA carto: file (MCA v2.0, API v2.0, Component v1.3.3)
 MCA maffinity: first_use (MCA v2.0, API v2.0, Component  
v1.3.3)

 MCA timer: darwin (MCA v2.0, API v2.0, Component v1.3.3)
   MCA installdirs: env (MCA v2.0, API v2.0, Component v1.3.3)
   MCA installdirs: config (MCA v2.0, API v2.0, Component v1.3.3)
   MCA dpm: orte (MCA v2.0, API v2.0, Component v1.3.3)
MCA pubsub: orte 

Re: [OMPI users] torque pbs behaviour...

[Klymak Jody]

On 11-Aug-09, at 6:16 AM, Jeff Squyres wrote:

This means that OMPI is finding an mca_iof_proxy.la file at run time  
from a prior version of Open MPI.  You might want to use "find" or  
"locate" to search your nodes and find it.  I suspect that you  
somehow have an OMPI 1.3.x install that overlaid an install of a  
prior OMPI version installation.



OK, right you were - the old file was in my new install directory.  I  
didn't erase /usr/local/openmpi before re-running the install...


However, after reinstalling on the nodes (but not cleaning out /usr/ 
lib on all the nodes) I still have the following:


Thanks,  Jody


saturna.cluster:17660] mca:base:select:(  plm) Querying component [rsh]
[saturna.cluster:17660] mca:base:select:(  plm) Query of component  
[rsh] set priority to 10
[saturna.cluster:17660] mca:base:select:(  plm) Querying component  
[slurm]
[saturna.cluster:17660] mca:base:select:(  plm) Skipping component  
[slurm]. Query failed to return a module

[saturna.cluster:17660] mca:base:select:(  plm) Querying component [tm]
[saturna.cluster:17660] mca:base:select:(  plm) Skipping component  
[tm]. Query failed to return a module
[saturna.cluster:17660] mca:base:select:(  plm) Querying component  
[xgrid]
[saturna.cluster:17660] mca:base:select:(  plm) Skipping component  
[xgrid]. Query failed to return a module

[saturna.cluster:17660] mca:base:select:(  plm) Selected component [rsh]
[saturna.cluster:17660] plm:base:set_hnp_name: initial bias 17660  
nodename hash 1656374957

[saturna.cluster:17660] plm:base:set_hnp_name: final jobfam 24811
[saturna.cluster:17660] [[24811,0],0] plm:base:receive start comm
[saturna.cluster:17660] mca:base:select:( odls) Querying component  
[default]
[saturna.cluster:17660] mca:base:select:( odls) Query of component  
[default] set priority to 1
[saturna.cluster:17660] mca:base:select:( odls) Selected component  
[default]

[saturna.cluster:17660] [[24811,0],0] plm:rsh: setting up job [24811,1]
[saturna.cluster:17660] [[24811,0],0] plm:base:setup_job for job  
[24811,1]

[saturna.cluster:17660] [[24811,0],0] plm:rsh: local shell: 0 (bash)
[saturna.cluster:17660] [[24811,0],0] plm:rsh: assuming same remote  
shell as local shell

[saturna.cluster:17660] [[24811,0],0] plm:rsh: remote shell: 0 (bash)
[saturna.cluster:17660] [[24811,0],0] plm:rsh: final template argv:
	/usr/bin/ssh   PATH=/usr/local/openmpi/bin:$PATH ; export  
PATH ; LD_LIBRARY_PATH=/usr/local/openmpi/lib:$LD_LIBRARY_PATH ;  
export LD_LIBRARY_PATH ;  /usr/local/openmpi/bin/orted --debug-daemons  
-mca ess env -mca orte_ess_jobid 1626013696 -mca orte_ess_vpid  
 -mca orte_ess_num_procs 3 --hnp-uri "1626013696.0;tcp:// 
142.104.154.96:49710;tcp://192.168.2.254:49710" -mca plm_base_verbose  
5 -mca odls_base_verbose 5
[saturna.cluster:17660] [[24811,0],0] plm:rsh: launching on node  
xserve01
[saturna.cluster:17660] [[24811,0],0] plm:rsh: recording launch of  
daemon [[24811,0],1]
[saturna.cluster:17660] [[24811,0],0] plm:rsh: executing: (//usr/bin/ 
ssh) [/usr/bin/ssh xserve01  PATH=/usr/local/openmpi/bin:$PATH ;  
export PATH ; LD_LIBRARY_PATH=/usr/local/openmpi/lib: 
$LD_LIBRARY_PATH ; export LD_LIBRARY_PATH ;  /usr/local/openmpi/bin/ 
orted --debug-daemons -mca ess env -mca orte_ess_jobid 1626013696 -mca  
orte_ess_vpid 1 -mca orte_ess_num_procs 3 --hnp-uri  
"1626013696.0;tcp://142.104.154.96:49710;tcp://192.168.2.254:49710" - 
mca plm_base_verbose 5 -mca odls_base_verbose 5]

Daemon was launched on xserve01.cluster - beginning to initialize
[xserve01.cluster:42519] mca:base:select:( odls) Querying component  
[default]
[xserve01.cluster:42519] mca:base:select:( odls) Query of component  
[default] set priority to 1
[xserve01.cluster:42519] mca:base:select:( odls) Selected component  
[default]

Daemon [[24811,0],1] checking in as pid 42519 on host xserve01.cluster
Daemon [[24811,0],1] not using static ports
[saturna.cluster:17660] [[24811,0],0] plm:rsh: launching on node  
xserve02
[saturna.cluster:17660] [[24811,0],0] plm:rsh: recording launch of  
daemon [[24811,0],2]
[saturna.cluster:17660] [[24811,0],0] plm:rsh: executing: (//usr/bin/ 
ssh) [/usr/bin/ssh xserve02  PATH=/usr/local/openmpi/bin:$PATH ;  
export PATH ; LD_LIBRARY_PATH=/usr/local/openmpi/lib: 
$LD_LIBRARY_PATH ; export LD_LIBRARY_PATH ;  /usr/local/openmpi/bin/ 
orted --debug-daemons -mca ess env -mca orte_ess_jobid 1626013696 -mca  
orte_ess_vpid 2 -mca orte_ess_num_procs 3 --hnp-uri  
"1626013696.0;tcp://142.104.154.96:49710;tcp://192.168.2.254:49710" - 
mca plm_base_verbose 5 -mca odls_base_verbose 5]

Daemon was launched on xserve02.local - beginning to initialize
[xserve02.local:42180] mca:base:select:( odls) Querying component  
[default]
[xserve02.local:42180] mca:base:select:( odls) Query of component  
[default] set priority to 1
[xserve02.local:42180] mca:base:select:( odls) Selected component  
[default]

Daemon [[24811,0],2] checking in as pid 42180 on host xserve02.local
Daemon [[24811,0],2] not using 

Re: [OMPI users] torque pbs behaviour...

[Klymak Jody]

On 11-Aug-09, at 7:03 AM, Ralph Castain wrote:


Sigh - too early in the morning for this old brain, I fear...

You are right - the ranks are fine, and local rank doesn't matter.  
It sounds like a problem where the TCP messaging is getting a  
message ack'd from someone other than the process that was supposed  
to recv the message. This should cause us to abort, but we were just  
talking on the phone that the abort procedure may not be working  
correctly. Or it could be (as Jeff suggests) that the version  
mismatch is also preventing us from properly aborting too.


So I fear we are back to trying to find these other versions on your  
nodes...


Well, the old version is still on the nodes (in /usr/lib as default  
for OS X)...


I can try and clean those all out by hand but I'm still confused why  
the old version would be used - how does openMPI find the right library?


Note again, that I get these MCA warnings on the server when just  
running ompi_info and I *have* cleaned out /usr/lib on the server.  So  
I really don't understand how on the server I can still have a library  
issue.  Is there a way to trace at runtime what library an executable  
is dynamically linking to?  Can I rebuild openmpi statically?


Thanks,  Jody





On Tue, Aug 11, 2009 at 7:43 AM, Klymak Jody <jkly...@uvic.ca> wrote:

On 11-Aug-09, at 6:28 AM, Ralph Castain wrote:

The reason your job is hanging is sitting in the orte-ps output. You  
have multiple processes declaring themselves to be the same MPI  
rank. That definitely won't work.


Its the "local rank" if that makes any difference...

Any thoughts on this output?


[xserve03.local][[61029,1],4][btl_tcp_endpoint.c: 
486:mca_btl_tcp_endpoint_recv_connect_ack] received unexpected  
process identifier [[61029,1],3]


The question is why is that happening? We use Torque all the time,  
so we know that the basic support is correct. It -could- be related  
to lib confusion, but I can't tell for sure.


Just to be clear, this is not going through torque at this point.   
Its just vanilla ssh, for which this code worked with 1.1.5.




Can you rebuild OMPI with --enable-debug, and rerun the job with the  
following added to your cmd line?


-mca plm_base_verbose 5 --debug-daemons -mca odls_base_verbose 5

Working on this...

Thanks,  Jody

___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] torque pbs behaviour...

[Klymak Jody]

On 11-Aug-09, at 6:28 AM, Ralph Castain wrote:


-mca plm_base_verbose 5 --debug-daemons -mca odls_base_verbose 5

I'm afraid the output will be a tad verbose, but I would appreciate  
seeing it. Might also tell us something about the lib issue.



Command line was:

/usr/local/openmpi/bin/mpirun -mca plm_base_verbose 5 --debug-daemons - 
mca odls_base_verbose 5 -n 16 --host xserve03,xserve04 ../build/mitgcmuv



Starting: ../results//TasGaussRestart16
[saturna.cluster:07360] mca:base:select:(  plm) Querying component [rsh]
[saturna.cluster:07360] mca:base:select:(  plm) Query of component  
[rsh] set priority to 10
[saturna.cluster:07360] mca:base:select:(  plm) Querying component  
[slurm]
[saturna.cluster:07360] mca:base:select:(  plm) Skipping component  
[slurm]. Query failed to return a module

[saturna.cluster:07360] mca:base:select:(  plm) Querying component [tm]
[saturna.cluster:07360] mca:base:select:(  plm) Skipping component  
[tm]. Query failed to return a module
[saturna.cluster:07360] mca:base:select:(  plm) Querying component  
[xgrid]
[saturna.cluster:07360] mca:base:select:(  plm) Skipping component  
[xgrid]. Query failed to return a module

[saturna.cluster:07360] mca:base:select:(  plm) Selected component [rsh]
[saturna.cluster:07360] plm:base:set_hnp_name: initial bias 7360  
nodename hash 1656374957

[saturna.cluster:07360] plm:base:set_hnp_name: final jobfam 14551
[saturna.cluster:07360] [[14551,0],0] plm:base:receive start comm
[saturna.cluster:07360] mca: base: component_find: ras  
"mca_ras_dash_host" uses an MCA interface that is not recognized  
(component MCA v1.0.0 != supported MCA v2.0.0) -- ignored
[saturna.cluster:07360] mca: base: component_find: ras  
"mca_ras_hostfile" uses an MCA interface that is not recognized  
(component MCA v1.0.0 != supported MCA v2.0.0) -- ignored
[saturna.cluster:07360] mca: base: component_find: ras  
"mca_ras_localhost" uses an MCA interface that is not recognized  
(component MCA v1.0.0 != supported MCA v2.0.0) -- ignored
[saturna.cluster:07360] mca: base: component_find: ras "mca_ras_xgrid"  
uses an MCA interface that is not recognized (component MCA v1.0.0 !=  
supported MCA v2.0.0) -- ignored
[saturna.cluster:07360] mca:base:select:( odls) Querying component  
[default]
[saturna.cluster:07360] mca:base:select:( odls) Query of component  
[default] set priority to 1
[saturna.cluster:07360] mca:base:select:( odls) Selected component  
[default]
[saturna.cluster:07360] mca: base: component_find: iof "mca_iof_proxy"  
uses an MCA interface that is not recognized (component MCA v1.0.0 !=  
supported MCA v2.0.0) -- ignored
[saturna.cluster:07360] mca: base: component_find: iof "mca_iof_svc"  
uses an MCA interface that is not recognized (component MCA v1.0.0 !=  
supported MCA v2.0.0) -- ignored

[saturna.cluster:07360] [[14551,0],0] plm:rsh: setting up job [14551,1]
[saturna.cluster:07360] [[14551,0],0] plm:base:setup_job for job  
[14551,1]

[saturna.cluster:07360] [[14551,0],0] plm:rsh: local shell: 0 (bash)
[saturna.cluster:07360] [[14551,0],0] plm:rsh: assuming same remote  
shell as local shell

[saturna.cluster:07360] [[14551,0],0] plm:rsh: remote shell: 0 (bash)
[saturna.cluster:07360] [[14551,0],0] plm:rsh: final template argv:
	/usr/bin/ssh   PATH=/usr/local/openmpi/bin:$PATH ; export  
PATH ; LD_LIBRARY_PATH=/usr/local/openmpi/lib:$LD_LIBRARY_PATH ;  
export LD_LIBRARY_PATH ;  /usr/local/openmpi/bin/orted --debug-daemons  
-mca ess env -mca orte_ess_jobid 953614336 -mca orte_ess_vpid  
 -mca orte_ess_num_procs 3 --hnp-uri "953614336.0;tcp:// 
142.104.154.96:49622;tcp://192.168.2.254:49622" -mca plm_base_verbose  
5 -mca odls_base_verbose 5
[saturna.cluster:07360] [[14551,0],0] plm:rsh: launching on node  
xserve03
[saturna.cluster:07360] [[14551,0],0] plm:rsh: recording launch of  
daemon [[14551,0],1]
[saturna.cluster:07360] [[14551,0],0] plm:rsh: executing: (//usr/bin/ 
ssh) [/usr/bin/ssh xserve03  PATH=/usr/local/openmpi/bin:$PATH ;  
export PATH ; LD_LIBRARY_PATH=/usr/local/openmpi/lib: 
$LD_LIBRARY_PATH ; export LD_LIBRARY_PATH ;  /usr/local/openmpi/bin/ 
orted --debug-daemons -mca ess env -mca orte_ess_jobid 953614336 -mca  
orte_ess_vpid 1 -mca orte_ess_num_procs 3 --hnp-uri "953614336.0;tcp:// 
142.104.154.96:49622;tcp://192.168.2.254:49622" -mca plm_base_verbose  
5 -mca odls_base_verbose 5]

Daemon was launched on xserve03.local - beginning to initialize
[xserve03.local:40708] mca:base:select:( odls) Querying component  
[default]
[xserve03.local:40708] mca:base:select:( odls) Query of component  
[default] set priority to 1
[xserve03.local:40708] mca:base:select:( odls) Selected component  
[default]
[xserve03.local:40708] mca: base: component_find: iof "mca_iof_proxy"  
uses an MCA interface that is not recognized (component MCA v1.0.0 !=  
supported MCA v2.0.0) -- ignored
[xserve03.local:40708] mca: base: component_find: iof "mca_iof_svc"  
uses an MCA interface that is not recognized (component MCA v1.0.0 !=  

Re: [OMPI users] torque pbs behaviour...

[Klymak Jody]

On 11-Aug-09, at 6:28 AM, Ralph Castain wrote:

The reason your job is hanging is sitting in the orte-ps output. You  
have multiple processes declaring themselves to be the same MPI  
rank. That definitely won't work.


Its the "local rank" if that makes any difference...

Any thoughts on this output?

[xserve03.local][[61029,1],4][btl_tcp_endpoint.c: 
486:mca_btl_tcp_endpoint_recv_connect_ack] received unexpected process  
identifier [[61029,1],3]


The question is why is that happening? We use Torque all the time,  
so we know that the basic support is correct. It -could- be related  
to lib confusion, but I can't tell for sure.


Just to be clear, this is not going through torque at this point.  Its  
just vanilla ssh, for which this code worked with 1.1.5.



Can you rebuild OMPI with --enable-debug, and rerun the job with the  
following added to your cmd line?


-mca plm_base_verbose 5 --debug-daemons -mca odls_base_verbose 5


Working on this...

Thanks,  Jody

Re: [OMPI users] torque pbs behaviour...

[Klymak Jody]

On 10-Aug-09, at 8:03 PM, Ralph Castain wrote:

Interesting! Well, I always make sure I have my personal OMPI build  
before any system stuff, and I work exclusively on Mac OS-X:


I am still finding this very mysterious

I have removed all the OS-X -supplied libraries, recompiled and  
installed openmpi 1.3.3, and I am *still* getting this warning when  
running ompi_info:


[saturna.cluster:50307] mca: base: component_find: iof "mca_iof_proxy"  
uses an MCA interface that is not recognized (component MCA v1.0.0 !=  
supported MCA v2.0.0) -- ignored
[saturna.cluster:50307] mca: base: component_find: iof "mca_iof_svc"  
uses an MCA interface that is not recognized (component MCA v1.0.0 !=  
supported MCA v2.0.0) -- ignored
[saturna.cluster:50307] mca: base: component_find: ras  
"mca_ras_dash_host" uses an MCA interface that is not recognized  
(component MCA v1.0.0 != supported MCA v2.0.0) -- ignored
[saturna.cluster:50307] mca: base: component_find: ras  
"mca_ras_hostfile" uses an MCA interface that is not recognized  
(component MCA v1.0.0 != supported MCA v2.0.0) -- ignored
[saturna.cluster:50307] mca: base: component_find: ras  
"mca_ras_localhost" uses an MCA interface that is not recognized  
(component MCA v1.0.0 != supported MCA v2.0.0) -- ignored
[saturna.cluster:50307] mca: base: component_find: ras "mca_ras_xgrid"  
uses an MCA interface that is not recognized (component MCA v1.0.0 !=  
supported MCA v2.0.0) -- ignored
[saturna.cluster:50307] mca: base: component_find: rcache  
"mca_rcache_rb" uses an MCA interface that is not recognized  
(component MCA v1.0.0 != supported MCA v2.0.0) -- ignored


So, I guess I'm not clear how the library can be an issue...

I *do* get another error from running the gcm that I do not get from  
running simpler jobs - hopefully this helps explain things:


[xserve03.local][[61029,1],4][btl_tcp_endpoint.c: 
486:mca_btl_tcp_endpoint_recv_connect_ack] received unexpected process  
identifier [[61029,1],3]


The processes are running, the mitgcmuv processes are running on the  
xserves, and using considerable resources!  They open STDERR/STDOUT  
but nothing is flushed into them, including the few print statements  
I've put in before and after MPI_INIT (as Ralph suggested).


On 11-Aug-09, at 4:17 AM, Ashley Pittman wrote:

If you suspect a hang then you can use the command orte-ps (on the  
node
where the mpirun is running) and it should show you your job.  This  
will

tell you if the job is started and still running or if there was a
problem launching.


/usr/local/openmpi/bin/orte-ps
[saturna.cluster:51840] mca: base: component_find: iof "mca_iof_proxy"  
uses an MCA interface that is not recognized (component MCA v1.0.0 !=  
supported MCA v2.0.0) -- ignored
[saturna.cluster:51840] mca: base: component_find: iof "mca_iof_svc"  
uses an MCA interface that is not recognized (component MCA v1.0.0 !=  
supported MCA v2.0.0) -- ignored



Information from mpirun [61029,0]
---

JobID |   State |  Slots | Num Procs |
--
[61029,1] | Running |  2 |16 |
	 Process Name |  ORTE Name | Local Rank |PID | Node  
|   State |


---
	../build/mitgcmuv |  [[61029,1],0] |  0 |  40206 | xserve03 |  
Running |
	../build/mitgcmuv |  [[61029,1],1] |  0 |  40005 | xserve04 |  
Running |
	../build/mitgcmuv |  [[61029,1],2] |  1 |  40207 | xserve03 |  
Running |
	../build/mitgcmuv |  [[61029,1],3] |  1 |  40006 | xserve04 |  
Running |
	../build/mitgcmuv |  [[61029,1],4] |  2 |  40208 | xserve03 |  
Running |
	../build/mitgcmuv |  [[61029,1],5] |  2 |  40007 | xserve04 |  
Running |
	../build/mitgcmuv |  [[61029,1],6] |  3 |  40209 | xserve03 |  
Running |
	../build/mitgcmuv |  [[61029,1],7] |  3 |  40008 | xserve04 |  
Running |
	../build/mitgcmuv |  [[61029,1],8] |  4 |  40210 | xserve03 |  
Running |
	../build/mitgcmuv |  [[61029,1],9] |  4 |  40009 | xserve04 |  
Running |
	../build/mitgcmuv | [[61029,1],10] |  5 |  40211 | xserve03 |  
Running |
	../build/mitgcmuv | [[61029,1],11] |  5 |  40010 | xserve04 |  
Running |
	../build/mitgcmuv | [[61029,1],12] |  6 |  40212 | xserve03 |  
Running |
	../build/mitgcmuv | [[61029,1],13] |  6 |  40011 | xserve04 |  
Running |
	../build/mitgcmuv | [[61029,1],14] |  7 |  40213 | xserve03 |  
Running |
	../build/mitgcmuv | [[61029,1],15] |  7 |  40012 | xserve04 |  
Running |


Thanks,  Jody

Re: [OMPI users] torque pbs behaviour...

[Klymak Jody]

On 10-Aug-09, at 8:03 PM, Ralph Castain wrote:

Interesting! Well, I always make sure I have my personal OMPI build  
before any system stuff, and I work exclusively on Mac OS-X:


Note that I always configure with --prefix=somewhere-in-my-own-dir,  
never to a system directory. Avoids this kind of confusion.


Yeah, I did configure --prefix=/usr/local/openmpi

What the errors are saying is that we are picking up components from  
a very old version of OMPI that is distributed by Apple. It may or  
may not be causing confusion for the system - hard to tell. However,  
the fact that it is the IO forwarding subsystem that is picking them  
up, and the fact that you aren't seeing any output from your job,  
makes me a tad suspicious.


Me too!

Can you run other jobs? In other words, do you get stdout/stderr  
from other programs you run, or does every MPI program hang (even  
simple ones)? If it is just your program, then it could just be that  
your application is hanging before any output is generated. Can you  
have it print something to stderr right when it starts?


No simple ones, like the examples I gave before, run fine, just with  
the suspicious warnings.


I'm running a big general circulation model (MITgcm).  Under normal  
conditions it spits something out almost right away, and that is not  
being done here.  STDOUT.0001 etc are all opened, but nothing is put  
into them.


I'm pretty sure I'm compliling the gcm properly:

otool -L mitgcmuv
mitgcmuv:
	/usr/local/openmpi/lib/libmpi_f77.0.dylib (compatibility version  
1.0.0, current version 1.0.0)
	/usr/local/openmpi/lib/libmpi.0.dylib (compatibility version 1.0.0,  
current version 1.0.0)
	/usr/local/openmpi/lib/libopen-rte.0.dylib (compatibility version  
1.0.0, current version 1.0.0)
	/usr/local/openmpi/lib/libopen-pal.0.dylib (compatibility version  
1.0.0, current version 1.0.0)
	/usr/lib/libutil.dylib (compatibility version 1.0.0, current version  
1.0.0)
	/usr/local/lib/libgfortran.3.dylib (compatibility version 4.0.0,  
current version 4.0.0)
	/usr/local/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current  
version 1.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current  
version 111.1.3)


Thanks,  Jody




On Aug 10, 2009, at 8:53 PM, Klymak Jody wrote:



On 10-Aug-09, at 6:44 PM, Ralph Castain wrote:

Check your LD_LIBRARY_PATH - there is an earlier version of OMPI  
in your path that is interfering with operation (i.e., it comes  
before your 1.3.3 installation).


H, The OS X faq says not to do this:

"Note that there is no need to add Open MPI's libdir to  
LD_LIBRARY_PATH; Open MPI's shared library build process  
automatically uses the "rpath" mechanism to automatically find the  
correct shared libraries (i.e., the ones associated with this  
build, vs., for example, the OS X-shipped OMPI shared libraries).  
Also note that we specifically do not recommend adding Open MPI's  
libdir to DYLD_LIBRARY_PATH."


http://www.open-mpi.org/faq/?category=osx

Regardless, if I set either, and run ompi_info I still get:

[saturna.cluster:94981] mca: base: component_find: iof  
"mca_iof_proxy" uses an MCA interface that is not recognized  
(component MCA v1.0.0 != supported MCA v2.0.0) -- ignored
[saturna.cluster:94981] mca: base: component_find: iof  
"mca_iof_svc" uses an MCA interface that is not recognized  
(component MCA v1.0.0 != supported MCA v2.0.0) -- ignored


echo $DYLD_LIBRARY_PATH $LD_LIBRARY_PATH
/usr/local/openmpi/lib: /usr/local/openmpi/lib:

So I'm afraid I'm stumped again.  I suppose I could go clean out  
all the libraries in /usr/lib/...


Thanks again, sorry to be a pain...

Cheers,  Jody






On Aug 10, 2009, at 7:38 PM, Klymak Jody wrote:


So,

mpirun --display-allocation -pernode --display-map hostname

gives me the output below.  Simple jobs seem to run, but the  
MITgcm does not, either under ssh or torque.  It hangs at some  
early point in execution before anything is written, so its hard  
for me to tell what the error is.  Could these MCA warnings have  
anything to do with it?


I've recompiled the gcm with -L /usr/local/openmpi/lib, so  
hopefully that catches the right library.


Thanks,  Jody


[xserve02.local:38126] mca: base: component_find: ras  
"mca_ras_dash_host" uses an MCA interface that is not recogniz

ed (component MCA v1.0.0 != supported MCA v2.0.0) -- ignored
[xserve02.local:38126] mca: base: component_find: ras  
"mca_ras_hostfile" uses an MCA interface that is not recognize

d (component MCA v1.0.0 != supported MCA v2.0.0) -- ignored
[xserve02.local:38126] mca: base: component_find: ras  
"mca_ras_localhost" uses an MCA interface that is not recogniz

ed (component MCA v1.0.0 != supported MCA v2.0.0) -- ignored
[xserve02.local:38126] mca: base: component_find: ras  
"mca_ras_xgrid" uses an MCA interface that is not recognized (

component MCA v1.0.0 != supported MCA v2.0.0) -

Re: [OMPI users] torque pbs behaviour...

[Klymak Jody]

On 10-Aug-09, at 6:44 PM, Ralph Castain wrote:

Check your LD_LIBRARY_PATH - there is an earlier version of OMPI in  
your path that is interfering with operation (i.e., it comes before  
your 1.3.3 installation).


H, The OS X faq says not to do this:

"Note that there is no need to add Open MPI's libdir to  
LD_LIBRARY_PATH; Open MPI's shared library build process automatically  
uses the "rpath" mechanism to automatically find the correct shared  
libraries (i.e., the ones associated with this build, vs., for  
example, the OS X-shipped OMPI shared libraries). Also note that we  
specifically do not recommend adding Open MPI's libdir to  
DYLD_LIBRARY_PATH."


http://www.open-mpi.org/faq/?category=osx

Regardless, if I set either, and run ompi_info I still get:

[saturna.cluster:94981] mca: base: component_find: iof "mca_iof_proxy"  
uses an MCA interface that is not recognized (component MCA v1.0.0 !=  
supported MCA v2.0.0) -- ignored
[saturna.cluster:94981] mca: base: component_find: iof "mca_iof_svc"  
uses an MCA interface that is not recognized (component MCA v1.0.0 !=  
supported MCA v2.0.0) -- ignored


echo $DYLD_LIBRARY_PATH $LD_LIBRARY_PATH
/usr/local/openmpi/lib: /usr/local/openmpi/lib:

So I'm afraid I'm stumped again.  I suppose I could go clean out all  
the libraries in /usr/lib/...


Thanks again, sorry to be a pain...

Cheers,  Jody






On Aug 10, 2009, at 7:38 PM, Klymak Jody wrote:


So,

mpirun --display-allocation -pernode --display-map hostname

gives me the output below.  Simple jobs seem to run, but the MITgcm  
does not, either under ssh or torque.  It hangs at some early point  
in execution before anything is written, so its hard for me to tell  
what the error is.  Could these MCA warnings have anything to do  
with it?


I've recompiled the gcm with -L /usr/local/openmpi/lib, so  
hopefully that catches the right library.


Thanks,  Jody


[xserve02.local:38126] mca: base: component_find: ras  
"mca_ras_dash_host" uses an MCA interface that is not recogniz

ed (component MCA v1.0.0 != supported MCA v2.0.0) -- ignored
[xserve02.local:38126] mca: base: component_find: ras  
"mca_ras_hostfile" uses an MCA interface that is not recognize

d (component MCA v1.0.0 != supported MCA v2.0.0) -- ignored
[xserve02.local:38126] mca: base: component_find: ras  
"mca_ras_localhost" uses an MCA interface that is not recogniz

ed (component MCA v1.0.0 != supported MCA v2.0.0) -- ignored
[xserve02.local:38126] mca: base: component_find: ras  
"mca_ras_xgrid" uses an MCA interface that is not recognized (

component MCA v1.0.0 != supported MCA v2.0.0) -- ignored
[xserve02.local:38126] mca: base: component_find: iof  
"mca_iof_proxy" uses an MCA interface that is not recognized (

component MCA v1.0.0 != supported MCA v2.0.0) -- ignored
[xserve02.local:38126] mca: base: component_find: iof "mca_iof_svc"  
uses an MCA interface that is not recognized (co

mponent MCA v1.0.0 != supported MCA v2.0.0) -- ignored

==   ALLOCATED NODES   ==

Data for node: Name: xserve02.localNum slots: 8Max slots: 0
Data for node: Name: xserve01.localNum slots: 8Max slots: 0

=

   JOB MAP   

Data for node: Name: xserve02.localNum procs: 1
  Process OMPI jobid: [20967,1] Process rank: 0

Data for node: Name: xserve01.localNum procs: 1
  Process OMPI jobid: [20967,1] Process rank: 1

=
[xserve01.cluster:38518] mca: base: component_find: iof  
"mca_iof_proxy" uses an MCA interface that is not recognized

(component MCA v1.0.0 != supported MCA v2.0.0) -- ignored
[xserve01.cluster:38518] mca: base: component_find: iof  
"mca_iof_svc" uses an MCA interface that is not recognized (

component MCA v1.0.0 != supported MCA v2.0.0) -- ignored
xserve02.local
xserve01.cluster


___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users


___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] torque pbs behaviour...

[Klymak Jody]

So,

mpirun --display-allocation -pernode --display-map hostname

gives me the output below.  Simple jobs seem to run, but the MITgcm  
does not, either under ssh or torque.  It hangs at some early point in  
execution before anything is written, so its hard for me to tell what  
the error is.  Could these MCA warnings have anything to do with it?


I've recompiled the gcm with -L /usr/local/openmpi/lib, so hopefully  
that catches the right library.


Thanks,  Jody


[xserve02.local:38126] mca: base: component_find: ras  
"mca_ras_dash_host" uses an MCA interface that is not recogniz

ed (component MCA v1.0.0 != supported MCA v2.0.0) -- ignored
[xserve02.local:38126] mca: base: component_find: ras  
"mca_ras_hostfile" uses an MCA interface that is not recognize

d (component MCA v1.0.0 != supported MCA v2.0.0) -- ignored
[xserve02.local:38126] mca: base: component_find: ras  
"mca_ras_localhost" uses an MCA interface that is not recogniz

ed (component MCA v1.0.0 != supported MCA v2.0.0) -- ignored
[xserve02.local:38126] mca: base: component_find: ras "mca_ras_xgrid"  
uses an MCA interface that is not recognized (

component MCA v1.0.0 != supported MCA v2.0.0) -- ignored
[xserve02.local:38126] mca: base: component_find: iof "mca_iof_proxy"  
uses an MCA interface that is not recognized (

component MCA v1.0.0 != supported MCA v2.0.0) -- ignored
[xserve02.local:38126] mca: base: component_find: iof "mca_iof_svc"  
uses an MCA interface that is not recognized (co

mponent MCA v1.0.0 != supported MCA v2.0.0) -- ignored

==   ALLOCATED NODES   ==

 Data for node: Name: xserve02.localNum slots: 8Max slots: 0
 Data for node: Name: xserve01.localNum slots: 8Max slots: 0

=

    JOB MAP   

 Data for node: Name: xserve02.localNum procs: 1
Process OMPI jobid: [20967,1] Process rank: 0

 Data for node: Name: xserve01.localNum procs: 1
Process OMPI jobid: [20967,1] Process rank: 1

 =
[xserve01.cluster:38518] mca: base: component_find: iof  
"mca_iof_proxy" uses an MCA interface that is not recognized

 (component MCA v1.0.0 != supported MCA v2.0.0) -- ignored
[xserve01.cluster:38518] mca: base: component_find: iof "mca_iof_svc"  
uses an MCA interface that is not recognized (

component MCA v1.0.0 != supported MCA v2.0.0) -- ignored
xserve02.local
xserve01.cluster

Re: [OMPI users] 2 to 1 oversubscription

[Klymak Jody]

Hi Robert,

Sorry if this is offtopic for the more knowledgeable here...

On 14-Jul-09, at 7:50 PM, Robert Kubrick wrote:
By setting processor affinity you can force execution of each  
process on a specific core, thus limiting context switching. I know  
affinity wasn't supported on MacOS last year, I don't know if the  
situation has changed.
But running oversubscription without process affinity might cancel  
the benefit of SMT because the OS will try to allocate each process  
to whatever core becomes available, thus increasing context switching.


This is a little over my head (i.e. SMT?).  However, to explain, the  
jobs were a gridded simulation, with the grid divided into 8, or 16  
'tiles' .  Each core gets a tile and passes info the the adjacent  
ones.  I would be very surprised to find out that the tiles were  
changing cores mid simulation. Why would the OS do something so silly?


The machines were certainly still running other processes to keep the  
operating system going.  If you watch the cpu monitor, the total would  
occasionally drop from 100% to 98% as some operating system process  
kicked in, but in general the jobs were pegged, leaving little  
opportunity for one core to decide to take over what another core was  
already doing.


Thanks, and if I'm incorrect about how the jobs get distributed  
between cores, I'd be more than happy to be corrected.  As I said, my  
knowledge of this stuff is pretty abstract.


Thanks,   Jody

Re: [OMPI users] 2 to 1 oversubscription

[Klymak Jody]

On 14-Jul-09, at 5:14 PM, Robert Kubrick wrote:


Jody,

Just to make sure, you did set processor affinity during your test  
right?


I'm not sure what that means in the context of OS X.

Hyperthreading was turned on.

Cheers,  Jody



On Jul 13, 2009, at 9:28 PM, Klymak Jody wrote:


Hi Robert,

I got inspired by your question to run a few more tests.  They are  
crude, and I don't have actual cpu timing information because of a  
library mismatch.  However:


Setup:
Xserve, 2x2.26 GHz Quad-core Intel Xeon
6.0 Gb memory 1067 MHz DDR3
Mac OS X 10.5.6

Nodes are connected with a dedicated gigabit ethernet switch.

I'm running the MITgcm, a nonhydrostatic global circulation model.   
The grid size is modest: 10x150x1600, so bear that in mind.   
Message passing is on the dimension that is 150x10, and typically  
is 3 grid cells in either direction.  I'm not sure how many  
variables are passed, but I would guess on the order of 24.


I turned off all the I/O I knew of to reduce disk latency.

1 node:  8 processes:  54 minutes
1 node: 16 processes: 40 minutes (oversubscribed)
2 nodes, 16 processes:29 minutes

So, oversubscribing was faster (in this case), but it didn't double  
the speed.  Certainly spreading the load to another node was much  
faster.


I haven't had a chance to implement Warner's suggestion of turning  
hyperthreading off to see what affect that has on the speed.


Cheers,  Jody
___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users


___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] 2 to 1 oversubscription

[Klymak Jody]

Hi Robert,

I got inspired by your question to run a few more tests.  They are  
crude, and I don't have actual cpu timing information because of a  
library mismatch.  However:


Setup:
Xserve, 2x2.26 GHz Quad-core Intel Xeon
6.0 Gb memory 1067 MHz DDR3
Mac OS X 10.5.6

Nodes are connected with a dedicated gigabit ethernet switch.

I'm running the MITgcm, a nonhydrostatic global circulation model.   
The grid size is modest: 10x150x1600, so bear that in mind.  Message  
passing is on the dimension that is 150x10, and typically is 3 grid  
cells in either direction.  I'm not sure how many variables are  
passed, but I would guess on the order of 24.


I turned off all the I/O I knew of to reduce disk latency.

1 node:  8 processes:  54 minutes
1 node: 16 processes: 40 minutes (oversubscribed)
2 nodes, 16 processes:29 minutes

So, oversubscribing was faster (in this case), but it didn't double  
the speed.  Certainly spreading the load to another node was much  
faster.


I haven't had a chance to implement Warner's suggestion of turning  
hyperthreading off to see what affect that has on the speed.


Cheers,  Jody

Re: [OMPI users] Xgrid and choosing agents...

[Klymak Jody]

Hi Ralph,

On 12-Jul-09, at 4:07 AM, Ralph Castain wrote:

Assuming that Scoreboard is appropriately licensed (i.e., is not  
licensed under GPL, but preferably something like FreeBSD), and that  
it has an accessible API, then we can link against it when in that  
environment and interact any way we desire - including asking  
Scoreboard for its recommended list of nodes.




The info I have is all from http://www.macresearch.org/node/4654.

To use scoreboard you pass a run-time argument to the xgrid command  
using "-art" and "-artid".


xgrid -job submit -art scorescript.pl -artid score MyJob

The script can return any score calculated from any of the machine  
attributes, i.e. you can send the job to machines with more than 2 Gb  
of memory, or that are faster than 2GHz.  So its a pretty nice  
feature.  I don't think there is any way to manually query the nodes.


Unfortunately, I'm not sure if it is possible to do this via the API  
which does not look to have any hooks into Scorecard:


http://developer.apple.com/documentation/Performance/Conceptual/XgridDeveloper/index.html#/ 
/apple_ref/doc/framework/XgridFoundation_reference


Curiously, looking at the API, I further get the impression it has no  
concept of identifying the nodes.  The xgrid controller assigns them,  
and there does not seem to be a mechanism for the user to choose.  So  
until the API is improved to account for the Scoreboard capability I  
guess this can't be accessed via openMPI.


Thanks a lot for thinking about this.  As I said, I have a solution  
now that works just great, so this is all just for curiousity's sake  
at this point.


Cheers,  Jody

Re: [OMPI users] Xgrid and choosing agents...

[Klymak Jody]

Hi Vitorio,

On 11-Jul-09, at 8:40 PM, Luis Vitorio Cargnini wrote:


did you saw that, maybe, just maybe using:
xserve01.local slots=8 max-slots=8
xserve02.local slots=8 max-slots=8
xserve03.local slots=8 max-slots=8
xserve04.local slots=8 max-slots=8

it can set the number of process specifically for each node, the  
"slots" does this setting the configuration of slots per each node,  
try it with the old conf of Xgrid and also test with your new Xgrid  
conf.


As per Ralph's message, the xgrid launcher ignores --hostfiles...   
Further, "max_slots=2" is the same as "slots=2 max_slots=2" according  
to the man page.


Xgrid does have a somewhat convoluted, and poorly documented, method  
of directing jobs to specified machines.  Its called Scoreboard and it  
allows the scheduler to query each machine with a script that gathers  
info about the machine and compute a "score".  Nodes with the highest  
score get the job.  However, how one would implement that using  
openMPI is unclear to me.  Does openMPI have the capability of passing  
arbitrary arguments to the resource managers?


Thanks,  Jody



Regards.
Vitorio.


Le 09-07-11 à 18:11, Klymak Jody a écrit :

If anyone else is using xgrid, there is a mechanism to limit the  
processes per machine:


sudo defaults write /Library/Preferences/com.apple.xgrid.agent  
MaximumTaskCount 8


on each of the nodes and then restarting xgrid tells the controller  
to only send 8 processes to that node.  For now that is fine  
solution for my need.  I'll try and figure out how to specify hosts  
via xgrid and get back to the list...


Thanks for everyone's help,

Cheers, Jody

On 11-Jul-09, at 12:42 PM, Ralph Castain wrote:

Looking at the code, you are correct in that the Xgrid launcher is  
ignoring hostfiles. I'll have to look at it to determine how to  
correct that situation - I didn't write that code, nor do I have a  
way to test any changes I might make to it.


For now, though, if you add --bynode to your command line, you  
should get the layout you want. I'm not sure you'll get the rank  
layout you'll want, though...or if that is important to what you  
are doing.


Ralph

On Jul 11, 2009, at 1:18 PM, Klymak Jody wrote:


Hi Vitorio,

Thanks for getting back to me!  My hostfile is

xserve01.local max-slots=8
xserve02.local max-slots=8
xserve03.local max-slots=8
xserve04.local max-slots=8

I've now checked, and this seems to work fine just using ssh.   
i.e. if I turn off the Xgrid queue manager I can submit jobs  
manually to the appropriate nodes using --hosts.


However, I'd really like to use Xgrid as my queue manager as it  
is already set up (though I'll happily take hints on how to set  
up other queue managers on an OS X cluster).


So you have 4 nodes each one with 2 processors, each processor 4- 
core - quad-core.

So you have capacity for 32 process in parallel.


The new Xeon chips designate 2-processes per core, though at a  
reduced clock rate.  This means that Xgrid believes I have 16  
processors/node.  For large jobs I expect that to be useful, but  
for my more modest jobs I really only want 8 processes/node.


It appears that the default way xgrid assigns the jobs is to fill  
all 16 slots on one node before moving to the next.  OpenMPI  
doesn't appear to look at the hostfile configuration when using  
Xgrid, so it makes it hard for me to deprecate this behaviour.


Thanks,  Jody



I think that only using the hostfile is enough is how I use. If  
you to specify a specific host or a different sequence, the  
mpirun will obey the host sequence in your hostfile to start the  
process, also can you put how you configured your host files ?  
I'm asking this because you should have something like:

# This is an example hostfile. Comments begin with
# #
# The following node is a single processor machine:
foo.example.com
# The following node is a dual-processor machine:
bar.example.com slots=2
# The following node is a quad-processor machine, and we  
absolutely

# want to disallow over-subscribing it:
yow.example.com slots=4 max-slots=4
so in your case like mine you should have something like:
your.hostname.domain slots=8 max-slots=8 # for each node

I hope this will help you.
Regards.
Vitorio.


Le 09-07-11 à 10:56, Klymak Jody a écrit :


Hi all,

Sorry in advance if these are naive questions - I'm not  
experienced in running a grid...


I'm using openMPI on 4  duo Quad-core Xeon xserves.  The 8  
cores mimic 16 cores and show up in xgrid as each agent having  
16 processors.  However, the processing speed goes down as the  
used processors exceeds 8, so if possible I'd prefer to not  
have more than 8 processors working on each machine at a time.


Unfortunately, if I submit a 16-processor job to xgrid it all  
goes to "xserve03".  Or even worse, it does so if I submit two  
separate 8-processor jobs.  Is there anyway to steer jobs to  
less-busy agents?


I tried making a hostfile

Re: [OMPI users] 2 to 1 oversubscription

[Klymak Jody]

Hi Robert,

I ran some very crude tests and found that things slowed down once you  
got over 8 cores at a time.  However, they didn't slow down by 50% if  
you went to 16 processes.  Sadly, the tests were so crude, I did not  
keep good notes (it appears).


I'm running a gcm, so my benchmarks may not be very useful to most  
folks.  If there was an easy-to-compile benhmark that I could run on  
my cluster, I'd be curious what the results are too.


Thanks,  Jody

On 11-Jul-09, at 2:16 PM, Robert Kubrick wrote:

The Open MPI FAQ recommends not to oversubscribe the available cores  
for best performances, but is this still true? The new Nehalem  
processors are built to run 2 threads on each core. On a 8 sockets  
systems, that sums up to 128 threads that Intel claims can be run  
without significant performance degradation. I guess the last word  
is to those who have tried to run some benchmarks and applications  
on the new Intel processors. Any experience to share?


http://www.open-mpi.org/faq/?category=running#oversubscribing
http://en.wikipedia.org/wiki/Simultaneous_multithreading
http://communities.intel.com/community/openportit/server/blog/2009/06/11/nehalem-ex-brings-new-economics-to-scalable-systems
___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] Xgrid and choosing agents...

[Klymak Jody]

If anyone else is using xgrid, there is a mechanism to limit the  
processes per machine:


sudo defaults write /Library/Preferences/com.apple.xgrid.agent  
MaximumTaskCount 8


on each of the nodes and then restarting xgrid tells the controller to  
only send 8 processes to that node.  For now that is fine solution for  
my need.  I'll try and figure out how to specify hosts via xgrid and  
get back to the list...


Thanks for everyone's help,

Cheers, Jody

On 11-Jul-09, at 12:42 PM, Ralph Castain wrote:

Looking at the code, you are correct in that the Xgrid launcher is  
ignoring hostfiles. I'll have to look at it to determine how to  
correct that situation - I didn't write that code, nor do I have a  
way to test any changes I might make to it.


For now, though, if you add --bynode to your command line, you  
should get the layout you want. I'm not sure you'll get the rank  
layout you'll want, though...or if that is important to what you are  
doing.


Ralph

On Jul 11, 2009, at 1:18 PM, Klymak Jody wrote:


Hi Vitorio,

Thanks for getting back to me!  My hostfile is

xserve01.local max-slots=8
xserve02.local max-slots=8
xserve03.local max-slots=8
xserve04.local max-slots=8

I've now checked, and this seems to work fine just using ssh.  i.e.  
if I turn off the Xgrid queue manager I can submit jobs manually to  
the appropriate nodes using --hosts.


However, I'd really like to use Xgrid as my queue manager as it is  
already set up (though I'll happily take hints on how to set up  
other queue managers on an OS X cluster).


So you have 4 nodes each one with 2 processors, each processor 4- 
core - quad-core.

So you have capacity for 32 process in parallel.


The new Xeon chips designate 2-processes per core, though at a  
reduced clock rate.  This means that Xgrid believes I have 16  
processors/node.  For large jobs I expect that to be useful, but  
for my more modest jobs I really only want 8 processes/node.


It appears that the default way xgrid assigns the jobs is to fill  
all 16 slots on one node before moving to the next.  OpenMPI  
doesn't appear to look at the hostfile configuration when using  
Xgrid, so it makes it hard for me to deprecate this behaviour.


Thanks,  Jody



I think that only using the hostfile is enough is how I use. If  
you to specify a specific host or a different sequence, the mpirun  
will obey the host sequence in your hostfile to start the process,  
also can you put how you configured your host files ? I'm asking  
this because you should have something like:

# This is an example hostfile. Comments begin with
# #
# The following node is a single processor machine:
foo.example.com
# The following node is a dual-processor machine:
bar.example.com slots=2
# The following node is a quad-processor machine, and we absolutely
# want to disallow over-subscribing it:
yow.example.com slots=4 max-slots=4
so in your case like mine you should have something like:
your.hostname.domain slots=8 max-slots=8 # for each node

I hope this will help you.
Regards.
Vitorio.


Le 09-07-11 à 10:56, Klymak Jody a écrit :


Hi all,

Sorry in advance if these are naive questions - I'm not  
experienced in running a grid...


I'm using openMPI on 4  duo Quad-core Xeon xserves.  The 8 cores  
mimic 16 cores and show up in xgrid as each agent having 16  
processors.  However, the processing speed goes down as the used  
processors exceeds 8, so if possible I'd prefer to not have more  
than 8 processors working on each machine at a time.


Unfortunately, if I submit a 16-processor job to xgrid it all  
goes to "xserve03".  Or even worse, it does so if I submit two  
separate 8-processor jobs.  Is there anyway to steer jobs to less- 
busy agents?


I tried making a hostfile and then specifying the host, but I get:

/usr/local/openmpi/bin/mpirun -n 8 --hostfile hostfile --host  
xserve01.local ../build/mitgcmuv


Some of the requested hosts are not included in the current  
allocation for the

application:
../build/mitgcmuv
The requested hosts were:
xserve01.local

so I assume --host doesn't work with xgrid?

Is a reasonable alternative to simply not use xgrid and rely on  
ssh?


Thanks,  Jody

--
Jody Klymak
http://web.uvic.ca/~jklymak

___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users


___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users



___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users



___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] Xgrid and choosing agents...

[Klymak Jody]

Hi Vitorio,

Thanks for getting back to me!  My hostfile is

xserve01.local max-slots=8
xserve02.local max-slots=8
xserve03.local max-slots=8
xserve04.local max-slots=8

I've now checked, and this seems to work fine just using ssh.  i.e. if  
I turn off the Xgrid queue manager I can submit jobs manually to the  
appropriate nodes using --hosts.


However, I'd really like to use Xgrid as my queue manager as it is  
already set up (though I'll happily take hints on how to set up other  
queue managers on an OS X cluster).


So you have 4 nodes each one with 2 processors, each processor 4- 
core - quad-core.

So you have capacity for 32 process in parallel.


The new Xeon chips designate 2-processes per core, though at a reduced  
clock rate.  This means that Xgrid believes I have 16 processors/ 
node.  For large jobs I expect that to be useful, but for my more  
modest jobs I really only want 8 processes/node.


It appears that the default way xgrid assigns the jobs is to fill all  
16 slots on one node before moving to the next.  OpenMPI doesn't  
appear to look at the hostfile configuration when using Xgrid, so it  
makes it hard for me to deprecate this behaviour.


Thanks,  Jody



I think that only using the hostfile is enough is how I use. If you  
to specify a specific host or a different sequence, the mpirun will  
obey the host sequence in your hostfile to start the process, also  
can you put how you configured your host files ? I'm asking this  
because you should have something like:

# This is an example hostfile. Comments begin with
# #
# The following node is a single processor machine:
foo.example.com
# The following node is a dual-processor machine:
bar.example.com slots=2
# The following node is a quad-processor machine, and we absolutely
# want to disallow over-subscribing it:
yow.example.com slots=4 max-slots=4
so in your case like mine you should have something like:
your.hostname.domain slots=8 max-slots=8 # for each node

I hope this will help you.
Regards.
Vitorio.


Le 09-07-11 à 10:56, Klymak Jody a écrit :


Hi all,

Sorry in advance if these are naive questions - I'm not experienced  
in running a grid...


I'm using openMPI on 4  duo Quad-core Xeon xserves.  The 8 cores  
mimic 16 cores and show up in xgrid as each agent having 16  
processors.  However, the processing speed goes down as the used  
processors exceeds 8, so if possible I'd prefer to not have more  
than 8 processors working on each machine at a time.


Unfortunately, if I submit a 16-processor job to xgrid it all goes  
to "xserve03".  Or even worse, it does so if I submit two separate  
8-processor jobs.  Is there anyway to steer jobs to less-busy agents?


I tried making a hostfile and then specifying the host, but I get:

/usr/local/openmpi/bin/mpirun -n 8 --hostfile hostfile --host  
xserve01.local ../build/mitgcmuv


Some of the requested hosts are not included in the current  
allocation for the

application:
../build/mitgcmuv
The requested hosts were:
xserve01.local

so I assume --host doesn't work with xgrid?

Is a reasonable alternative to simply not use xgrid and rely on ssh?

Thanks,  Jody

--
Jody Klymak
http://web.uvic.ca/~jklymak

___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users


___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

[OMPI users] Xgrid and choosing agents...

[Klymak Jody]

Hi all,

Sorry in advance if these are naive questions - I'm not experienced in  
running a grid...


I'm using openMPI on 4  duo Quad-core Xeon xserves.  The 8 cores mimic  
16 cores and show up in xgrid as each agent having 16 processors.   
However, the processing speed goes down as the used processors exceeds  
8, so if possible I'd prefer to not have more than 8 processors  
working on each machine at a time.


Unfortunately, if I submit a 16-processor job to xgrid it all goes to  
"xserve03".  Or even worse, it does so if I submit two separate 8- 
processor jobs.  Is there anyway to steer jobs to less-busy agents?


I tried making a hostfile and then specifying the host, but I get:

/usr/local/openmpi/bin/mpirun -n 8 --hostfile hostfile --host  
xserve01.local ../build/mitgcmuv


Some of the requested hosts are not included in the current allocation  
for the

application:
  ../build/mitgcmuv
The requested hosts were:
  xserve01.local

so I assume --host doesn't work with xgrid?

Is a reasonable alternative to simply not use xgrid and rely on ssh?

Thanks,  Jody

--
Jody Klymak
http://web.uvic.ca/~jklymak