Re: [OMPI users] Using OpenMPI / ORTE as cluster aware GNU Parallel

[Mark Santcroos]

Hi Brock, Angel, Reuti,



You might want to look at a tool we developed:
http://radical-cybertools.github.io/radical-pilot/index.html

This was actually one of the drivers for isolating the persistent ORTE DVM 
thats being discussed in this thread.

With RADICAL-Pilot you can use a Python API to launch an ORTE DVM on a 
computational resource and then run tasks on top of that.

Happy to answer questions off-list.



Regards,

Mark
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] resolution of MPI_Wtime

[Mark Santcroos]

> On 05 Apr 2016, at 16:46 , Aurélien Bouteiller  wrote:
> Open MPI uses clock_gettime when it is available, and defaults to 
> gettimeofday only when this better option can't be found. Check that your 
> system has clock_gettime and the resolution of this timer. 

Depending on what you mean, I don't this is universally true, as I recently ran 
into this issue in another part of the code.
AFAIK, some parts of the code have a macro that check for clock_gettime and use 
it, but in other places there is just direct usage of gettimeofday.

If you commented on MPI_Wtime specifically, then I don't know :-)

Mark

Re: [OMPI users] Orted path with module manager on cluster

[Mark Santcroos]

> On 03 Mar 2016, at 23:22 , Davide Vanzo  wrote:
> I have built OpenMPI 1.10.2 with RoCE network support on our test cluster. On 
> the cluster we use lmod to manage paths to different versions of softwares. 
> The problem I have is that I receive the "orted: command not found" message 
> given that the path to the orted binary is not exported to the other nodes 
> where my run is launched via a non-interactive ssh connection. I temporarily 
> solved the problem by exporting PATH with the correct path to orted on my 
> .bashrc file but this is not obviously a solution.
> Any idea how I can fix it?

Did you configure with '--enable-orterun-prefix-by-default' ?

Re: [OMPI users] how to benchmark a server with openmpi?

[Mark Santcroos]

Another canonical benchmarking suite can be found at 
http://www.nas.nasa.gov/publications/npb.html

> On 24 Jan 2016, at 20:51 , Ibrahim Ikhlawi  wrote:
> 
> Thanks for reply.
> 
> But I want to have an imagination about the behaviour of my server. Therefore 
> I need an Code which I can run it on my server.
> Could anyone give me an example for any code? My last code was a simple 
> example and not enough to get an imagination.
> 
> Thanx in advance
> Date: Sun, 24 Jan 2016 10:21:07 -0500
> From: esal...@gmail.com
> To: us...@open-mpi.org
> Subject: Re: [OMPI users] how to benchmark a server with openmpi?
> 
> We've looked at performance in detail with regard to OpenMPI Java for large 
> scale real data analytics. Here's a paper still in submission that identifies 
> 5 rules you'd find useful to get good performance. It talks about how the 
> number of processes affect performance as well. Tests were done up 3072 cores 
> on a large Intel Haswell HPC cluster
> 
> https://www.researchgate.net/publication/291695527_SPIDAL_Java_High_Performance_Data_Analytics_with_Java_and_MPI_on_Large_Multicore_HPC_Clusters
> 
> Thank you,
> Saliya
> 
> On Sun, Jan 24, 2016 at 9:41 AM, Nick Papior  wrote:
> *All* codes scale differently. 
> So you should do these tests with your own code, and not a different code 
> (such as MM).
> 
> 
> 2016-01-24 15:38 GMT+01:00 Ibrahim Ikhlawi :
> 
> 
> Hallo,
> 
> I am working on a server and run java codes with OpenMPI. I want to know 
> which number of process is the fastest to run my code with?
> For this reason I wrote a code that multiply two matrices but the differences 
> between the results is not significant. 
> 
> Therefore I want to know how can I benchmark my server? Or is there any 
> example which I can run many times on a different number of processes, so 
> that I can see which number of process is the best?
> 
> Or is there examples with results which I can compare with my results?
> 
> Thanx in advance
> Ibrahim
> 
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2016/01/28352.php
> 
> 
> 
> -- 
> Kind regards Nick
> 
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2016/01/28353.php
> 
> 
> 
> -- 
> Saliya Ekanayake
> Ph.D. Candidate | Research Assistant
> School of Informatics and Computing | Digital Science Center
> Indiana University, Bloomington
> Cell 812-391-4914
> http://saliya.org
> 
> ___ users mailing list 
> us...@open-mpi.org Subscription: 
> http://www.open-mpi.org/mailman/listinfo.cgi/users Link to this post: 
> http://www.open-mpi.org/community/lists/users/2016/01/28354.php
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2016/01/28355.php

Re: [OMPI users] OpenMPI 1.8.5 build question

[Mark Santcroos]

> On 23 Sep 2015, at 13:49 , Kumar, Sudhir  wrote:
> I have a version of OpenMPI 1.8.5 installed. Is there any way of knowing, 
> with which version of gcc it was compiled with.

ompi_info |grep -i compiler

Re: [OMPI users] Building OpenMPI 1.8.7 on XC30

[Mark Santcroos]

Hi Erik,

> On 29 Jul 2015, at 3:35 , Erik Schnetter  wrote:
> I was able to build openmpi-v2.x-dev-96-g918650a without problems on Edison, 
> and also on other systems.

And does it also work as expected after you have build it? :-)

Thanks

Mark

Re: [OMPI users] Building OpenMPI 1.8.7 on XC30

[Mark Santcroos]

Sorry, dont know about that, would have to pass that on to others.

> On 26 Jul 2015, at 5:25 , Erik Schnetter <schnet...@gmail.com> wrote:
> 
> Mark
> 
> No, it doesn't need to be 1.8.7.
> 
> I just tried v2.x-dev-96-g918650a. This leads to run-time warnings on OS X; I 
> see messages such as
> 
> [warn] select: Bad file descriptor
> 
> Are these important? If not, how can I suppress them?
> 
> -erik
> 
> 
> On Sat, Jul 25, 2015 at 7:49 AM, Mark Santcroos <mark.santcr...@rutgers.edu> 
> wrote:
> Hi Erik,
> 
> Do you really want 1.8.7, otherwise you might want to give latest master a 
> try. Other including myself had more luck with that on Cray's, including 
> Edison.
> 
> Mark
> 
> > On 25 Jul 2015, at 1:35 , Erik Schnetter <schnet...@gmail.com> wrote:
> >
> > I want to build OpenMPI 1.8.7 on a Cray XC30 (Edison at NERSC). I've tried 
> > various configuration options, but I am always encountering either OpenMPI 
> > build errors, application build errors, or run-time errors.
> >
> > I'm currently looking at 
> > <http://www.open-mpi.org/community/lists/users/2015/06/27230.php>, which 
> > seems to describe my case. I'm now configuring OpenMPI without any options, 
> > except setting compilers to clang/gfortran and pointing it to a self-built 
> > hwloc. For completeness, here are my configure options as recorded by 
> > config.status:
> >
> > '/project/projectdirs/m152/schnette/edison/software/src/openmpi-1.8.7/src/openmpi-1.8.7/configure'
> >   
> > '--prefix=/project/projectdirs/m152/schnette/edison/software/openmpi-1.8.7' 
> > '--with-hwloc=/project/projectdirs/m152/schnette/edison/software/hwloc-1.11.0'
> >  '--disable-vt' 
> > 'CC=/project/projectdirs/m152/schnette/edison/software/llvm-3.6.2/bin/wrap-clang'
> >  
> > 'CXX=/project/projectdirs/m152/schnette/edison/software/llvm-3.6.2/bin/wrap-clang++'
> >  
> > 'FC=/project/projectdirs/m152/schnette/edison/software/gcc-5.2.0/bin/wrap-gfortran'
> >  'CFLAGS=-I/opt/ofed/include 
> > -I/project/projectdirs/m152/schnette/edison/software/hwloc-1.11.0/include' 
> > 'CXXFLAGS=-I/opt/ofed/include 
> > -I/project/projectdirs/m152/schnette/edison/software/hwloc-1.11.0/include' 
> > 'LDFLAGS=-L/opt/ofed/lib64 
> > -L/project/projectdirs/m152/schnette/edison/software/hwloc-1.11.0/lib 
> > -Wl,-rpath,/project/projectdirs/m152/schnette/edison/software/hwloc-1.11.0/lib'
> >  'LIBS=-lhwloc -lpthread -lpthread' 
> > '--with-wrapper-ldflags=-L/project/projectdirs/
>  m152/schnette/edison/software/hwloc-1.11.0/lib 
> -Wl,-rpath,/project/projectdirs/m152/schnette/edison/software/hwloc-1.11.0/lib'
>  '--with-wrapper-libs=-lhwloc -lpthread'
> >
> > This builds and installs fine, and works when running on a single node. 
> > However, multi-node runs are stalling: The queue starts the job, but mpirun 
> > produces no output. The "-v" option to mpirun doesn't help.
> >
> > When I use aprun instead of mpirun to start my application, then all 
> > processes think they are rank 0.
> >
> > Do you have any pointers for how to debug this?
> >
> > -erik
> >
> > --
> > Erik Schnetter <schnet...@gmail.com> 
> > http://www.perimeterinstitute.ca/personal/eschnetter/
> > ___
> > users mailing list
> > us...@open-mpi.org
> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> > Link to this post: 
> > http://www.open-mpi.org/community/lists/users/2015/07/27324.php
> 
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2015/07/27327.php
> 
> 
> 
> -- 
> Erik Schnetter <schnet...@gmail.com> 
> http://www.perimeterinstitute.ca/personal/eschnetter/
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2015/07/27329.php

Re: [OMPI users] Building OpenMPI 1.8.7 on XC30

[Mark Santcroos]

Hi Erik,

Do you really want 1.8.7, otherwise you might want to give latest master a try. 
Other including myself had more luck with that on Cray's, including Edison.

Mark

> On 25 Jul 2015, at 1:35 , Erik Schnetter  wrote:
> 
> I want to build OpenMPI 1.8.7 on a Cray XC30 (Edison at NERSC). I've tried 
> various configuration options, but I am always encountering either OpenMPI 
> build errors, application build errors, or run-time errors.
> 
> I'm currently looking at 
> , which 
> seems to describe my case. I'm now configuring OpenMPI without any options, 
> except setting compilers to clang/gfortran and pointing it to a self-built 
> hwloc. For completeness, here are my configure options as recorded by 
> config.status:
> 
> '/project/projectdirs/m152/schnette/edison/software/src/openmpi-1.8.7/src/openmpi-1.8.7/configure'
>   '--prefix=/project/projectdirs/m152/schnette/edison/software/openmpi-1.8.7' 
> '--with-hwloc=/project/projectdirs/m152/schnette/edison/software/hwloc-1.11.0'
>  '--disable-vt' 
> 'CC=/project/projectdirs/m152/schnette/edison/software/llvm-3.6.2/bin/wrap-clang'
>  
> 'CXX=/project/projectdirs/m152/schnette/edison/software/llvm-3.6.2/bin/wrap-clang++'
>  
> 'FC=/project/projectdirs/m152/schnette/edison/software/gcc-5.2.0/bin/wrap-gfortran'
>  'CFLAGS=-I/opt/ofed/include 
> -I/project/projectdirs/m152/schnette/edison/software/hwloc-1.11.0/include' 
> 'CXXFLAGS=-I/opt/ofed/include 
> -I/project/projectdirs/m152/schnette/edison/software/hwloc-1.11.0/include' 
> 'LDFLAGS=-L/opt/ofed/lib64 
> -L/project/projectdirs/m152/schnette/edison/software/hwloc-1.11.0/lib 
> -Wl,-rpath,/project/projectdirs/m152/schnette/edison/software/hwloc-1.11.0/lib'
>  'LIBS=-lhwloc -lpthread -lpthread' 
> '--with-wrapper-ldflags=-L/project/projectdirs/m152/schnette/edison/software/hwloc-1.11.0/lib
>  
> -Wl,-rpath,/project/projectdirs/m152/schnette/edison/software/hwloc-1.11.0/lib'
>  '--with-wrapper-libs=-lhwloc -lpthread'
> 
> This builds and installs fine, and works when running on a single node. 
> However, multi-node runs are stalling: The queue starts the job, but mpirun 
> produces no output. The "-v" option to mpirun doesn't help.
> 
> When I use aprun instead of mpirun to start my application, then all 
> processes think they are rank 0.
> 
> Do you have any pointers for how to debug this?
> 
> -erik
> 
> -- 
> Erik Schnetter  
> http://www.perimeterinstitute.ca/personal/eschnetter/
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2015/07/27324.php

Re: [OMPI users] open mpi on blue waters

[Mark Santcroos]

> On 26 Mar 2015, at 16:01 , Ralph Castain <r...@open-mpi.org> wrote:
> 
>> 
>> On Mar 26, 2015, at 1:33 AM, Mark Santcroos <mark.santcr...@rutgers.edu> 
>> wrote:
>> 
>> Hi guys,
>> 
>> Thanks for the follow-up.
>> 
>> It appears that you are ruling out that Munge is required because the system 
>> runs TORQUE, but as far as I can see Munge is/can be used by both SLURM and 
>> TORQUE.
>> (http://docs.adaptivecomputing.com/torque/4-0-2/Content/topics/1-installConfig/serverConfig.htm#usingMUNGEAuth)
> 
> Not really ruling it out, Mark, but I didn’t consider it likely because then 
> munge would indeed have to be on the compute nodes. Otherwise, the sister 
> moms wouldn’t be able to authenticate back to the mom on the IO node.
> 
> To be clear, I’m not 100% sure what is using munge on the IO node. My real 
> point was only that there are other subsystems that can use such security 
> services, and that those subsystems might not extend into the compute node 
> itself. Thus, the need to work in multiple security domains is going to exist 
> into the future.

Right, I think I'm clear on the issue now :-)

Re: [OMPI users] open mpi on blue waters

[Mark Santcroos]

Hi Ralph,

> On 25 Mar 2015, at 21:59 , Mark Santcroos <mark.santcr...@rutgers.edu> wrote:
>> Anyway, see if this fixes the problem.
>> 
>> https://github.com/open-mpi/ompi/pull/497

Can confirm the fallback works now without setting explicitly to basic (with 
the merged changes).

Thanks!

Mark

Re: [OMPI users] open mpi on blue waters

[Mark Santcroos]

Hi guys,

Thanks for the follow-up.

It appears that you are ruling out that Munge is required because the system 
runs TORQUE, but as far as I can see Munge is/can be used by both SLURM and 
TORQUE.
(http://docs.adaptivecomputing.com/torque/4-0-2/Content/topics/1-installConfig/serverConfig.htm#usingMUNGEAuth)

If I misunderstood the drift, please ignore ;-)

Mark


> On 26 Mar 2015, at 5:38 , Gilles Gouaillardet  
> wrote:
> 
> On 2015/03/26 13:00, Ralph Castain wrote:
>> Well, I did some digging around, and this PR looks like the right solution.
> ok then :-)
> 
> following stuff is not directly related to ompi, but you might want to
> comment on that anyway ...
>> Second, the running of munge on the IO nodes is not only okay but required 
>> by Luster.
> this is the first time i hear that.
> i googled "lustre munge" and could not find any relevant info about that.
> is this a future feature of Lustre ?
> as far as i am concerned, only Lustre MDS need a "unix" authentication
> system
> (ldap, nis, /etc/passwd, ...) and munge does not provide this service.
>> Future systems are increasingly going to run the user’s job script 
>> (including mpirun) on the IO nodes as this (a) frees up the login node for 
>> interactive editing, and (b) avoids the jitter introduced by running the job 
>> script on the same node as application procs, or wasting a compute node to 
>> just run the job script.
> that does make sense not to run the script on a compute node.
> but once again i am surprised ... as far as i am concerned, lustre IO
> nodes (MDS and/or OSS) do not mount the filesystem
> (i mean you cannot access the filesystem as if you were on a lustre client).
> of course, you can write your script so it does not require any access
> to the lustre filesystem, but that sounds like a lot of pain
> for a small benefit.
> /* that is specific to Lustre. GPFS for example can access the
> filesystem from an IO node */
> 
> Cheers,
> 
> Gilles
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2015/03/26539.php

Re: [OMPI users] open mpi on blue waters

[Mark Santcroos]

Hi Ralph,

> On 25 Mar 2015, at 21:25 , Ralph Castain  wrote:
> I think I have this resolved,
> though that I still suspect their is something wrong on that system. You 
> shouldn’t have some nodes running munge and others not running it.

For completeness, it's not "some" nodes, its the MOM (service) nodes that run 
it, and the compute nodes don't.
I don't know munge well enough to judge whether it makes sense to have it there 
only and not on the compute nodes?

> I wonder if someone was experimenting and started munge on some of the nodes, 
> and forgot to turn it off afterwards??

If the answer to my request for clarification is along the lines of "No!", then 
I can ask the admins whats up.

> Anyway, see if this fixes the problem.
> 
> https://github.com/open-mpi/ompi/pull/497

Will get back to you later how that works for me.

Thanks

Mark

Re: [OMPI users] open mpi on blue waters

[Mark Santcroos]

> On 25 Mar 2015, at 17:39 , Ralph Castain  wrote:
> Not surprising - I’m surprised to find munge on the mom’s node anyway given 
> that you are using Torque.
> 
> I have to finish something else first, and it sounds like you aren’t blocked 
> at the moment. I’ll provide a patch for you to try later, if you’re willing.

Right. Don't worry, Im not blocked indeed, and just wanted to offer some 
debugging assistance as a courtesy.

Re: [OMPI users] open mpi on blue waters

[Mark Santcroos]

Ok.

FYI:
> aprun munge -n
munge: Error: Unable to access "/var/run/munge/munge.socket.2": No such file or 
directory
Application 23792792 exit codes: 6
Application 23792792 resources: utime ~0s, stime ~1s, Rss ~27304, inblocks ~35, 
outblocks ~58


> On 25 Mar 2015, at 17:29 , Ralph Castain <r...@open-mpi.org> wrote:
> 
> Yeah, what’s happening is that mpirun is picking one security mechanism for 
> authenticating connections, but the backend daemons are picking another, and 
> hence we get the conflict. The weird thing here is that you usually don’t see 
> this kind of mismatch for the very reason you are hitting - it becomes 
> difficult to resolve authentications.
> 
> Let me ponder a bit. We can resolve it easily enough, but I want to ensure we 
> don’t do it by creating a security hole.
> 
>> On Mar 25, 2015, at 9:25 AM, Mark Santcroos <mark.santcr...@rutgers.edu> 
>> wrote:
>> 
>> 
>>> On 25 Mar 2015, at 17:06 , Ralph Castain <r...@open-mpi.org> wrote:
>>> 
>>> OHO! You have munge running on the head node, but not on the backends!
>> 
>> Ok, so I now know that munge is ... :)
>> 
>> It's running on the MOM node (not on the head node):
>> 
>> daemon   18800  0.0  0.0 118476  3212 ?Sl   01:27   0:00 
>> /usr/sbin/munged --key-file /opt/munge/munge.key --num-threads 8
>> 
>> Any tests you would like me to perform?
>> 
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/users/2015/03/26524.php
> 
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2015/03/26525.php

Re: [OMPI users] open mpi on blue waters

[Mark Santcroos]

> On 25 Mar 2015, at 17:06 , Ralph Castain  wrote:
> 
> OHO! You have munge running on the head node, but not on the backends!

Ok, so I now know that munge is ... :)

It's running on the MOM node (not on the head node):

daemon   18800  0.0  0.0 118476  3212 ?Sl   01:27   0:00 
/usr/sbin/munged --key-file /opt/munge/munge.key --num-threads 8

Any tests you would like me to perform?

Re: [OMPI users] open mpi on blue waters

[Mark Santcroos]

> On 25 Mar 2015, at 17:06 , Ralph Castain  wrote:
> OHO! You have munge running on the head node, but not on the backends!

Im all for munching, but what does that mean? ;-)

Is that something actively running or do you mean library available or such?

> Okay, all you have to do is set the MCA param “sec” to “basic” in your 
> environment, or add “-mca sec basic” on your cmd line

Aye! Works beautifully now! Pfew!

Re: [OMPI users] open mpi on blue waters

[Mark Santcroos]

> On 25 Mar 2015, at 16:52 , Ralph Castain  wrote:
> 
> Hmmm…okay, sorry to keep drilling down here, but let’s try adding “-mca 
> sec_base_verbose 100” now


> /u/sciteam/marksant/openmpi/installation/bin/mpirun -mca oob_base_verbose 100 
> -mca sec_base_verbose 100 ./a.out 
[nid25257:09727] mca: base: components_register: registering sec components
[nid25257:09727] mca: base: components_register: found loaded component munge
[nid25257:09727] mca: base: components_register: component munge has no 
register or open function
[nid25257:09727] mca: base: components_register: found loaded component basic
[nid25257:09727] mca: base: components_register: component basic has no 
register or open function
[nid25257:09727] mca: base: components_open: opening sec components
[nid25257:09727] mca: base: components_open: found loaded component munge
[nid25257:09727] mca: base: components_open: component munge open function 
successful
[nid25257:09727] mca: base: components_open: found loaded component basic
[nid25257:09727] mca: base: components_open: component basic open function 
successful
[nid25257:09727] mca:sec:select: checking available component munge
[nid25257:09727] mca:sec:select: Querying component [munge]
[nid25257:09727] sec: munge init
[nid25257:09727] mca:sec:select: checking available component basic
[nid25257:09727] mca:sec:select: Querying component [basic]
[nid25257:09727] mca: base: components_register: registering oob components
[nid25257:09727] mca: base: components_register: found loaded component usock
[nid25257:09727] mca: base: components_register: component usock register 
function successful
[nid25257:09727] mca: base: components_register: found loaded component alps
[nid25257:09727] mca: base: components_register: component alps register 
function successful
[nid25257:09727] mca: base: components_register: found loaded component ud
[nid25257:09727] mca: base: components_register: component ud register function 
successful
[nid25257:09727] mca: base: components_register: found loaded component tcp
[nid25257:09727] mca: base: components_register: component tcp register 
function successful
[nid25257:09727] mca: base: components_open: opening oob components
[nid25257:09727] mca: base: components_open: found loaded component usock
[nid25257:09727] mca: base: components_open: component usock open function 
successful
[nid25257:09727] mca: base: components_open: found loaded component alps
[nid25257:09727] mca: base: components_open: component alps open function 
successful
[nid25257:09727] mca: base: components_open: found loaded component ud
[nid25257:09727] mca: base: components_open: component ud open function 
successful
[nid25257:09727] mca: base: components_open: found loaded component tcp
[nid25257:09727] mca: base: components_open: component tcp open function 
successful
[nid25257:09727] mca:oob:select: checking available component usock
[nid25257:09727] mca:oob:select: Querying component [usock]
[nid25257:09727] oob:usock: component_available called
[nid25257:09727] [[9128,0],0] USOCK STARTUP
[nid25257:09727] SUNPATH: 
/var/tmp/openmpi-sessions-45504@nid25257_0/9128/0/usock
[nid25257:09727] [[9128,0],0] START USOCK LISTENING ON 
/var/tmp/openmpi-sessions-45504@nid25257_0/9128/0/usock
[nid25257:09727] mca:oob:select: Adding component to end
[nid25257:09727] mca:oob:select: checking available component alps
[nid25257:09727] mca:oob:select: Querying component [alps]
[nid25257:09727] mca:oob:select: Skipping component [alps] - no available 
interfaces
[nid25257:09727] mca:oob:select: checking available component ud
[nid25257:09727] mca:oob:select: Querying component [ud]
[nid25257:09727] oob:ud: component_available called
[nid25257:09727] [[9128,0],0] oob:ud:component_init no devices found
[nid25257:09727] mca:oob:select: Skipping component [ud] - failed to startup
[nid25257:09727] mca:oob:select: checking available component tcp
[nid25257:09727] mca:oob:select: Querying component [tcp]
[nid25257:09727] oob:tcp: component_available called
[nid25257:09727] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4
[nid25257:09727] [[9128,0],0] oob:tcp:init rejecting loopback interface lo
[nid25257:09727] WORKING INTERFACE 2 KERNEL INDEX 1 FAMILY: V4
[nid25257:09727] [[9128,0],0] oob:tcp:init rejecting loopback interface lo
[nid25257:09727] WORKING INTERFACE 3 KERNEL INDEX 3 FAMILY: V4
[nid25257:09727] [[9128,0],0] oob:tcp:init adding 10.128.99.112 to our list of 
V4 connections
[nid25257:09727] [[9128,0],0] TCP STARTUP
[nid25257:09727] [[9128,0],0] attempting to bind to IPv4 port 0
[nid25257:09727] [[9128,0],0] assigned IPv4 port 60755
[nid25257:09727] mca:oob:select: Adding component to end
[nid25257:09727] mca:oob:select: Found 2 active transports
[nid25257:09727] [[9128,0],0] mca_oob_tcp_listen_thread: new connection: (16, 
0) 10.128.69.144:41619
[nid25257:09727] [[9128,0],0] connection_handler: working connection (16, 2) 
10.128.69.144:41619
[nid25257:09727] [[9128,0],0] 

Re: [OMPI users] open mpi on blue waters

[Mark Santcroos]

],0] waiting for connect ack from UNKNOWN
[nid25257:09350] [[8913,0],0] connect ack received from UNKNOWN
[nid25257:09350] [[8913,0],0] connect-ack recvd from UNKNOWN
[nid25257:09350] [[8913,0],0] mca_oob_tcp_recv_connect: connection from new peer
[nid25257:09350] [[8913,0],0] connect-ack header from [[8913,0],1] is okay
[nid25257:09350] [[8913,0],0] waiting for connect ack from [[8913,0],1]
[nid25257:09350] [[8913,0],0] connect ack received from [[8913,0],1]
[nid25257:09350] [[8913,0],0] connect-ack version from [[8913,0],1] matches ours
[nid25257:09350] [[8913,0],0] ORTE_ERROR_LOG: Authentication failed in file 
../../../../../orte/mca/oob/tcp/oob_tcp_connection.c at line 803


> On 25 Mar 2015, at 16:49 , Ralph Castain <r...@open-mpi.org> wrote:
> 
> Hmmm…well, it will generate some output, so keep the system down to two nodes 
> if you can just to minimize the chatter. Add “-mca oob_base_verbose 100” to 
> your cmd line
> 
>> On Mar 25, 2015, at 8:45 AM, Mark Santcroos <mark.santcr...@rutgers.edu> 
>> wrote:
>> 
>> Hi Ralph,
>> 
>> There is no OMPI in system space and PATH and LD_LIBRARY_PATH look good.
>> Any suggestion on how to get more relevant debugging info above the table?
>> 
>> Thanks
>> 
>> Mark
>> 
>> 
>>> On 25 Mar 2015, at 16:33 , Ralph Castain <r...@open-mpi.org> wrote:
>>> 
>>> Hey Mark
>>> 
>>> Your original error flag indicates that you are picking up a connection 
>>> from some proc built against a different OMPI installation. It’s a very 
>>> low-level check that looks for matching version numbers. Not sure who is 
>>> trying to connect, but that is the problem.
>>> 
>>> Check you LD_LIBRARY_PATH
>>> 
>>>> On Mar 25, 2015, at 7:46 AM, Howard Pritchard <hpprit...@gmail.com> wrote:
>>>> 
>>>> turn off the disable getpwuid.
>>>> 
>>>> On Mar 25, 2015 8:14 AM, "Mark Santcroos" <mark.santcr...@rutgers.edu> 
>>>> wrote:
>>>> Hi Howard,
>>>> 
>>>>> On 25 Mar 2015, at 14:58 , Howard Pritchard <hpprit...@gmail.com> wrote:
>>>>> How are you building ompi?
>>>> 
>>>> My configure is rather straightforward:
>>>> ./configure --prefix=$OMPI_PREFIX --disable-getpwuid
>>>> 
>>>> Maybe I got spoiled on Hopper/Edison and I need more explicit 
>>>> configuration on BW ...
>>>> 
>>>>> Also what happens if you use. aprun.
>>>> 
>>>> Not sure if you meant in combination with mpirun or not, so I'll provide 
>>>> both:
>>>> 
>>>>> aprun -n2 ./a.out
>>>> Hello from rank 1, thread 0, on nid16869. (core affinity = 0)
>>>> Hello from rank 0, thread 0, on nid16868. (core affinity = 0)
>>>> After sleep from rank 1, thread 0, on nid16869. (core affinity = 0)
>>>> After sleep from rank 0, thread 0, on nid16868. (core affinity = 0)
>>>> Application 23791589 resources: utime ~0s, stime ~2s, Rss ~27304, inblocks 
>>>> ~13229, outblocks ~66
>>>> 
>>>>> aprun -n2 mpirun ./a.out
>>>> apstat: error opening /ufs/alps_shared/reservations: No such file or 
>>>> directory
>>>> apstat: error opening /ufs/alps_shared/reservations: No such file or 
>>>> directory
>>>> [nid16868:17876] [[699,0],0] ORTE_ERROR_LOG: File open failure in file 
>>>> ../../../../../orte/mca/ras/tm/ras_tm_module.c at line 159
>>>> [nid16868:17876] [[699,0],0] ORTE_ERROR_LOG: File open failure in file 
>>>> ../../../../../orte/mca/ras/tm/ras_tm_module.c at line 85
>>>> [nid16868:17876] [[699,0],0] ORTE_ERROR_LOG: File open failure in file 
>>>> ../../../../orte/mca/ras/base/ras_base_allocate.c at line 190
>>>> [nid16869:17034] [[9344,0],0] ORTE_ERROR_LOG: File open failure in file 
>>>> ../../../../../orte/mca/ras/tm/ras_tm_module.c at line 159
>>>> [nid16869:17034] [[9344,0],0] ORTE_ERROR_LOG: File open failure in file 
>>>> ../../../../../orte/mca/ras/tm/ras_tm_module.c at line 85
>>>> [nid16869:17034] [[9344,0],0] ORTE_ERROR_LOG: File open failure in file 
>>>> ../../../../orte/mca/ras/base/ras_base_allocate.c at line 190
>>>> Application 23791590 exit codes: 1
>>>> Application 23791590 resources: utime ~0s, stime ~2s, Rss ~27304, inblocks 
>>>> ~9596, outblocks ~478
>>>> 
>>>>> I work with ompi on the nersc edison and hopper daily.
>>>> 
>>>> I use Ediso

Re: [OMPI users] open mpi on blue waters

[Mark Santcroos]

Hi Ralph,

There is no OMPI in system space and PATH and LD_LIBRARY_PATH look good.
Any suggestion on how to get more relevant debugging info above the table?

Thanks

Mark


> On 25 Mar 2015, at 16:33 , Ralph Castain <r...@open-mpi.org> wrote:
> 
> Hey Mark
> 
> Your original error flag indicates that you are picking up a connection from 
> some proc built against a different OMPI installation. It’s a very low-level 
> check that looks for matching version numbers. Not sure who is trying to 
> connect, but that is the problem.
> 
> Check you LD_LIBRARY_PATH
> 
>> On Mar 25, 2015, at 7:46 AM, Howard Pritchard <hpprit...@gmail.com> wrote:
>> 
>> turn off the disable getpwuid.
>> 
>> On Mar 25, 2015 8:14 AM, "Mark Santcroos" <mark.santcr...@rutgers.edu> wrote:
>> Hi Howard,
>> 
>> > On 25 Mar 2015, at 14:58 , Howard Pritchard <hpprit...@gmail.com> wrote:
>> > How are you building ompi?
>> 
>> My configure is rather straightforward:
>> ./configure --prefix=$OMPI_PREFIX --disable-getpwuid
>> 
>> Maybe I got spoiled on Hopper/Edison and I need more explicit configuration 
>> on BW ...
>> 
>> >  Also what happens if you use. aprun.
>> 
>> Not sure if you meant in combination with mpirun or not, so I'll provide 
>> both:
>> 
>> > aprun -n2 ./a.out
>> Hello from rank 1, thread 0, on nid16869. (core affinity = 0)
>> Hello from rank 0, thread 0, on nid16868. (core affinity = 0)
>> After sleep from rank 1, thread 0, on nid16869. (core affinity = 0)
>> After sleep from rank 0, thread 0, on nid16868. (core affinity = 0)
>> Application 23791589 resources: utime ~0s, stime ~2s, Rss ~27304, inblocks 
>> ~13229, outblocks ~66
>> 
>> > aprun -n2 mpirun ./a.out
>> apstat: error opening /ufs/alps_shared/reservations: No such file or 
>> directory
>> apstat: error opening /ufs/alps_shared/reservations: No such file or 
>> directory
>> [nid16868:17876] [[699,0],0] ORTE_ERROR_LOG: File open failure in file 
>> ../../../../../orte/mca/ras/tm/ras_tm_module.c at line 159
>> [nid16868:17876] [[699,0],0] ORTE_ERROR_LOG: File open failure in file 
>> ../../../../../orte/mca/ras/tm/ras_tm_module.c at line 85
>> [nid16868:17876] [[699,0],0] ORTE_ERROR_LOG: File open failure in file 
>> ../../../../orte/mca/ras/base/ras_base_allocate.c at line 190
>> [nid16869:17034] [[9344,0],0] ORTE_ERROR_LOG: File open failure in file 
>> ../../../../../orte/mca/ras/tm/ras_tm_module.c at line 159
>> [nid16869:17034] [[9344,0],0] ORTE_ERROR_LOG: File open failure in file 
>> ../../../../../orte/mca/ras/tm/ras_tm_module.c at line 85
>> [nid16869:17034] [[9344,0],0] ORTE_ERROR_LOG: File open failure in file 
>> ../../../../orte/mca/ras/base/ras_base_allocate.c at line 190
>> Application 23791590 exit codes: 1
>> Application 23791590 resources: utime ~0s, stime ~2s, Rss ~27304, inblocks 
>> ~9596, outblocks ~478
>> 
>> > I work with ompi on the nersc edison and hopper daily.
>> 
>> I use Edison and Hopper too, and there it works for me indeed.
>> 
>> > typically i use aprun though.
>> 
>> I want to use orte-submit and friends, so I "explicitly" don't want to use 
>> aprun.
>> 
>> > you definitely dont need to use ccm.
>> > and shouldnt.
>> 
>> Depends on the use-case, but happy to leave that out of scope for now :-)
>> 
>> Thanks!
>> 
>> Mark
>> 
>> 
>> >
>> > On Mar 25, 2015 6:00 AM, "Mark Santcroos" <mark.santcr...@rutgers.edu> 
>> > wrote:
>> > Hi,
>> >
>> > Any users of Open MPI on Blue Waters here?
>> > And then I specifically mean in "native" mode, not inside CCM.
>> >
>> > After configuring and building as I do on other Cray's, mpirun gives me 
>> > the following:
>> > [nid25263:31700] [[23896,0],0] ORTE_ERROR_LOG: Authentication failed in 
>> > file ../../../../../orte/mca/oob/tcp/oob_tcp_connection.c at line 803
>> > [nid25263:31700] [[23896,0],0] ORTE_ERROR_LOG: Authentication failed in 
>> > file ../../../../../orte/mca/oob/tcp/oob_tcp_connection.c at line 803
>> >
>> > Version is the latest and greatest from git.
>> >
>> > So I'm interested to hear whether people have been successful on Blue 
>> > Waters and/or whether the error rings a bell for people.
>> >
>> > Thanks!
>> >
>> > Mark
>> > ___
>> > users mailing list
>> > us...@open-mp

Re: [OMPI users] open mpi on blue waters

[Mark Santcroos]

> On 25 Mar 2015, at 15:46 , Howard Pritchard  wrote:
> turn off the disable getpwuid.

That doesn't seem to make a difference.

Have their been changes in this area? Last time I checked this a couple of 
months ago on Edison I needed this flag not to get spammed.

Re: [OMPI users] open mpi on blue waters

[Mark Santcroos]

Hi Howard,

> On 25 Mar 2015, at 14:58 , Howard Pritchard <hpprit...@gmail.com> wrote:
> How are you building ompi?

My configure is rather straightforward:
./configure --prefix=$OMPI_PREFIX --disable-getpwuid

Maybe I got spoiled on Hopper/Edison and I need more explicit configuration on 
BW ...

>  Also what happens if you use. aprun.

Not sure if you meant in combination with mpirun or not, so I'll provide both:

> aprun -n2 ./a.out 
Hello from rank 1, thread 0, on nid16869. (core affinity = 0)
Hello from rank 0, thread 0, on nid16868. (core affinity = 0)
After sleep from rank 1, thread 0, on nid16869. (core affinity = 0)
After sleep from rank 0, thread 0, on nid16868. (core affinity = 0)
Application 23791589 resources: utime ~0s, stime ~2s, Rss ~27304, inblocks 
~13229, outblocks ~66

> aprun -n2 mpirun ./a.out 
apstat: error opening /ufs/alps_shared/reservations: No such file or directory
apstat: error opening /ufs/alps_shared/reservations: No such file or directory
[nid16868:17876] [[699,0],0] ORTE_ERROR_LOG: File open failure in file 
../../../../../orte/mca/ras/tm/ras_tm_module.c at line 159
[nid16868:17876] [[699,0],0] ORTE_ERROR_LOG: File open failure in file 
../../../../../orte/mca/ras/tm/ras_tm_module.c at line 85
[nid16868:17876] [[699,0],0] ORTE_ERROR_LOG: File open failure in file 
../../../../orte/mca/ras/base/ras_base_allocate.c at line 190
[nid16869:17034] [[9344,0],0] ORTE_ERROR_LOG: File open failure in file 
../../../../../orte/mca/ras/tm/ras_tm_module.c at line 159
[nid16869:17034] [[9344,0],0] ORTE_ERROR_LOG: File open failure in file 
../../../../../orte/mca/ras/tm/ras_tm_module.c at line 85
[nid16869:17034] [[9344,0],0] ORTE_ERROR_LOG: File open failure in file 
../../../../orte/mca/ras/base/ras_base_allocate.c at line 190
Application 23791590 exit codes: 1
Application 23791590 resources: utime ~0s, stime ~2s, Rss ~27304, inblocks 
~9596, outblocks ~478

> I work with ompi on the nersc edison and hopper daily.

I use Edison and Hopper too, and there it works for me indeed.

> typically i use aprun though.

I want to use orte-submit and friends, so I "explicitly" don't want to use 
aprun.

> you definitely dont need to use ccm.
> and shouldnt.

Depends on the use-case, but happy to leave that out of scope for now :-)

Thanks!

Mark


> 
> On Mar 25, 2015 6:00 AM, "Mark Santcroos" <mark.santcr...@rutgers.edu> wrote:
> Hi,
> 
> Any users of Open MPI on Blue Waters here?
> And then I specifically mean in "native" mode, not inside CCM.
> 
> After configuring and building as I do on other Cray's, mpirun gives me the 
> following:
> [nid25263:31700] [[23896,0],0] ORTE_ERROR_LOG: Authentication failed in file 
> ../../../../../orte/mca/oob/tcp/oob_tcp_connection.c at line 803
> [nid25263:31700] [[23896,0],0] ORTE_ERROR_LOG: Authentication failed in file 
> ../../../../../orte/mca/oob/tcp/oob_tcp_connection.c at line 803
> 
> Version is the latest and greatest from git.
> 
> So I'm interested to hear whether people have been successful on Blue Waters 
> and/or whether the error rings a bell for people.
> 
> Thanks!
> 
> Mark
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2015/03/26505.php
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2015/03/26506.php

[OMPI users] open mpi on blue waters

[Mark Santcroos]

Hi,

Any users of Open MPI on Blue Waters here?
And then I specifically mean in "native" mode, not inside CCM.

After configuring and building as I do on other Cray's, mpirun gives me the 
following:
[nid25263:31700] [[23896,0],0] ORTE_ERROR_LOG: Authentication failed in file 
../../../../../orte/mca/oob/tcp/oob_tcp_connection.c at line 803
[nid25263:31700] [[23896,0],0] ORTE_ERROR_LOG: Authentication failed in file 
../../../../../orte/mca/oob/tcp/oob_tcp_connection.c at line 803

Version is the latest and greatest from git.

So I'm interested to hear whether people have been successful on Blue Waters 
and/or whether the error rings a bell for people.

Thanks!

Mark

Re: [OMPI users] independent startup of orted and orterun

[Mark Santcroos]

Ok great, sounds like a plan!

> On 04 Feb 2015, at 2:53 , Ralph Castain <r...@open-mpi.org> wrote:
> 
> Appreciate your patience! I'm somewhat limited this week by being on travel 
> to our HQ, so I don't have access to my usual test cluster. I'll be better 
> situated to complete the implementation once I get home.
> 
> For now, some quick thoughts:
> 
> 1. stdout/stderr: yes, I just need to "register" orte-submit as the one to 
> receive those from the submitted job.
> 
> 2. That one is going to be a tad trickier, but is resolvable. May take me a 
> little longer to fix.
> 
> 3. dang - I thought I had it doing so. I'll look to find the issue. I suspect 
> it's just a case of correctly setting the return code of orte-submit.
> 
> I'd welcome the help! Let me ponder the best way to point you to the areas 
> needing work, and we can kick around off-list about who does what.
> 
> Great to hear this is working with your tool so quickly!!
> Ralph
> 
> 
> On Tue, Feb 3, 2015 at 3:49 PM, Mark Santcroos <mark.santcr...@rutgers.edu> 
> wrote:
> Hi Ralph,
> 
> Besides the items in the other mail, I have three more items that would need 
> resolving at some point.
> 
> 1. STDOUT/STDERR currently go to the orte-dvm console.
>I'm sure this is not a fundamental limitation.
>Even if getting the information to the orte-submit instance would be 
> problematic, the orte-dvm writing this to a file per session would be good 
> enough too.
> 
> 2. Failing applications currently tear down the dvm.
>Ideally that would not be the case, and this would be handled in relation 
> to item (3).
>Possibly this needs to be configurable, if others would like to see 
> different behaviour.
> 
> 3. orte-submit doesn't return the exit code of the application.
> 
> To be clear, I realise the current implementation is a proof of concept, so 
> these are no complaints, just wishes of where I hope to see this going!
> 
> FWIW: these items might require less intricate knowledge of OMPI in general, 
> so with some pointers/guidance I can probably work on these myself if needed.
> 
> Cheers,
> 
> Mark
> 
> ps. I did a quick-and-dirty integration with our own tool and the ORTE 
> abstraction maps like a charm!
> 
> (https://github.com/radical-cybertools/radical.pilot/commit/2d36e886081bf8531097edfc95ada1826257e460)
> 
> > On 03 Feb 2015, at 20:38 , Mark Santcroos <mark.santcr...@rutgers.edu> 
> > wrote:
> >
> > Hi Ralph,
> >
> >> On 03 Feb 2015, at 16:28 , Ralph Castain <r...@open-mpi.org> wrote:
> >> I think I fixed some of the handshake issues - please give it another try.
> >> You should see orte-submit properly shutdown upon completion,
> >
> > Indeed, it works on my laptop now! Great!
> > It feels quite fast too, for sort tasks :-)
> >
> >> and orte-dvm properly shutdown when sent the terminate cmd.
> >
> > ACK. This also works as expected.
> >
> >> I was able to cleanly run MPI jobs on my laptop.
> >
> > Do you also see the following errors/warnings on the dvm side?
> >
> > [netbook:28324] [[20896,0],0] Releasing job data for [INVALID]
> > Hello, world, I am 0 of 1, (Open MPI v1.9a1, package: Open MPI mark@netbook 
> > Distribution, ident: 1.9.0a1, repo rev: dev-811-g7299cc3, Unreleased 
> > developer copy, 132)
> > [netbook:28324] sess_dir_finalize: proc session dir does not exist
> > [netbook:28324] [[20896,0],0] dvm: job [20896,20] has completed
> > [netbook:28324] [[20896,0],0] Releasing job data for [20896,20]
> >
> > The "INVALID" message is there for every "submit", the sess_dir_finalize 
> > exists per instance/core.
> > Is that something to worry about, that needs fixing or is that a 
> > configuration issue?
> >
> > I haven't been able to test on Edison because of maintenance 
> > (today+tomorrow), so I will report on that later.
> >
> > Thanks again!
> >
> > Mark
> 
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2015/02/26282.php
> 
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2015/02/26284.php

Re: [OMPI users] independent startup of orted and orterun

[Mark Santcroos]

Hi Ralph,

Besides the items in the other mail, I have three more items that would need 
resolving at some point.

1. STDOUT/STDERR currently go to the orte-dvm console.
   I'm sure this is not a fundamental limitation.
   Even if getting the information to the orte-submit instance would be 
problematic, the orte-dvm writing this to a file per session would be good 
enough too.

2. Failing applications currently tear down the dvm.
   Ideally that would not be the case, and this would be handled in relation to 
item (3).
   Possibly this needs to be configurable, if others would like to see 
different behaviour.

3. orte-submit doesn't return the exit code of the application.

To be clear, I realise the current implementation is a proof of concept, so 
these are no complaints, just wishes of where I hope to see this going!

FWIW: these items might require less intricate knowledge of OMPI in general, so 
with some pointers/guidance I can probably work on these myself if needed.

Cheers,

Mark 

ps. I did a quick-and-dirty integration with our own tool and the ORTE 
abstraction maps like a charm!

(https://github.com/radical-cybertools/radical.pilot/commit/2d36e886081bf8531097edfc95ada1826257e460)

> On 03 Feb 2015, at 20:38 , Mark Santcroos <mark.santcr...@rutgers.edu> wrote:
> 
> Hi Ralph,
> 
>> On 03 Feb 2015, at 16:28 , Ralph Castain <r...@open-mpi.org> wrote:
>> I think I fixed some of the handshake issues - please give it another try.
>> You should see orte-submit properly shutdown upon completion,
> 
> Indeed, it works on my laptop now! Great!
> It feels quite fast too, for sort tasks :-)
> 
>> and orte-dvm properly shutdown when sent the terminate cmd.
> 
> ACK. This also works as expected.
> 
>> I was able to cleanly run MPI jobs on my laptop.
> 
> Do you also see the following errors/warnings on the dvm side?
> 
> [netbook:28324] [[20896,0],0] Releasing job data for [INVALID]
> Hello, world, I am 0 of 1, (Open MPI v1.9a1, package: Open MPI mark@netbook 
> Distribution, ident: 1.9.0a1, repo rev: dev-811-g7299cc3, Unreleased 
> developer copy, 132)
> [netbook:28324] sess_dir_finalize: proc session dir does not exist
> [netbook:28324] [[20896,0],0] dvm: job [20896,20] has completed
> [netbook:28324] [[20896,0],0] Releasing job data for [20896,20]
> 
> The "INVALID" message is there for every "submit", the sess_dir_finalize 
> exists per instance/core.
> Is that something to worry about, that needs fixing or is that a 
> configuration issue?
> 
> I haven't been able to test on Edison because of maintenance 
> (today+tomorrow), so I will report on that later.
> 
> Thanks again!
> 
> Mark

Re: [OMPI users] independent startup of orted and orterun

[Mark Santcroos]

Hi Ralph,

> On 03 Feb 2015, at 16:28 , Ralph Castain  wrote:
> I think I fixed some of the handshake issues - please give it another try.
> You should see orte-submit properly shutdown upon completion,

Indeed, it works on my laptop now! Great!
It feels quite fast too, for sort tasks :-)

> and orte-dvm properly shutdown when sent the terminate cmd.

ACK. This also works as expected.

> I was able to cleanly run MPI jobs on my laptop.

Do you also see the following errors/warnings on the dvm side?

[netbook:28324] [[20896,0],0] Releasing job data for [INVALID]
Hello, world, I am 0 of 1, (Open MPI v1.9a1, package: Open MPI mark@netbook 
Distribution, ident: 1.9.0a1, repo rev: dev-811-g7299cc3, Unreleased developer 
copy, 132)
[netbook:28324] sess_dir_finalize: proc session dir does not exist
[netbook:28324] [[20896,0],0] dvm: job [20896,20] has completed
[netbook:28324] [[20896,0],0] Releasing job data for [20896,20]

The "INVALID" message is there for every "submit", the sess_dir_finalize exists 
per instance/core.
Is that something to worry about, that needs fixing or is that a configuration 
issue?

I haven't been able to test on Edison because of maintenance (today+tomorrow), 
so I will report on that later.

Thanks again!

Mark

Re: [OMPI users] independent startup of orted and orterun

[Mark Santcroos]

On 03 Feb 2015, at 0:20 , Ralph Castain  wrote:
> Okay, thanks - I'll get on it tonight. Looks like a fairly simple bug, so 
> hopefully I'll have it ironed out tonight.

Sorry, I was not completely accurate. Let me be more specific:

* The orte-submit does not return though, so that symptom is similar.

* On my laptop I dont get the libevent warnings (the output redirection hid 
that fact).

* The "-np" behavior is different too, but I guess that is because there is no 
RM in place on my laptop.

Re: [OMPI users] independent startup of orted and orterun

[Mark Santcroos]

FWIW: I see similar behaviour on my laptop (OS X Yosemite 10.10.2).

> On 02 Feb 2015, at 21:26 , Mark Santcroos <mark.santcr...@rutgers.edu> wrote:
> 
> Ok, let me check on some other systems too though, it might be Cray specific.
> 
> 
>> On 02 Feb 2015, at 19:07 , Ralph Castain <r...@open-mpi.org> wrote:
>> 
>> Yikes - looks like a bug crept into there at the last minute. I actually had 
>> it working just fine - not sure what happened here. I'm on travel this week, 
>> but I'll try to dig into this a bit and spot the issue.
>> 
>> Thanks!
>> Ralph
>> 
>> 
>> On Mon, Feb 2, 2015 at 3:50 AM, Mark Santcroos <mark.santcr...@rutgers.edu> 
>> wrote:
>> Hi Ralph,
>> 
>> Great, the semantics look exactly as what I need!
>> 
>> (To aid in debugging I added "--debug-devel" to orte-dvm.c which was useful 
>> to detect and come by some initial bumps)
>> 
>> The current status:
>> 
>> * I can submit applications and see their output on the orte-dvm console
>> 
>> * The following message is reported infinitely on the orte-submit console:
>> 
>> [warn] opal_libevent2022_event_base_loop: reentrant invocation.  Only one 
>> event_base_loop can run on each event_base at once.
>> 
>> * orte-submit doesn't return, while I see "[nid02819:20571] [[2120,0],0] 
>> dvm: job [2120,9] has completed" on the orte-dvm console.
>> 
>> * On the orte-dvm console I see the following when submitting (so also for 
>> "successful" runs):
>> 
>> [nid02434:00564] [[9021,0],0] Releasing job data for [INVALID]
>> [nid03388:26474] [[9021,0],2] ORTE_ERROR_LOG: Not found in file 
>> ../../../../orte/mca/odls/base/odls_base_default_fns.c at line 433
>> [nid03534:31545] procdir: /tmp/openmpi-sessions-62758@nid03534_0/9021/1/0
>> [nid03534:31545] jobdir: /tmp/openmpi-sessions-62758@nid03534_0/9021/1
>> [nid03534:31545] top: openmpi-sessions-62758@nid03534_0
>> [nid03534:31545] tmp: /tmp
>> [nid03534:31545] sess_dir_finalize: proc session dir does not exist
>> 
>> * If I dont specify any "-np" on the orte-submit, then I see on the orte-dvm 
>> console:
>> 
>> [nid02434:00564] [[9021,0],0] Releasing job data for [INVALID]
>> [nid03388:26474] [[9021,0],2] ORTE_ERROR_LOG: Not found in file 
>> ../../../../orte/mca/odls/base/odls_base_default_fns.c at line 433
>> [nid03534:31544] [[9021,0],1] ORTE_ERROR_LOG: Not found in file 
>> ../../../../orte/mca/odls/base/odls_base_default_fns.c at line 433
>> 
>> * It only seems to work for single nodes (probably related to the previous 
>> point).
>> 
>> 
>> Is this all expected behaviour given the current implementation?
>> 
>> 
>> Thanks!
>> 
>> Mark
>> 
>> 
>> 
>>> On 02 Feb 2015, at 4:21 , Ralph Castain <r...@open-mpi.org> wrote:
>>> 
>>> I have pushed the changes to the OMPI master. It took a little bit more 
>>> than I had hoped due to the changes to the ORTE infrastructure, but 
>>> hopefully this will meet your needs. It consists of two new tools:
>>> 
>>> (a) orte-dvm - starts the virtual machine by launching a daemon on every 
>>> node of the allocation, as constrained by -host and/or -hostfile. Check the 
>>> options for outputting the URI as you’ll need that info for the other tool. 
>>> The DVM remains “up” until you issue the orte-submit -terminate command, or 
>>> hit the orte-dvm process with a sigterm.
>>> 
>>> (b) orte-submit - takes the place of mpirun. Basically just packages your 
>>> app and arguments and sends it to orte-dvm for execution. Requires the URI 
>>> of orte-dvm. The tool exits once the job has completed execution, though 
>>> you can run multiple jobs in parallel by backgrounding orte-submit or 
>>> issuing commands from separate shells.
>>> 
>>> I’ve added man pages for both tools, though they may not be complete. Also, 
>>> I don’t have all the mapping/ranking/binding options supported just yet as 
>>> I first wanted to see if this meets your basic needs before worrying about 
>>> the detail.
>>> 
>>> Let me know what you think
>>> Ralph
>>> 
>>> 
>>>> On Jan 21, 2015, at 4:07 PM, Mark Santcroos <mark.santcr...@rutgers.edu> 
>>>> wrote:
>>>> 
>>>> Hi Ralph,
>>>> 
>>>> All makes sense! Thanks a lot!
>>>> 
>>>> Looking forward to your modifications.
>>>> Please d

Re: [OMPI users] independent startup of orted and orterun

[Mark Santcroos]

Ok, let me check on some other systems too though, it might be Cray specific.


> On 02 Feb 2015, at 19:07 , Ralph Castain <r...@open-mpi.org> wrote:
> 
> Yikes - looks like a bug crept into there at the last minute. I actually had 
> it working just fine - not sure what happened here. I'm on travel this week, 
> but I'll try to dig into this a bit and spot the issue.
> 
> Thanks!
> Ralph
> 
> 
> On Mon, Feb 2, 2015 at 3:50 AM, Mark Santcroos <mark.santcr...@rutgers.edu> 
> wrote:
> Hi Ralph,
> 
> Great, the semantics look exactly as what I need!
> 
> (To aid in debugging I added "--debug-devel" to orte-dvm.c which was useful 
> to detect and come by some initial bumps)
> 
> The current status:
> 
> * I can submit applications and see their output on the orte-dvm console
> 
> * The following message is reported infinitely on the orte-submit console:
> 
> [warn] opal_libevent2022_event_base_loop: reentrant invocation.  Only one 
> event_base_loop can run on each event_base at once.
> 
> * orte-submit doesn't return, while I see "[nid02819:20571] [[2120,0],0] dvm: 
> job [2120,9] has completed" on the orte-dvm console.
> 
> * On the orte-dvm console I see the following when submitting (so also for 
> "successful" runs):
> 
> [nid02434:00564] [[9021,0],0] Releasing job data for [INVALID]
> [nid03388:26474] [[9021,0],2] ORTE_ERROR_LOG: Not found in file 
> ../../../../orte/mca/odls/base/odls_base_default_fns.c at line 433
> [nid03534:31545] procdir: /tmp/openmpi-sessions-62758@nid03534_0/9021/1/0
> [nid03534:31545] jobdir: /tmp/openmpi-sessions-62758@nid03534_0/9021/1
> [nid03534:31545] top: openmpi-sessions-62758@nid03534_0
> [nid03534:31545] tmp: /tmp
> [nid03534:31545] sess_dir_finalize: proc session dir does not exist
> 
> * If I dont specify any "-np" on the orte-submit, then I see on the orte-dvm 
> console:
> 
> [nid02434:00564] [[9021,0],0] Releasing job data for [INVALID]
> [nid03388:26474] [[9021,0],2] ORTE_ERROR_LOG: Not found in file 
> ../../../../orte/mca/odls/base/odls_base_default_fns.c at line 433
> [nid03534:31544] [[9021,0],1] ORTE_ERROR_LOG: Not found in file 
> ../../../../orte/mca/odls/base/odls_base_default_fns.c at line 433
> 
> * It only seems to work for single nodes (probably related to the previous 
> point).
> 
> 
> Is this all expected behaviour given the current implementation?
> 
> 
> Thanks!
> 
> Mark
> 
> 
> 
> > On 02 Feb 2015, at 4:21 , Ralph Castain <r...@open-mpi.org> wrote:
> >
> > I have pushed the changes to the OMPI master. It took a little bit more 
> > than I had hoped due to the changes to the ORTE infrastructure, but 
> > hopefully this will meet your needs. It consists of two new tools:
> >
> > (a) orte-dvm - starts the virtual machine by launching a daemon on every 
> > node of the allocation, as constrained by -host and/or -hostfile. Check the 
> > options for outputting the URI as you’ll need that info for the other tool. 
> > The DVM remains “up” until you issue the orte-submit -terminate command, or 
> > hit the orte-dvm process with a sigterm.
> >
> > (b) orte-submit - takes the place of mpirun. Basically just packages your 
> > app and arguments and sends it to orte-dvm for execution. Requires the URI 
> > of orte-dvm. The tool exits once the job has completed execution, though 
> > you can run multiple jobs in parallel by backgrounding orte-submit or 
> > issuing commands from separate shells.
> >
> > I’ve added man pages for both tools, though they may not be complete. Also, 
> > I don’t have all the mapping/ranking/binding options supported just yet as 
> > I first wanted to see if this meets your basic needs before worrying about 
> > the detail.
> >
> > Let me know what you think
> > Ralph
> >
> >
> >> On Jan 21, 2015, at 4:07 PM, Mark Santcroos <mark.santcr...@rutgers.edu> 
> >> wrote:
> >>
> >> Hi Ralph,
> >>
> >> All makes sense! Thanks a lot!
> >>
> >> Looking forward to your modifications.
> >> Please don't hesitate to through things with rough-edges to me!
> >>
> >> Cheers,
> >>
> >> Mark
> >>
> >>> On 21 Jan 2015, at 23:21 , Ralph Castain <r...@open-mpi.org> wrote:
> >>>
> >>> Let me address your questions up here so you don’t have to scan thru the 
> >>> entire note.
> >>>
> >>> PMIx rationale: PMI has been around for a long time, primarily used 
> >>> inside the MPI library implementations to perform wireup. It pro

Re: [OMPI users] independent startup of orted and orterun

[Mark Santcroos]

Hi Ralph,

Great, the semantics look exactly as what I need!

(To aid in debugging I added "--debug-devel" to orte-dvm.c which was useful to 
detect and come by some initial bumps)

The current status:

* I can submit applications and see their output on the orte-dvm console

* The following message is reported infinitely on the orte-submit console:

[warn] opal_libevent2022_event_base_loop: reentrant invocation.  Only one 
event_base_loop can run on each event_base at once.

* orte-submit doesn't return, while I see "[nid02819:20571] [[2120,0],0] dvm: 
job [2120,9] has completed" on the orte-dvm console.

* On the orte-dvm console I see the following when submitting (so also for 
"successful" runs):

[nid02434:00564] [[9021,0],0] Releasing job data for [INVALID]
[nid03388:26474] [[9021,0],2] ORTE_ERROR_LOG: Not found in file 
../../../../orte/mca/odls/base/odls_base_default_fns.c at line 433
[nid03534:31545] procdir: /tmp/openmpi-sessions-62758@nid03534_0/9021/1/0
[nid03534:31545] jobdir: /tmp/openmpi-sessions-62758@nid03534_0/9021/1
[nid03534:31545] top: openmpi-sessions-62758@nid03534_0
[nid03534:31545] tmp: /tmp
[nid03534:31545] sess_dir_finalize: proc session dir does not exist

* If I dont specify any "-np" on the orte-submit, then I see on the orte-dvm 
console:

[nid02434:00564] [[9021,0],0] Releasing job data for [INVALID]
[nid03388:26474] [[9021,0],2] ORTE_ERROR_LOG: Not found in file 
../../../../orte/mca/odls/base/odls_base_default_fns.c at line 433
[nid03534:31544] [[9021,0],1] ORTE_ERROR_LOG: Not found in file 
../../../../orte/mca/odls/base/odls_base_default_fns.c at line 433

* It only seems to work for single nodes (probably related to the previous 
point).


Is this all expected behaviour given the current implementation?


Thanks!

Mark



> On 02 Feb 2015, at 4:21 , Ralph Castain <r...@open-mpi.org> wrote:
> 
> I have pushed the changes to the OMPI master. It took a little bit more than 
> I had hoped due to the changes to the ORTE infrastructure, but hopefully this 
> will meet your needs. It consists of two new tools:
> 
> (a) orte-dvm - starts the virtual machine by launching a daemon on every node 
> of the allocation, as constrained by -host and/or -hostfile. Check the 
> options for outputting the URI as you’ll need that info for the other tool. 
> The DVM remains “up” until you issue the orte-submit -terminate command, or 
> hit the orte-dvm process with a sigterm.
> 
> (b) orte-submit - takes the place of mpirun. Basically just packages your app 
> and arguments and sends it to orte-dvm for execution. Requires the URI of 
> orte-dvm. The tool exits once the job has completed execution, though you can 
> run multiple jobs in parallel by backgrounding orte-submit or issuing 
> commands from separate shells.
> 
> I’ve added man pages for both tools, though they may not be complete. Also, I 
> don’t have all the mapping/ranking/binding options supported just yet as I 
> first wanted to see if this meets your basic needs before worrying about the 
> detail.
> 
> Let me know what you think
> Ralph
> 
> 
>> On Jan 21, 2015, at 4:07 PM, Mark Santcroos <mark.santcr...@rutgers.edu> 
>> wrote:
>> 
>> Hi Ralph,
>> 
>> All makes sense! Thanks a lot!
>> 
>> Looking forward to your modifications.
>> Please don't hesitate to through things with rough-edges to me!
>> 
>> Cheers,
>> 
>> Mark
>> 
>>> On 21 Jan 2015, at 23:21 , Ralph Castain <r...@open-mpi.org> wrote:
>>> 
>>> Let me address your questions up here so you don’t have to scan thru the 
>>> entire note.
>>> 
>>> PMIx rationale: PMI has been around for a long time, primarily used inside 
>>> the MPI library implementations to perform wireup. It provided a link from 
>>> the MPI library to the local resource manager. However, as we move towards 
>>> exascale, two things became apparent:
>>> 
>>> 1. the current PMI implementations don’t scale adequately to get there. The 
>>> API created too many communications and assumed everything was a blocking 
>>> operation, thus preventing asynchronous progress
>>> 
>>> 2. there were increasing requests for application-level interactions to the 
>>> resource manager. People want ways to spawn jobs (and not just from within 
>>> MPI), request pre-location of data, control power, etc. Rather than having 
>>> every RM write its own interface (and thus make everyone’s code 
>>> non-portable), we at Intel decided to extend the existing PMI definitions 
>>> to support those functions. Thus, an application developer can directly 
>>> access PMIx functions to perform all those operations.
>>>

Re: [OMPI users] independent startup of orted and orterun

[Mark Santcroos]

Hi Ralph,

All makes sense! Thanks a lot!

Looking forward to your modifications.
Please don't hesitate to through things with rough-edges to me!

Cheers,

Mark

> On 21 Jan 2015, at 23:21 , Ralph Castain <r...@open-mpi.org> wrote:
> 
> Let me address your questions up here so you don’t have to scan thru the 
> entire note.
> 
> PMIx rationale: PMI has been around for a long time, primarily used inside 
> the MPI library implementations to perform wireup. It provided a link from 
> the MPI library to the local resource manager. However, as we move towards 
> exascale, two things became apparent:
> 
> 1. the current PMI implementations don’t scale adequately to get there. The 
> API created too many communications and assumed everything was a blocking 
> operation, thus preventing asynchronous progress
> 
> 2. there were increasing requests for application-level interactions to the 
> resource manager. People want ways to spawn jobs (and not just from within 
> MPI), request pre-location of data, control power, etc. Rather than having 
> every RM write its own interface (and thus make everyone’s code 
> non-portable), we at Intel decided to extend the existing PMI definitions to 
> support those functions. Thus, an application developer can directly access 
> PMIx functions to perform all those operations.
> 
> PMIx v1.0 is about to be released - it’ll be backward compatible with PMI-1 
> and PMI-2, plus add non-blocking operations and significantly reduce the 
> number of communications. PMIx 2.0 is slated for this summer and will include 
> the advanced controls capabilities.
> 
> ORCM is being developed because we needed a BSD-licensed, fully featured 
> resource manager. This will allow us to integrate the RM even more tightly to 
> the file system, networking, and other subsystems, thus achieving higher 
> launch performance and providing desired features such as QoS management. 
> PMIx is a part of that plan, but as you say, they each play their separate 
> roles in the overall stack.
> 
> 
> Persistent ORTE: there is a learning curve on ORTE, I fear. We do have some 
> videos on the web site that can help get you started, and I’ve given a number 
> of “classes" at Intel now for that purpose. I still have it on my “to-do” 
> list that I summarize those classes and post them on the web site.
> 
> For now, let me summarize how things work. At startup, mpirun reads the 
> allocation (usually from the environment, but it depends on the host RM) and 
> launches a daemon on each allocated node. Each daemon reads its local 
> hardware environment and “phones home” to let mpirun know it is alive. Once 
> all daemons have reported, mpirun maps the processes to the nodes and sends 
> that map to all the daemons in a scalable broadcast pattern.
> 
> Upon receipt of the launch message, each daemon parses it to identify which 
> procs it needs to locally spawn. Once spawned, each proc connects back to its 
> local daemon via a Unix domain socket for wireup support. As procs complete, 
> the daemon maintains bookkeeping and reports back to mpirun once all procs 
> are done. When all procs are reported complete (or one reports as abnormally 
> terminated), mpirun sends a “die” message to every daemon so it will cleanly 
> terminate.
> 
> What I will do is simply tell mpirun to not do that last step, but instead to 
> wait to receive a “terminate” cmd before ending the daemons. This will allow 
> you to reuse the existing DVM, making each independent job start a great deal 
> faster. You’ll need to either manually terminate the DVM, or the RM will do 
> so when the allocation expires.
> 
> HTH
> Ralph
> 
> 
>> On Jan 21, 2015, at 12:52 PM, Mark Santcroos <mark.santcr...@rutgers.edu> 
>> wrote:
>> 
>> Hi Ralph,
>> 
>>> On 21 Jan 2015, at 21:20 , Ralph Castain <r...@open-mpi.org> wrote:
>>> 
>>> Hi Mark
>>> 
>>>> On Jan 21, 2015, at 11:21 AM, Mark Santcroos <mark.santcr...@rutgers.edu> 
>>>> wrote:
>>>> 
>>>> Hi Ralph, all,
>>>> 
>>>> To give some background, I'm part of the RADICAL-Pilot [1] development 
>>>> team.
>>>> RADICAL-Pilot is a Pilot System, an implementation of the Pilot (job) 
>>>> concept, which is in its most minimal form takes care of the decoupling of 
>>>> resource acquisition and workload management.
>>>> So instead of launching your real_science.exe through PBS, you submit a 
>>>> Pilot, which will allow you to perform application level scheduling.
>>>> Most obvious use-case if you want to run many (relatively) small tasks, 
>>>> then you really d

Re: [OMPI users] independent startup of orted and orterun

[Mark Santcroos]

Hi Ralph,

> On 21 Jan 2015, at 21:20 , Ralph Castain <r...@open-mpi.org> wrote:
> 
> Hi Mark
> 
>> On Jan 21, 2015, at 11:21 AM, Mark Santcroos <mark.santcr...@rutgers.edu> 
>> wrote:
>> 
>> Hi Ralph, all,
>> 
>> To give some background, I'm part of the RADICAL-Pilot [1] development team.
>> RADICAL-Pilot is a Pilot System, an implementation of the Pilot (job) 
>> concept, which is in its most minimal form takes care of the decoupling of 
>> resource acquisition and workload management.
>> So instead of launching your real_science.exe through PBS, you submit a 
>> Pilot, which will allow you to perform application level scheduling.
>> Most obvious use-case if you want to run many (relatively) small tasks, then 
>> you really don;t want to go through the batch system every time. That is 
>> besides the fact that these machines are very bad in managing many tasks 
>> anyway.
> 
> Yeah, we sympathize.

Thats always good :-)

> Of course, one obvious solution is to get an allocation and execute a shell 
> script that runs the tasks within that allocation - yes?

Not really. Most of our use-cases have dynamic runtime properties, which means 
that at t=0 the exact workload is not known.

In addition, I don't think such a script would allow me to work around the 
aprun bottleneck, as I'm not aware of a way to start MPI tasks that span 
multiple nodes from a Cray worker node.

>> I looked a bit better at ORCM and it clearly overlaps with what I want to 
>> achieve.
> 
> Agreed. In ORCM, we allow a user to request a “session” that results in 
> allocation of resources. Each session is given an “orchestrator” - the ORCM 
> “shepherd” daemon - responsible for executing the individual tasks across the 
> assigned allocation, and a collection of “lamb” daemons (one on each node of 
> the allocation) that forms a distributed VM. The orchestrator can execute the 
> tasks very quickly since it doesn’t have to go back to the scheduler, and we 
> allow it to do so according to any provided precedence requirement. Again, 
> for simplicity, a shell script is the default mechanism for submitting the 
> individual tasks.

Yeah, similar solution to a similar problem.
I noticed that Exascale is also part of the motivation? How does this relate to 
the pmix effort? Different part of the stack I guess.

>> One thing I noticed is that parts of it runs as root, why is that?
> 
> ORCM is a full resource manager, which means it has a scheduler (rudimentary 
> today) and boot-time daemons that must run as root so they can fork/exec the 
> session-level daemons (that run at the user level). The orchestrator and its 
> daemons all run at the user-level.

Ok. Our solution is user-space only, as one of our features is that we are able 
to run across different type of systems. Both approaches come with a tradeoff 
obviously.

>>> We used to have a cmd line option in ORTE for what you propose - it 
>>> wouldn’t be too hard to restore. Is there some reason to do so?
>> 
>> Can you point me to something that I could look for in the repo history, 
>> then I can see if it serves my purpose.
> 
> It would be back in the svn repo, I fear - would take awhile to hunt it down. 
> Basically, it just (a) started all the daemons to create a VM, and (b) told 
> mpirun to stick around as a persistent daemon. All subsequent calls to mpirun 
> would reference back to the persistent one, thus using it to launch the jobs 
> against the standing VM instead of starting a new one every time.

*nod* That's what I tried to do this afternoon actually with the 
"--ompi-server", but that was not meant to be.

> For ORCM, we just took that capability and expressed it as the “shepherd” 
> plus “lamb” daemon architecture described above.

ACK.

> If you don’t want to replace the base RM, then using ORTE to establish a 
> persistent VM is probably the way to go.

Indeed, thats what it sounds like. Plus that ORTE is generic enough that I can 
re-use it on other type of systems too.

> I can probably make it do that again fairly readily. We have a developer’s 
> meeting next week, which usually means I have some free time (during evenings 
> and topics I’m not involved with), so I can take a crack at this then if that 
> would be timely enough.

Happy to accept that offer. At this stage I'm not sure if I would want a CLI or 
would be more interested to be able to do this programmatically though.
Also more than willing to assist in any way I can.

I tried to see how it all worked, but because of the modular nature of ompi 
that was quite daunting. There is some learning curve I guess :-)
So it seems that mpirun is persistent, and opens up a listening port, then some 
orded's get launched that phone home.
From there I got lost in the MCA maze. How do the tasks get unto the compute 
nodes and started?

Thanks a lot again, I appreciate your help.

Cheers,

Mark

Re: [OMPI users] independent startup of orted and orterun

[Mark Santcroos]

Hi Ralph, all,

To give some background, I'm part of the RADICAL-Pilot [1] development team.
RADICAL-Pilot is a Pilot System, an implementation of the Pilot (job) concept, 
which is in its most minimal form takes care of the decoupling of resource 
acquisition and workload management.
So instead of launching your real_science.exe through PBS, you submit a Pilot, 
which will allow you to perform application level scheduling.
Most obvious use-case if you want to run many (relatively) small tasks, then 
you really don;t want to go through the batch system every time. That is 
besides the fact that these machines are very bad in managing many tasks anyway.

The recent discussion we had on Spawn() on Cray's also originates here.
I want to free myself from having to use aprun for every task, and therefore I 
am interested to see if ompi and/or orte can be the vehicle for that.

> On 21 Jan 2015, at 17:16 , Ralph Castain  wrote:
> Theoretically, yes - see the ORCM project, which basically does what you ask.
> The launch system in there isn’t fully implemented yet, but the fundamental 
> idea is valid and supports some range of capability.

I looked a bit better at ORCM and it clearly overlaps with what I want to 
achieve.
One thing I noticed is that parts of it runs as root, why is that?

> We used to have a cmd line option in ORTE for what you propose - it wouldn’t 
> be too hard to restore. Is there some reason to do so?

Can you point me to something that I could look for in the repo history, then I 
can see if it serves my purpose.

If you think there is enough to warrant looking in more detail at ORCM I'm 
happy to do that too.

Cheers,

Mark

1. https://github.com/radical-cybertools/radical.pilot

Re: [OMPI users] independent startup of orted and orterun

[Mark Santcroos]

Hi Ralph,

I figured that MR+ had to require something similar too (assuming it actually 
does use ORCM).

Let me study that wiki a bit and I will come back with a description of my 
use-case.

Thanks!

Mark


> On 21 Jan 2015, at 17:18 , Ralph Castain <r...@open-mpi.org> wrote:
> 
> Sorry - should have included the link to the ORCM project:
> 
> https://github.com/open-mpi/orcm/wiki
> 
> 
>> On Jan 21, 2015, at 8:16 AM, Ralph Castain <r...@open-mpi.org> wrote:
>> 
>> Theoretically, yes - see the ORCM project, which basically does what you 
>> ask. The launch system in there isn’t fully implemented yet, but the 
>> fundamental idea is valid and supports some range of capability.
>> 
>> We used to have a cmd line option in ORTE for what you propose - it wouldn’t 
>> be too hard to restore. Is there some reason to do so?
>> 
>> 
>>> On Jan 21, 2015, at 7:53 AM, Mark Santcroos <mark.santcr...@rutgers.edu> 
>>> wrote:
>>> 
>>> Hi,
>>> 
>>> Would it be possible to initially run "idle" orted's on my resources and 
>>> then use orterun to launch my applications to these already running orted's.
>>> 
>>> Thanks!
>>> 
>>> Mark
>>> ___
>>> users mailing list
>>> us...@open-mpi.org
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>> Link to this post: 
>>> http://www.open-mpi.org/community/lists/users/2015/01/26220.php
>> 
> 
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2015/01/26222.php

[OMPI users] independent startup of orted and orterun

[Mark Santcroos]

Hi,

Would it be possible to initially run "idle" orted's on my resources and then 
use orterun to launch my applications to these already running orted's.

Thanks!

Mark