Re: [OMPI users] independent startup of orted and orterun

[Ralph Castain]

We're going to take this off-list so we quit peppering you all with the
development...will report back when we have something more concrete should
anyone else be interested.



On Wed, Feb 4, 2015 at 2:22 AM, Mark Santcroos 
wrote:

> Ok great, sounds like a plan!
>
> > On 04 Feb 2015, at 2:53 , Ralph Castain  wrote:
> >
> > Appreciate your patience! I'm somewhat limited this week by being on
> travel to our HQ, so I don't have access to my usual test cluster. I'll be
> better situated to complete the implementation once I get home.
> >
> > For now, some quick thoughts:
> >
> > 1. stdout/stderr: yes, I just need to "register" orte-submit as the one
> to receive those from the submitted job.
> >
> > 2. That one is going to be a tad trickier, but is resolvable. May take
> me a little longer to fix.
> >
> > 3. dang - I thought I had it doing so. I'll look to find the issue. I
> suspect it's just a case of correctly setting the return code of
> orte-submit.
> >
> > I'd welcome the help! Let me ponder the best way to point you to the
> areas needing work, and we can kick around off-list about who does what.
> >
> > Great to hear this is working with your tool so quickly!!
> > Ralph
> >
> >
> > On Tue, Feb 3, 2015 at 3:49 PM, Mark Santcroos <
> mark.santcr...@rutgers.edu> wrote:
> > Hi Ralph,
> >
> > Besides the items in the other mail, I have three more items that would
> need resolving at some point.
> >
> > 1. STDOUT/STDERR currently go to the orte-dvm console.
> >I'm sure this is not a fundamental limitation.
> >Even if getting the information to the orte-submit instance would be
> problematic, the orte-dvm writing this to a file per session would be good
> enough too.
> >
> > 2. Failing applications currently tear down the dvm.
> >Ideally that would not be the case, and this would be handled in
> relation to item (3).
> >Possibly this needs to be configurable, if others would like to see
> different behaviour.
> >
> > 3. orte-submit doesn't return the exit code of the application.
> >
> > To be clear, I realise the current implementation is a proof of concept,
> so these are no complaints, just wishes of where I hope to see this going!
> >
> > FWIW: these items might require less intricate knowledge of OMPI in
> general, so with some pointers/guidance I can probably work on these myself
> if needed.
> >
> > Cheers,
> >
> > Mark
> >
> > ps. I did a quick-and-dirty integration with our own tool and the ORTE
> abstraction maps like a charm!
> > (
> https://github.com/radical-cybertools/radical.pilot/commit/2d36e886081bf8531097edfc95ada1826257e460
> )
> >
> > > On 03 Feb 2015, at 20:38 , Mark Santcroos 
> wrote:
> > >
> > > Hi Ralph,
> > >
> > >> On 03 Feb 2015, at 16:28 , Ralph Castain  wrote:
> > >> I think I fixed some of the handshake issues - please give it another
> try.
> > >> You should see orte-submit properly shutdown upon completion,
> > >
> > > Indeed, it works on my laptop now! Great!
> > > It feels quite fast too, for sort tasks :-)
> > >
> > >> and orte-dvm properly shutdown when sent the terminate cmd.
> > >
> > > ACK. This also works as expected.
> > >
> > >> I was able to cleanly run MPI jobs on my laptop.
> > >
> > > Do you also see the following errors/warnings on the dvm side?
> > >
> > > [netbook:28324] [[20896,0],0] Releasing job data for [INVALID]
> > > Hello, world, I am 0 of 1, (Open MPI v1.9a1, package: Open MPI
> mark@netbook Distribution, ident: 1.9.0a1, repo rev: dev-811-g7299cc3,
> Unreleased developer copy, 132)
> > > [netbook:28324] sess_dir_finalize: proc session dir does not exist
> > > [netbook:28324] [[20896,0],0] dvm: job [20896,20] has completed
> > > [netbook:28324] [[20896,0],0] Releasing job data for [20896,20]
> > >
> > > The "INVALID" message is there for every "submit", the
> sess_dir_finalize exists per instance/core.
> > > Is that something to worry about, that needs fixing or is that a
> configuration issue?
> > >
> > > I haven't been able to test on Edison because of maintenance
> (today+tomorrow), so I will report on that later.
> > >
> > > Thanks again!
> > >
> > > Mark
> >
> > ___
> > users mailing list
> > us...@open-mpi.org
> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> > Link to this post:
> http://www.open-mpi.org/community/lists/users/2015/02/26282.php
> >
> > ___
> > users mailing list
> > us...@open-mpi.org
> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> > Link to this post:
> http://www.open-mpi.org/community/lists/users/2015/02/26284.php
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2015/02/26289.php
>

Re: [OMPI users] independent startup of orted and orterun

[Mark Santcroos]

Ok great, sounds like a plan!

> On 04 Feb 2015, at 2:53 , Ralph Castain  wrote:
> 
> Appreciate your patience! I'm somewhat limited this week by being on travel 
> to our HQ, so I don't have access to my usual test cluster. I'll be better 
> situated to complete the implementation once I get home.
> 
> For now, some quick thoughts:
> 
> 1. stdout/stderr: yes, I just need to "register" orte-submit as the one to 
> receive those from the submitted job.
> 
> 2. That one is going to be a tad trickier, but is resolvable. May take me a 
> little longer to fix.
> 
> 3. dang - I thought I had it doing so. I'll look to find the issue. I suspect 
> it's just a case of correctly setting the return code of orte-submit.
> 
> I'd welcome the help! Let me ponder the best way to point you to the areas 
> needing work, and we can kick around off-list about who does what.
> 
> Great to hear this is working with your tool so quickly!!
> Ralph
> 
> 
> On Tue, Feb 3, 2015 at 3:49 PM, Mark Santcroos  
> wrote:
> Hi Ralph,
> 
> Besides the items in the other mail, I have three more items that would need 
> resolving at some point.
> 
> 1. STDOUT/STDERR currently go to the orte-dvm console.
>I'm sure this is not a fundamental limitation.
>Even if getting the information to the orte-submit instance would be 
> problematic, the orte-dvm writing this to a file per session would be good 
> enough too.
> 
> 2. Failing applications currently tear down the dvm.
>Ideally that would not be the case, and this would be handled in relation 
> to item (3).
>Possibly this needs to be configurable, if others would like to see 
> different behaviour.
> 
> 3. orte-submit doesn't return the exit code of the application.
> 
> To be clear, I realise the current implementation is a proof of concept, so 
> these are no complaints, just wishes of where I hope to see this going!
> 
> FWIW: these items might require less intricate knowledge of OMPI in general, 
> so with some pointers/guidance I can probably work on these myself if needed.
> 
> Cheers,
> 
> Mark
> 
> ps. I did a quick-and-dirty integration with our own tool and the ORTE 
> abstraction maps like a charm!
> 
> (https://github.com/radical-cybertools/radical.pilot/commit/2d36e886081bf8531097edfc95ada1826257e460)
> 
> > On 03 Feb 2015, at 20:38 , Mark Santcroos  
> > wrote:
> >
> > Hi Ralph,
> >
> >> On 03 Feb 2015, at 16:28 , Ralph Castain  wrote:
> >> I think I fixed some of the handshake issues - please give it another try.
> >> You should see orte-submit properly shutdown upon completion,
> >
> > Indeed, it works on my laptop now! Great!
> > It feels quite fast too, for sort tasks :-)
> >
> >> and orte-dvm properly shutdown when sent the terminate cmd.
> >
> > ACK. This also works as expected.
> >
> >> I was able to cleanly run MPI jobs on my laptop.
> >
> > Do you also see the following errors/warnings on the dvm side?
> >
> > [netbook:28324] [[20896,0],0] Releasing job data for [INVALID]
> > Hello, world, I am 0 of 1, (Open MPI v1.9a1, package: Open MPI mark@netbook 
> > Distribution, ident: 1.9.0a1, repo rev: dev-811-g7299cc3, Unreleased 
> > developer copy, 132)
> > [netbook:28324] sess_dir_finalize: proc session dir does not exist
> > [netbook:28324] [[20896,0],0] dvm: job [20896,20] has completed
> > [netbook:28324] [[20896,0],0] Releasing job data for [20896,20]
> >
> > The "INVALID" message is there for every "submit", the sess_dir_finalize 
> > exists per instance/core.
> > Is that something to worry about, that needs fixing or is that a 
> > configuration issue?
> >
> > I haven't been able to test on Edison because of maintenance 
> > (today+tomorrow), so I will report on that later.
> >
> > Thanks again!
> >
> > Mark
> 
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2015/02/26282.php
> 
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2015/02/26284.php

Re: [OMPI users] independent startup of orted and orterun

[Ralph Castain]

Appreciate your patience! I'm somewhat limited this week by being on travel
to our HQ, so I don't have access to my usual test cluster. I'll be better
situated to complete the implementation once I get home.

For now, some quick thoughts:

1. stdout/stderr: yes, I just need to "register" orte-submit as the one to
receive those from the submitted job.

2. That one is going to be a tad trickier, but is resolvable. May take me a
little longer to fix.

3. dang - I thought I had it doing so. I'll look to find the issue. I
suspect it's just a case of correctly setting the return code of
orte-submit.

I'd welcome the help! Let me ponder the best way to point you to the areas
needing work, and we can kick around off-list about who does what.

Great to hear this is working with your tool so quickly!!
Ralph


On Tue, Feb 3, 2015 at 3:49 PM, Mark Santcroos 
wrote:

> Hi Ralph,
>
> Besides the items in the other mail, I have three more items that would
> need resolving at some point.
>
> 1. STDOUT/STDERR currently go to the orte-dvm console.
>I'm sure this is not a fundamental limitation.
>Even if getting the information to the orte-submit instance would be
> problematic, the orte-dvm writing this to a file per session would be good
> enough too.
>
> 2. Failing applications currently tear down the dvm.
>Ideally that would not be the case, and this would be handled in
> relation to item (3).
>Possibly this needs to be configurable, if others would like to see
> different behaviour.
>
> 3. orte-submit doesn't return the exit code of the application.
>
> To be clear, I realise the current implementation is a proof of concept,
> so these are no complaints, just wishes of where I hope to see this going!
>
> FWIW: these items might require less intricate knowledge of OMPI in
> general, so with some pointers/guidance I can probably work on these myself
> if needed.
>
> Cheers,
>
> Mark
>
> ps. I did a quick-and-dirty integration with our own tool and the ORTE
> abstraction maps like a charm!
> (
> https://github.com/radical-cybertools/radical.pilot/commit/2d36e886081bf8531097edfc95ada1826257e460
> )
>
> > On 03 Feb 2015, at 20:38 , Mark Santcroos 
> wrote:
> >
> > Hi Ralph,
> >
> >> On 03 Feb 2015, at 16:28 , Ralph Castain  wrote:
> >> I think I fixed some of the handshake issues - please give it another
> try.
> >> You should see orte-submit properly shutdown upon completion,
> >
> > Indeed, it works on my laptop now! Great!
> > It feels quite fast too, for sort tasks :-)
> >
> >> and orte-dvm properly shutdown when sent the terminate cmd.
> >
> > ACK. This also works as expected.
> >
> >> I was able to cleanly run MPI jobs on my laptop.
> >
> > Do you also see the following errors/warnings on the dvm side?
> >
> > [netbook:28324] [[20896,0],0] Releasing job data for [INVALID]
> > Hello, world, I am 0 of 1, (Open MPI v1.9a1, package: Open MPI
> mark@netbook Distribution, ident: 1.9.0a1, repo rev: dev-811-g7299cc3,
> Unreleased developer copy, 132)
> > [netbook:28324] sess_dir_finalize: proc session dir does not exist
> > [netbook:28324] [[20896,0],0] dvm: job [20896,20] has completed
> > [netbook:28324] [[20896,0],0] Releasing job data for [20896,20]
> >
> > The "INVALID" message is there for every "submit", the sess_dir_finalize
> exists per instance/core.
> > Is that something to worry about, that needs fixing or is that a
> configuration issue?
> >
> > I haven't been able to test on Edison because of maintenance
> (today+tomorrow), so I will report on that later.
> >
> > Thanks again!
> >
> > Mark
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2015/02/26282.php
>

Re: [OMPI users] independent startup of orted and orterun

[Ralph Castain]

Hmmmno, I wasn't seeing those warnings/errors, but I only ran one
submit job. I'll investigate.

On Tue, Feb 3, 2015 at 11:38 AM, Mark Santcroos 
wrote:

> Hi Ralph,
>
> > On 03 Feb 2015, at 16:28 , Ralph Castain  wrote:
> > I think I fixed some of the handshake issues - please give it another
> try.
> > You should see orte-submit properly shutdown upon completion,
>
> Indeed, it works on my laptop now! Great!
> It feels quite fast too, for sort tasks :-)
>
> > and orte-dvm properly shutdown when sent the terminate cmd.
>
> ACK. This also works as expected.
>
> > I was able to cleanly run MPI jobs on my laptop.
>
> Do you also see the following errors/warnings on the dvm side?
>
> [netbook:28324] [[20896,0],0] Releasing job data for [INVALID]
> Hello, world, I am 0 of 1, (Open MPI v1.9a1, package: Open MPI mark@netbook
> Distribution, ident: 1.9.0a1, repo rev: dev-811-g7299cc3, Unreleased
> developer copy, 132)
> [netbook:28324] sess_dir_finalize: proc session dir does not exist
> [netbook:28324] [[20896,0],0] dvm: job [20896,20] has completed
> [netbook:28324] [[20896,0],0] Releasing job data for [20896,20]
>
> The "INVALID" message is there for every "submit", the sess_dir_finalize
> exists per instance/core.
> Is that something to worry about, that needs fixing or is that a
> configuration issue?
>
> I haven't been able to test on Edison because of maintenance
> (today+tomorrow), so I will report on that later.
>
> Thanks again!
>
> Mark
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2015/02/26278.php
>

Re: [OMPI users] independent startup of orted and orterun

[Mark Santcroos]

Hi Ralph,

Besides the items in the other mail, I have three more items that would need 
resolving at some point.

1. STDOUT/STDERR currently go to the orte-dvm console.
   I'm sure this is not a fundamental limitation.
   Even if getting the information to the orte-submit instance would be 
problematic, the orte-dvm writing this to a file per session would be good 
enough too.

2. Failing applications currently tear down the dvm.
   Ideally that would not be the case, and this would be handled in relation to 
item (3).
   Possibly this needs to be configurable, if others would like to see 
different behaviour.

3. orte-submit doesn't return the exit code of the application.

To be clear, I realise the current implementation is a proof of concept, so 
these are no complaints, just wishes of where I hope to see this going!

FWIW: these items might require less intricate knowledge of OMPI in general, so 
with some pointers/guidance I can probably work on these myself if needed.

Cheers,

Mark 

ps. I did a quick-and-dirty integration with our own tool and the ORTE 
abstraction maps like a charm!

(https://github.com/radical-cybertools/radical.pilot/commit/2d36e886081bf8531097edfc95ada1826257e460)

> On 03 Feb 2015, at 20:38 , Mark Santcroos  wrote:
> 
> Hi Ralph,
> 
>> On 03 Feb 2015, at 16:28 , Ralph Castain  wrote:
>> I think I fixed some of the handshake issues - please give it another try.
>> You should see orte-submit properly shutdown upon completion,
> 
> Indeed, it works on my laptop now! Great!
> It feels quite fast too, for sort tasks :-)
> 
>> and orte-dvm properly shutdown when sent the terminate cmd.
> 
> ACK. This also works as expected.
> 
>> I was able to cleanly run MPI jobs on my laptop.
> 
> Do you also see the following errors/warnings on the dvm side?
> 
> [netbook:28324] [[20896,0],0] Releasing job data for [INVALID]
> Hello, world, I am 0 of 1, (Open MPI v1.9a1, package: Open MPI mark@netbook 
> Distribution, ident: 1.9.0a1, repo rev: dev-811-g7299cc3, Unreleased 
> developer copy, 132)
> [netbook:28324] sess_dir_finalize: proc session dir does not exist
> [netbook:28324] [[20896,0],0] dvm: job [20896,20] has completed
> [netbook:28324] [[20896,0],0] Releasing job data for [20896,20]
> 
> The "INVALID" message is there for every "submit", the sess_dir_finalize 
> exists per instance/core.
> Is that something to worry about, that needs fixing or is that a 
> configuration issue?
> 
> I haven't been able to test on Edison because of maintenance 
> (today+tomorrow), so I will report on that later.
> 
> Thanks again!
> 
> Mark

Re: [OMPI users] independent startup of orted and orterun

[Mark Santcroos]

Hi Ralph,

> On 03 Feb 2015, at 16:28 , Ralph Castain  wrote:
> I think I fixed some of the handshake issues - please give it another try.
> You should see orte-submit properly shutdown upon completion,

Indeed, it works on my laptop now! Great!
It feels quite fast too, for sort tasks :-)

> and orte-dvm properly shutdown when sent the terminate cmd.

ACK. This also works as expected.

> I was able to cleanly run MPI jobs on my laptop.

Do you also see the following errors/warnings on the dvm side?

[netbook:28324] [[20896,0],0] Releasing job data for [INVALID]
Hello, world, I am 0 of 1, (Open MPI v1.9a1, package: Open MPI mark@netbook 
Distribution, ident: 1.9.0a1, repo rev: dev-811-g7299cc3, Unreleased developer 
copy, 132)
[netbook:28324] sess_dir_finalize: proc session dir does not exist
[netbook:28324] [[20896,0],0] dvm: job [20896,20] has completed
[netbook:28324] [[20896,0],0] Releasing job data for [20896,20]

The "INVALID" message is there for every "submit", the sess_dir_finalize exists 
per instance/core.
Is that something to worry about, that needs fixing or is that a configuration 
issue?

I haven't been able to test on Edison because of maintenance (today+tomorrow), 
so I will report on that later.

Thanks again!

Mark

Re: [OMPI users] independent startup of orted and orterun

[Ralph Castain]

I think I fixed some of the handshake issues - please give it another try.
You should see orte-submit properly shutdown upon completion, and orte-dvm
properly shutdown when sent the terminate cmd. I was able to cleanly run
MPI jobs on my laptop.

On Mon, Feb 2, 2015 at 10:44 PM, Mark Santcroos 
wrote:

> On 03 Feb 2015, at 0:20 , Ralph Castain  wrote:
> > Okay, thanks - I'll get on it tonight. Looks like a fairly simple bug,
> so hopefully I'll have it ironed out tonight.
>
> Sorry, I was not completely accurate. Let me be more specific:
>
> * The orte-submit does not return though, so that symptom is similar.
>
> * On my laptop I dont get the libevent warnings (the output redirection
> hid that fact).
>
> * The "-np" behavior is different too, but I guess that is because there
> is no RM in place on my laptop.
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2015/02/26263.php
>

Re: [OMPI users] independent startup of orted and orterun

[Mark Santcroos]

On 03 Feb 2015, at 0:20 , Ralph Castain  wrote:
> Okay, thanks - I'll get on it tonight. Looks like a fairly simple bug, so 
> hopefully I'll have it ironed out tonight.

Sorry, I was not completely accurate. Let me be more specific:

* The orte-submit does not return though, so that symptom is similar.

* On my laptop I dont get the libevent warnings (the output redirection hid 
that fact).

* The "-np" behavior is different too, but I guess that is because there is no 
RM in place on my laptop.

Re: [OMPI users] independent startup of orted and orterun

[Ralph Castain]

Okay, thanks - I'll get on it tonight. Looks like a fairly simple bug, so
hopefully I'll have it ironed out tonight.

On Mon, Feb 2, 2015 at 1:40 PM, Mark Santcroos 
wrote:

> FWIW: I see similar behaviour on my laptop (OS X Yosemite 10.10.2).
>
> > On 02 Feb 2015, at 21:26 , Mark Santcroos 
> wrote:
> >
> > Ok, let me check on some other systems too though, it might be Cray
> specific.
> >
> >
> >> On 02 Feb 2015, at 19:07 , Ralph Castain  wrote:
> >>
> >> Yikes - looks like a bug crept into there at the last minute. I
> actually had it working just fine - not sure what happened here. I'm on
> travel this week, but I'll try to dig into this a bit and spot the issue.
> >>
> >> Thanks!
> >> Ralph
> >>
> >>
> >> On Mon, Feb 2, 2015 at 3:50 AM, Mark Santcroos <
> mark.santcr...@rutgers.edu> wrote:
> >> Hi Ralph,
> >>
> >> Great, the semantics look exactly as what I need!
> >>
> >> (To aid in debugging I added "--debug-devel" to orte-dvm.c which was
> useful to detect and come by some initial bumps)
> >>
> >> The current status:
> >>
> >> * I can submit applications and see their output on the orte-dvm console
> >>
> >> * The following message is reported infinitely on the orte-submit
> console:
> >>
> >> [warn] opal_libevent2022_event_base_loop: reentrant invocation.  Only
> one event_base_loop can run on each event_base at once.
> >>
> >> * orte-submit doesn't return, while I see "[nid02819:20571]
> [[2120,0],0] dvm: job [2120,9] has completed" on the orte-dvm console.
> >>
> >> * On the orte-dvm console I see the following when submitting (so also
> for "successful" runs):
> >>
> >> [nid02434:00564] [[9021,0],0] Releasing job data for [INVALID]
> >> [nid03388:26474] [[9021,0],2] ORTE_ERROR_LOG: Not found in file
> ../../../../orte/mca/odls/base/odls_base_default_fns.c at line 433
> >> [nid03534:31545] procdir: /tmp/openmpi-sessions-62758@nid03534_0
> /9021/1/0
> >> [nid03534:31545] jobdir: /tmp/openmpi-sessions-62758@nid03534_0/9021/1
> >> [nid03534:31545] top: openmpi-sessions-62758@nid03534_0
> >> [nid03534:31545] tmp: /tmp
> >> [nid03534:31545] sess_dir_finalize: proc session dir does not exist
> >>
> >> * If I dont specify any "-np" on the orte-submit, then I see on the
> orte-dvm console:
> >>
> >> [nid02434:00564] [[9021,0],0] Releasing job data for [INVALID]
> >> [nid03388:26474] [[9021,0],2] ORTE_ERROR_LOG: Not found in file
> ../../../../orte/mca/odls/base/odls_base_default_fns.c at line 433
> >> [nid03534:31544] [[9021,0],1] ORTE_ERROR_LOG: Not found in file
> ../../../../orte/mca/odls/base/odls_base_default_fns.c at line 433
> >>
> >> * It only seems to work for single nodes (probably related to the
> previous point).
> >>
> >>
> >> Is this all expected behaviour given the current implementation?
> >>
> >>
> >> Thanks!
> >>
> >> Mark
> >>
> >>
> >>
> >>> On 02 Feb 2015, at 4:21 , Ralph Castain  wrote:
> >>>
> >>> I have pushed the changes to the OMPI master. It took a little bit
> more than I had hoped due to the changes to the ORTE infrastructure, but
> hopefully this will meet your needs. It consists of two new tools:
> >>>
> >>> (a) orte-dvm - starts the virtual machine by launching a daemon on
> every node of the allocation, as constrained by -host and/or -hostfile.
> Check the options for outputting the URI as you’ll need that info for the
> other tool. The DVM remains “up” until you issue the orte-submit -terminate
> command, or hit the orte-dvm process with a sigterm.
> >>>
> >>> (b) orte-submit - takes the place of mpirun. Basically just packages
> your app and arguments and sends it to orte-dvm for execution. Requires the
> URI of orte-dvm. The tool exits once the job has completed execution,
> though you can run multiple jobs in parallel by backgrounding orte-submit
> or issuing commands from separate shells.
> >>>
> >>> I’ve added man pages for both tools, though they may not be complete.
> Also, I don’t have all the mapping/ranking/binding options supported just
> yet as I first wanted to see if this meets your basic needs before worrying
> about the detail.
> >>>
> >>> Let me know what you think
> >>> Ralph
> >>>
> >>>
>  On Jan 21, 2015, at 4:07 PM, Mark Santcroos <
> mark.santcr...@rutgers.edu> wrote:
> 
>  Hi Ralph,
> 
>  All makes sense! Thanks a lot!
> 
>  Looking forward to your modifications.
>  Please don't hesitate to through things with rough-edges to me!
> 
>  Cheers,
> 
>  Mark
> 
> > On 21 Jan 2015, at 23:21 , Ralph Castain  wrote:
> >
> > Let me address your questions up here so you don’t have to scan thru
> the entire note.
> >
> > PMIx rationale: PMI has been around for a long time, primarily used
> inside the MPI library implementations to perform wireup. It provided a
> link from the MPI library to the local resource manager. However, as we
> move towards exascale, two things became 

Re: [OMPI users] independent startup of orted and orterun

[Mark Santcroos]

FWIW: I see similar behaviour on my laptop (OS X Yosemite 10.10.2).

> On 02 Feb 2015, at 21:26 , Mark Santcroos  wrote:
> 
> Ok, let me check on some other systems too though, it might be Cray specific.
> 
> 
>> On 02 Feb 2015, at 19:07 , Ralph Castain  wrote:
>> 
>> Yikes - looks like a bug crept into there at the last minute. I actually had 
>> it working just fine - not sure what happened here. I'm on travel this week, 
>> but I'll try to dig into this a bit and spot the issue.
>> 
>> Thanks!
>> Ralph
>> 
>> 
>> On Mon, Feb 2, 2015 at 3:50 AM, Mark Santcroos  
>> wrote:
>> Hi Ralph,
>> 
>> Great, the semantics look exactly as what I need!
>> 
>> (To aid in debugging I added "--debug-devel" to orte-dvm.c which was useful 
>> to detect and come by some initial bumps)
>> 
>> The current status:
>> 
>> * I can submit applications and see their output on the orte-dvm console
>> 
>> * The following message is reported infinitely on the orte-submit console:
>> 
>> [warn] opal_libevent2022_event_base_loop: reentrant invocation.  Only one 
>> event_base_loop can run on each event_base at once.
>> 
>> * orte-submit doesn't return, while I see "[nid02819:20571] [[2120,0],0] 
>> dvm: job [2120,9] has completed" on the orte-dvm console.
>> 
>> * On the orte-dvm console I see the following when submitting (so also for 
>> "successful" runs):
>> 
>> [nid02434:00564] [[9021,0],0] Releasing job data for [INVALID]
>> [nid03388:26474] [[9021,0],2] ORTE_ERROR_LOG: Not found in file 
>> ../../../../orte/mca/odls/base/odls_base_default_fns.c at line 433
>> [nid03534:31545] procdir: /tmp/openmpi-sessions-62758@nid03534_0/9021/1/0
>> [nid03534:31545] jobdir: /tmp/openmpi-sessions-62758@nid03534_0/9021/1
>> [nid03534:31545] top: openmpi-sessions-62758@nid03534_0
>> [nid03534:31545] tmp: /tmp
>> [nid03534:31545] sess_dir_finalize: proc session dir does not exist
>> 
>> * If I dont specify any "-np" on the orte-submit, then I see on the orte-dvm 
>> console:
>> 
>> [nid02434:00564] [[9021,0],0] Releasing job data for [INVALID]
>> [nid03388:26474] [[9021,0],2] ORTE_ERROR_LOG: Not found in file 
>> ../../../../orte/mca/odls/base/odls_base_default_fns.c at line 433
>> [nid03534:31544] [[9021,0],1] ORTE_ERROR_LOG: Not found in file 
>> ../../../../orte/mca/odls/base/odls_base_default_fns.c at line 433
>> 
>> * It only seems to work for single nodes (probably related to the previous 
>> point).
>> 
>> 
>> Is this all expected behaviour given the current implementation?
>> 
>> 
>> Thanks!
>> 
>> Mark
>> 
>> 
>> 
>>> On 02 Feb 2015, at 4:21 , Ralph Castain  wrote:
>>> 
>>> I have pushed the changes to the OMPI master. It took a little bit more 
>>> than I had hoped due to the changes to the ORTE infrastructure, but 
>>> hopefully this will meet your needs. It consists of two new tools:
>>> 
>>> (a) orte-dvm - starts the virtual machine by launching a daemon on every 
>>> node of the allocation, as constrained by -host and/or -hostfile. Check the 
>>> options for outputting the URI as you’ll need that info for the other tool. 
>>> The DVM remains “up” until you issue the orte-submit -terminate command, or 
>>> hit the orte-dvm process with a sigterm.
>>> 
>>> (b) orte-submit - takes the place of mpirun. Basically just packages your 
>>> app and arguments and sends it to orte-dvm for execution. Requires the URI 
>>> of orte-dvm. The tool exits once the job has completed execution, though 
>>> you can run multiple jobs in parallel by backgrounding orte-submit or 
>>> issuing commands from separate shells.
>>> 
>>> I’ve added man pages for both tools, though they may not be complete. Also, 
>>> I don’t have all the mapping/ranking/binding options supported just yet as 
>>> I first wanted to see if this meets your basic needs before worrying about 
>>> the detail.
>>> 
>>> Let me know what you think
>>> Ralph
>>> 
>>> 
 On Jan 21, 2015, at 4:07 PM, Mark Santcroos  
 wrote:
 
 Hi Ralph,
 
 All makes sense! Thanks a lot!
 
 Looking forward to your modifications.
 Please don't hesitate to through things with rough-edges to me!
 
 Cheers,
 
 Mark
 
> On 21 Jan 2015, at 23:21 , Ralph Castain  wrote:
> 
> Let me address your questions up here so you don’t have to scan thru the 
> entire note.
> 
> PMIx rationale: PMI has been around for a long time, primarily used 
> inside the MPI library implementations to perform wireup. It provided a 
> link from the MPI library to the local resource manager. However, as we 
> move towards exascale, two things became apparent:
> 
> 1. the current PMI implementations don’t scale adequately to get there. 
> The API created too many communications and assumed everything was a 
> blocking operation, thus preventing asynchronous progress
> 
> 2. there were 

Re: [OMPI users] independent startup of orted and orterun

[Mark Santcroos]

Ok, let me check on some other systems too though, it might be Cray specific.


> On 02 Feb 2015, at 19:07 , Ralph Castain  wrote:
> 
> Yikes - looks like a bug crept into there at the last minute. I actually had 
> it working just fine - not sure what happened here. I'm on travel this week, 
> but I'll try to dig into this a bit and spot the issue.
> 
> Thanks!
> Ralph
> 
> 
> On Mon, Feb 2, 2015 at 3:50 AM, Mark Santcroos  
> wrote:
> Hi Ralph,
> 
> Great, the semantics look exactly as what I need!
> 
> (To aid in debugging I added "--debug-devel" to orte-dvm.c which was useful 
> to detect and come by some initial bumps)
> 
> The current status:
> 
> * I can submit applications and see their output on the orte-dvm console
> 
> * The following message is reported infinitely on the orte-submit console:
> 
> [warn] opal_libevent2022_event_base_loop: reentrant invocation.  Only one 
> event_base_loop can run on each event_base at once.
> 
> * orte-submit doesn't return, while I see "[nid02819:20571] [[2120,0],0] dvm: 
> job [2120,9] has completed" on the orte-dvm console.
> 
> * On the orte-dvm console I see the following when submitting (so also for 
> "successful" runs):
> 
> [nid02434:00564] [[9021,0],0] Releasing job data for [INVALID]
> [nid03388:26474] [[9021,0],2] ORTE_ERROR_LOG: Not found in file 
> ../../../../orte/mca/odls/base/odls_base_default_fns.c at line 433
> [nid03534:31545] procdir: /tmp/openmpi-sessions-62758@nid03534_0/9021/1/0
> [nid03534:31545] jobdir: /tmp/openmpi-sessions-62758@nid03534_0/9021/1
> [nid03534:31545] top: openmpi-sessions-62758@nid03534_0
> [nid03534:31545] tmp: /tmp
> [nid03534:31545] sess_dir_finalize: proc session dir does not exist
> 
> * If I dont specify any "-np" on the orte-submit, then I see on the orte-dvm 
> console:
> 
> [nid02434:00564] [[9021,0],0] Releasing job data for [INVALID]
> [nid03388:26474] [[9021,0],2] ORTE_ERROR_LOG: Not found in file 
> ../../../../orte/mca/odls/base/odls_base_default_fns.c at line 433
> [nid03534:31544] [[9021,0],1] ORTE_ERROR_LOG: Not found in file 
> ../../../../orte/mca/odls/base/odls_base_default_fns.c at line 433
> 
> * It only seems to work for single nodes (probably related to the previous 
> point).
> 
> 
> Is this all expected behaviour given the current implementation?
> 
> 
> Thanks!
> 
> Mark
> 
> 
> 
> > On 02 Feb 2015, at 4:21 , Ralph Castain  wrote:
> >
> > I have pushed the changes to the OMPI master. It took a little bit more 
> > than I had hoped due to the changes to the ORTE infrastructure, but 
> > hopefully this will meet your needs. It consists of two new tools:
> >
> > (a) orte-dvm - starts the virtual machine by launching a daemon on every 
> > node of the allocation, as constrained by -host and/or -hostfile. Check the 
> > options for outputting the URI as you’ll need that info for the other tool. 
> > The DVM remains “up” until you issue the orte-submit -terminate command, or 
> > hit the orte-dvm process with a sigterm.
> >
> > (b) orte-submit - takes the place of mpirun. Basically just packages your 
> > app and arguments and sends it to orte-dvm for execution. Requires the URI 
> > of orte-dvm. The tool exits once the job has completed execution, though 
> > you can run multiple jobs in parallel by backgrounding orte-submit or 
> > issuing commands from separate shells.
> >
> > I’ve added man pages for both tools, though they may not be complete. Also, 
> > I don’t have all the mapping/ranking/binding options supported just yet as 
> > I first wanted to see if this meets your basic needs before worrying about 
> > the detail.
> >
> > Let me know what you think
> > Ralph
> >
> >
> >> On Jan 21, 2015, at 4:07 PM, Mark Santcroos  
> >> wrote:
> >>
> >> Hi Ralph,
> >>
> >> All makes sense! Thanks a lot!
> >>
> >> Looking forward to your modifications.
> >> Please don't hesitate to through things with rough-edges to me!
> >>
> >> Cheers,
> >>
> >> Mark
> >>
> >>> On 21 Jan 2015, at 23:21 , Ralph Castain  wrote:
> >>>
> >>> Let me address your questions up here so you don’t have to scan thru the 
> >>> entire note.
> >>>
> >>> PMIx rationale: PMI has been around for a long time, primarily used 
> >>> inside the MPI library implementations to perform wireup. It provided a 
> >>> link from the MPI library to the local resource manager. However, as we 
> >>> move towards exascale, two things became apparent:
> >>>
> >>> 1. the current PMI implementations don’t scale adequately to get there. 
> >>> The API created too many communications and assumed everything was a 
> >>> blocking operation, thus preventing asynchronous progress
> >>>
> >>> 2. there were increasing requests for application-level interactions to 
> >>> the resource manager. People want ways to spawn jobs (and not just from 
> >>> within MPI), request pre-location of data, control power, etc. Rather 
> >>> than having every RM 

Re: [OMPI users] independent startup of orted and orterun

[Ralph Castain]

Yikes - looks like a bug crept into there at the last minute. I actually
had it working just fine - not sure what happened here. I'm on travel this
week, but I'll try to dig into this a bit and spot the issue.

Thanks!
Ralph


On Mon, Feb 2, 2015 at 3:50 AM, Mark Santcroos 
wrote:

> Hi Ralph,
>
> Great, the semantics look exactly as what I need!
>
> (To aid in debugging I added "--debug-devel" to orte-dvm.c which was
> useful to detect and come by some initial bumps)
>
> The current status:
>
> * I can submit applications and see their output on the orte-dvm console
>
> * The following message is reported infinitely on the orte-submit console:
>
> [warn] opal_libevent2022_event_base_loop: reentrant invocation.  Only one
> event_base_loop can run on each event_base at once.
>
> * orte-submit doesn't return, while I see "[nid02819:20571] [[2120,0],0]
> dvm: job [2120,9] has completed" on the orte-dvm console.
>
> * On the orte-dvm console I see the following when submitting (so also for
> "successful" runs):
>
> [nid02434:00564] [[9021,0],0] Releasing job data for [INVALID]
> [nid03388:26474] [[9021,0],2] ORTE_ERROR_LOG: Not found in file
> ../../../../orte/mca/odls/base/odls_base_default_fns.c at line 433
> [nid03534:31545] procdir: /tmp/openmpi-sessions-62758@nid03534_0/9021/1/0
> [nid03534:31545] jobdir: /tmp/openmpi-sessions-62758@nid03534_0/9021/1
> [nid03534:31545] top: openmpi-sessions-62758@nid03534_0
> [nid03534:31545] tmp: /tmp
> [nid03534:31545] sess_dir_finalize: proc session dir does not exist
>
> * If I dont specify any "-np" on the orte-submit, then I see on the
> orte-dvm console:
>
> [nid02434:00564] [[9021,0],0] Releasing job data for [INVALID]
> [nid03388:26474] [[9021,0],2] ORTE_ERROR_LOG: Not found in file
> ../../../../orte/mca/odls/base/odls_base_default_fns.c at line 433
> [nid03534:31544] [[9021,0],1] ORTE_ERROR_LOG: Not found in file
> ../../../../orte/mca/odls/base/odls_base_default_fns.c at line 433
>
> * It only seems to work for single nodes (probably related to the previous
> point).
>
>
> Is this all expected behaviour given the current implementation?
>
>
> Thanks!
>
> Mark
>
>
>
> > On 02 Feb 2015, at 4:21 , Ralph Castain  wrote:
> >
> > I have pushed the changes to the OMPI master. It took a little bit more
> than I had hoped due to the changes to the ORTE infrastructure, but
> hopefully this will meet your needs. It consists of two new tools:
> >
> > (a) orte-dvm - starts the virtual machine by launching a daemon on every
> node of the allocation, as constrained by -host and/or -hostfile. Check the
> options for outputting the URI as you’ll need that info for the other tool.
> The DVM remains “up” until you issue the orte-submit -terminate command, or
> hit the orte-dvm process with a sigterm.
> >
> > (b) orte-submit - takes the place of mpirun. Basically just packages
> your app and arguments and sends it to orte-dvm for execution. Requires the
> URI of orte-dvm. The tool exits once the job has completed execution,
> though you can run multiple jobs in parallel by backgrounding orte-submit
> or issuing commands from separate shells.
> >
> > I’ve added man pages for both tools, though they may not be complete.
> Also, I don’t have all the mapping/ranking/binding options supported just
> yet as I first wanted to see if this meets your basic needs before worrying
> about the detail.
> >
> > Let me know what you think
> > Ralph
> >
> >
> >> On Jan 21, 2015, at 4:07 PM, Mark Santcroos 
> wrote:
> >>
> >> Hi Ralph,
> >>
> >> All makes sense! Thanks a lot!
> >>
> >> Looking forward to your modifications.
> >> Please don't hesitate to through things with rough-edges to me!
> >>
> >> Cheers,
> >>
> >> Mark
> >>
> >>> On 21 Jan 2015, at 23:21 , Ralph Castain  wrote:
> >>>
> >>> Let me address your questions up here so you don’t have to scan thru
> the entire note.
> >>>
> >>> PMIx rationale: PMI has been around for a long time, primarily used
> inside the MPI library implementations to perform wireup. It provided a
> link from the MPI library to the local resource manager. However, as we
> move towards exascale, two things became apparent:
> >>>
> >>> 1. the current PMI implementations don’t scale adequately to get
> there. The API created too many communications and assumed everything was a
> blocking operation, thus preventing asynchronous progress
> >>>
> >>> 2. there were increasing requests for application-level interactions
> to the resource manager. People want ways to spawn jobs (and not just from
> within MPI), request pre-location of data, control power, etc. Rather than
> having every RM write its own interface (and thus make everyone’s code
> non-portable), we at Intel decided to extend the existing PMI definitions
> to support those functions. Thus, an application developer can directly
> access PMIx functions to perform all those operations.
> >>>
> >>> PMIx v1.0 is about 

Re: [OMPI users] independent startup of orted and orterun

[Mark Santcroos]

Hi Ralph,

Great, the semantics look exactly as what I need!

(To aid in debugging I added "--debug-devel" to orte-dvm.c which was useful to 
detect and come by some initial bumps)

The current status:

* I can submit applications and see their output on the orte-dvm console

* The following message is reported infinitely on the orte-submit console:

[warn] opal_libevent2022_event_base_loop: reentrant invocation.  Only one 
event_base_loop can run on each event_base at once.

* orte-submit doesn't return, while I see "[nid02819:20571] [[2120,0],0] dvm: 
job [2120,9] has completed" on the orte-dvm console.

* On the orte-dvm console I see the following when submitting (so also for 
"successful" runs):

[nid02434:00564] [[9021,0],0] Releasing job data for [INVALID]
[nid03388:26474] [[9021,0],2] ORTE_ERROR_LOG: Not found in file 
../../../../orte/mca/odls/base/odls_base_default_fns.c at line 433
[nid03534:31545] procdir: /tmp/openmpi-sessions-62758@nid03534_0/9021/1/0
[nid03534:31545] jobdir: /tmp/openmpi-sessions-62758@nid03534_0/9021/1
[nid03534:31545] top: openmpi-sessions-62758@nid03534_0
[nid03534:31545] tmp: /tmp
[nid03534:31545] sess_dir_finalize: proc session dir does not exist

* If I dont specify any "-np" on the orte-submit, then I see on the orte-dvm 
console:

[nid02434:00564] [[9021,0],0] Releasing job data for [INVALID]
[nid03388:26474] [[9021,0],2] ORTE_ERROR_LOG: Not found in file 
../../../../orte/mca/odls/base/odls_base_default_fns.c at line 433
[nid03534:31544] [[9021,0],1] ORTE_ERROR_LOG: Not found in file 
../../../../orte/mca/odls/base/odls_base_default_fns.c at line 433

* It only seems to work for single nodes (probably related to the previous 
point).


Is this all expected behaviour given the current implementation?


Thanks!

Mark



> On 02 Feb 2015, at 4:21 , Ralph Castain  wrote:
> 
> I have pushed the changes to the OMPI master. It took a little bit more than 
> I had hoped due to the changes to the ORTE infrastructure, but hopefully this 
> will meet your needs. It consists of two new tools:
> 
> (a) orte-dvm - starts the virtual machine by launching a daemon on every node 
> of the allocation, as constrained by -host and/or -hostfile. Check the 
> options for outputting the URI as you’ll need that info for the other tool. 
> The DVM remains “up” until you issue the orte-submit -terminate command, or 
> hit the orte-dvm process with a sigterm.
> 
> (b) orte-submit - takes the place of mpirun. Basically just packages your app 
> and arguments and sends it to orte-dvm for execution. Requires the URI of 
> orte-dvm. The tool exits once the job has completed execution, though you can 
> run multiple jobs in parallel by backgrounding orte-submit or issuing 
> commands from separate shells.
> 
> I’ve added man pages for both tools, though they may not be complete. Also, I 
> don’t have all the mapping/ranking/binding options supported just yet as I 
> first wanted to see if this meets your basic needs before worrying about the 
> detail.
> 
> Let me know what you think
> Ralph
> 
> 
>> On Jan 21, 2015, at 4:07 PM, Mark Santcroos  
>> wrote:
>> 
>> Hi Ralph,
>> 
>> All makes sense! Thanks a lot!
>> 
>> Looking forward to your modifications.
>> Please don't hesitate to through things with rough-edges to me!
>> 
>> Cheers,
>> 
>> Mark
>> 
>>> On 21 Jan 2015, at 23:21 , Ralph Castain  wrote:
>>> 
>>> Let me address your questions up here so you don’t have to scan thru the 
>>> entire note.
>>> 
>>> PMIx rationale: PMI has been around for a long time, primarily used inside 
>>> the MPI library implementations to perform wireup. It provided a link from 
>>> the MPI library to the local resource manager. However, as we move towards 
>>> exascale, two things became apparent:
>>> 
>>> 1. the current PMI implementations don’t scale adequately to get there. The 
>>> API created too many communications and assumed everything was a blocking 
>>> operation, thus preventing asynchronous progress
>>> 
>>> 2. there were increasing requests for application-level interactions to the 
>>> resource manager. People want ways to spawn jobs (and not just from within 
>>> MPI), request pre-location of data, control power, etc. Rather than having 
>>> every RM write its own interface (and thus make everyone’s code 
>>> non-portable), we at Intel decided to extend the existing PMI definitions 
>>> to support those functions. Thus, an application developer can directly 
>>> access PMIx functions to perform all those operations.
>>> 
>>> PMIx v1.0 is about to be released - it’ll be backward compatible with PMI-1 
>>> and PMI-2, plus add non-blocking operations and significantly reduce the 
>>> number of communications. PMIx 2.0 is slated for this summer and will 
>>> include the advanced controls capabilities.
>>> 
>>> ORCM is being developed because we needed a BSD-licensed, fully featured 
>>> resource manager. This will allow us to integrate 

Re: [OMPI users] independent startup of orted and orterun

[Ralph Castain]

I have pushed the changes to the OMPI master. It took a little bit more than I 
had hoped due to the changes to the ORTE infrastructure, but hopefully this 
will meet your needs. It consists of two new tools:

(a) orte-dvm - starts the virtual machine by launching a daemon on every node 
of the allocation, as constrained by -host and/or -hostfile. Check the options 
for outputting the URI as you’ll need that info for the other tool. The DVM 
remains “up” until you issue the orte-submit -terminate command, or hit the 
orte-dvm process with a sigterm.

(b) orte-submit - takes the place of mpirun. Basically just packages your app 
and arguments and sends it to orte-dvm for execution. Requires the URI of 
orte-dvm. The tool exits once the job has completed execution, though you can 
run multiple jobs in parallel by backgrounding orte-submit or issuing commands 
from separate shells.

I’ve added man pages for both tools, though they may not be complete. Also, I 
don’t have all the mapping/ranking/binding options supported just yet as I 
first wanted to see if this meets your basic needs before worrying about the 
detail.

Let me know what you think
Ralph


> On Jan 21, 2015, at 4:07 PM, Mark Santcroos  
> wrote:
> 
> Hi Ralph,
> 
> All makes sense! Thanks a lot!
> 
> Looking forward to your modifications.
> Please don't hesitate to through things with rough-edges to me!
> 
> Cheers,
> 
> Mark
> 
>> On 21 Jan 2015, at 23:21 , Ralph Castain  wrote:
>> 
>> Let me address your questions up here so you don’t have to scan thru the 
>> entire note.
>> 
>> PMIx rationale: PMI has been around for a long time, primarily used inside 
>> the MPI library implementations to perform wireup. It provided a link from 
>> the MPI library to the local resource manager. However, as we move towards 
>> exascale, two things became apparent:
>> 
>> 1. the current PMI implementations don’t scale adequately to get there. The 
>> API created too many communications and assumed everything was a blocking 
>> operation, thus preventing asynchronous progress
>> 
>> 2. there were increasing requests for application-level interactions to the 
>> resource manager. People want ways to spawn jobs (and not just from within 
>> MPI), request pre-location of data, control power, etc. Rather than having 
>> every RM write its own interface (and thus make everyone’s code 
>> non-portable), we at Intel decided to extend the existing PMI definitions to 
>> support those functions. Thus, an application developer can directly access 
>> PMIx functions to perform all those operations.
>> 
>> PMIx v1.0 is about to be released - it’ll be backward compatible with PMI-1 
>> and PMI-2, plus add non-blocking operations and significantly reduce the 
>> number of communications. PMIx 2.0 is slated for this summer and will 
>> include the advanced controls capabilities.
>> 
>> ORCM is being developed because we needed a BSD-licensed, fully featured 
>> resource manager. This will allow us to integrate the RM even more tightly 
>> to the file system, networking, and other subsystems, thus achieving higher 
>> launch performance and providing desired features such as QoS management. 
>> PMIx is a part of that plan, but as you say, they each play their separate 
>> roles in the overall stack.
>> 
>> 
>> Persistent ORTE: there is a learning curve on ORTE, I fear. We do have some 
>> videos on the web site that can help get you started, and I’ve given a 
>> number of “classes" at Intel now for that purpose. I still have it on my 
>> “to-do” list that I summarize those classes and post them on the web site.
>> 
>> For now, let me summarize how things work. At startup, mpirun reads the 
>> allocation (usually from the environment, but it depends on the host RM) and 
>> launches a daemon on each allocated node. Each daemon reads its local 
>> hardware environment and “phones home” to let mpirun know it is alive. Once 
>> all daemons have reported, mpirun maps the processes to the nodes and sends 
>> that map to all the daemons in a scalable broadcast pattern.
>> 
>> Upon receipt of the launch message, each daemon parses it to identify which 
>> procs it needs to locally spawn. Once spawned, each proc connects back to 
>> its local daemon via a Unix domain socket for wireup support. As procs 
>> complete, the daemon maintains bookkeeping and reports back to mpirun once 
>> all procs are done. When all procs are reported complete (or one reports as 
>> abnormally terminated), mpirun sends a “die” message to every daemon so it 
>> will cleanly terminate.
>> 
>> What I will do is simply tell mpirun to not do that last step, but instead 
>> to wait to receive a “terminate” cmd before ending the daemons. This will 
>> allow you to reuse the existing DVM, making each independent job start a 
>> great deal faster. You’ll need to either manually terminate the DVM, or the 
>> RM will do so when the allocation expires.
>> 
>> HTH
>> 

Re: [OMPI users] independent startup of orted and orterun

[Mark Santcroos]

Hi Ralph,

All makes sense! Thanks a lot!

Looking forward to your modifications.
Please don't hesitate to through things with rough-edges to me!

Cheers,

Mark

> On 21 Jan 2015, at 23:21 , Ralph Castain  wrote:
> 
> Let me address your questions up here so you don’t have to scan thru the 
> entire note.
> 
> PMIx rationale: PMI has been around for a long time, primarily used inside 
> the MPI library implementations to perform wireup. It provided a link from 
> the MPI library to the local resource manager. However, as we move towards 
> exascale, two things became apparent:
> 
> 1. the current PMI implementations don’t scale adequately to get there. The 
> API created too many communications and assumed everything was a blocking 
> operation, thus preventing asynchronous progress
> 
> 2. there were increasing requests for application-level interactions to the 
> resource manager. People want ways to spawn jobs (and not just from within 
> MPI), request pre-location of data, control power, etc. Rather than having 
> every RM write its own interface (and thus make everyone’s code 
> non-portable), we at Intel decided to extend the existing PMI definitions to 
> support those functions. Thus, an application developer can directly access 
> PMIx functions to perform all those operations.
> 
> PMIx v1.0 is about to be released - it’ll be backward compatible with PMI-1 
> and PMI-2, plus add non-blocking operations and significantly reduce the 
> number of communications. PMIx 2.0 is slated for this summer and will include 
> the advanced controls capabilities.
> 
> ORCM is being developed because we needed a BSD-licensed, fully featured 
> resource manager. This will allow us to integrate the RM even more tightly to 
> the file system, networking, and other subsystems, thus achieving higher 
> launch performance and providing desired features such as QoS management. 
> PMIx is a part of that plan, but as you say, they each play their separate 
> roles in the overall stack.
> 
> 
> Persistent ORTE: there is a learning curve on ORTE, I fear. We do have some 
> videos on the web site that can help get you started, and I’ve given a number 
> of “classes" at Intel now for that purpose. I still have it on my “to-do” 
> list that I summarize those classes and post them on the web site.
> 
> For now, let me summarize how things work. At startup, mpirun reads the 
> allocation (usually from the environment, but it depends on the host RM) and 
> launches a daemon on each allocated node. Each daemon reads its local 
> hardware environment and “phones home” to let mpirun know it is alive. Once 
> all daemons have reported, mpirun maps the processes to the nodes and sends 
> that map to all the daemons in a scalable broadcast pattern.
> 
> Upon receipt of the launch message, each daemon parses it to identify which 
> procs it needs to locally spawn. Once spawned, each proc connects back to its 
> local daemon via a Unix domain socket for wireup support. As procs complete, 
> the daemon maintains bookkeeping and reports back to mpirun once all procs 
> are done. When all procs are reported complete (or one reports as abnormally 
> terminated), mpirun sends a “die” message to every daemon so it will cleanly 
> terminate.
> 
> What I will do is simply tell mpirun to not do that last step, but instead to 
> wait to receive a “terminate” cmd before ending the daemons. This will allow 
> you to reuse the existing DVM, making each independent job start a great deal 
> faster. You’ll need to either manually terminate the DVM, or the RM will do 
> so when the allocation expires.
> 
> HTH
> Ralph
> 
> 
>> On Jan 21, 2015, at 12:52 PM, Mark Santcroos  
>> wrote:
>> 
>> Hi Ralph,
>> 
>>> On 21 Jan 2015, at 21:20 , Ralph Castain  wrote:
>>> 
>>> Hi Mark
>>> 
 On Jan 21, 2015, at 11:21 AM, Mark Santcroos  
 wrote:
 
 Hi Ralph, all,
 
 To give some background, I'm part of the RADICAL-Pilot [1] development 
 team.
 RADICAL-Pilot is a Pilot System, an implementation of the Pilot (job) 
 concept, which is in its most minimal form takes care of the decoupling of 
 resource acquisition and workload management.
 So instead of launching your real_science.exe through PBS, you submit a 
 Pilot, which will allow you to perform application level scheduling.
 Most obvious use-case if you want to run many (relatively) small tasks, 
 then you really don;t want to go through the batch system every time. That 
 is besides the fact that these machines are very bad in managing many 
 tasks anyway.
>>> 
>>> Yeah, we sympathize.
>> 
>> Thats always good :-)
>> 
>>> Of course, one obvious solution is to get an allocation and execute a shell 
>>> script that runs the tasks within that allocation - yes?
>> 
>> Not really. Most of our use-cases have dynamic runtime properties, which 
>> means that at 

Re: [OMPI users] independent startup of orted and orterun

[Ralph Castain]

Let me address your questions up here so you don’t have to scan thru the entire 
note.

PMIx rationale: PMI has been around for a long time, primarily used inside the 
MPI library implementations to perform wireup. It provided a link from the MPI 
library to the local resource manager. However, as we move towards exascale, 
two things became apparent:

1. the current PMI implementations don’t scale adequately to get there. The API 
created too many communications and assumed everything was a blocking 
operation, thus preventing asynchronous progress

2. there were increasing requests for application-level interactions to the 
resource manager. People want ways to spawn jobs (and not just from within 
MPI), request pre-location of data, control power, etc. Rather than having 
every RM write its own interface (and thus make everyone’s code non-portable), 
we at Intel decided to extend the existing PMI definitions to support those 
functions. Thus, an application developer can directly access PMIx functions to 
perform all those operations.

PMIx v1.0 is about to be released - it’ll be backward compatible with PMI-1 and 
PMI-2, plus add non-blocking operations and significantly reduce the number of 
communications. PMIx 2.0 is slated for this summer and will include the 
advanced controls capabilities.

ORCM is being developed because we needed a BSD-licensed, fully featured 
resource manager. This will allow us to integrate the RM even more tightly to 
the file system, networking, and other subsystems, thus achieving higher launch 
performance and providing desired features such as QoS management. PMIx is a 
part of that plan, but as you say, they each play their separate roles in the 
overall stack.


Persistent ORTE: there is a learning curve on ORTE, I fear. We do have some 
videos on the web site that can help get you started, and I’ve given a number 
of “classes" at Intel now for that purpose. I still have it on my “to-do” list 
that I summarize those classes and post them on the web site.

For now, let me summarize how things work. At startup, mpirun reads the 
allocation (usually from the environment, but it depends on the host RM) and 
launches a daemon on each allocated node. Each daemon reads its local hardware 
environment and “phones home” to let mpirun know it is alive. Once all daemons 
have reported, mpirun maps the processes to the nodes and sends that map to all 
the daemons in a scalable broadcast pattern.

Upon receipt of the launch message, each daemon parses it to identify which 
procs it needs to locally spawn. Once spawned, each proc connects back to its 
local daemon via a Unix domain socket for wireup support. As procs complete, 
the daemon maintains bookkeeping and reports back to mpirun once all procs are 
done. When all procs are reported complete (or one reports as abnormally 
terminated), mpirun sends a “die” message to every daemon so it will cleanly 
terminate.

What I will do is simply tell mpirun to not do that last step, but instead to 
wait to receive a “terminate” cmd before ending the daemons. This will allow 
you to reuse the existing DVM, making each independent job start a great deal 
faster. You’ll need to either manually terminate the DVM, or the RM will do so 
when the allocation expires.

HTH
Ralph


> On Jan 21, 2015, at 12:52 PM, Mark Santcroos  
> wrote:
> 
> Hi Ralph,
> 
>> On 21 Jan 2015, at 21:20 , Ralph Castain  wrote:
>> 
>> Hi Mark
>> 
>>> On Jan 21, 2015, at 11:21 AM, Mark Santcroos  
>>> wrote:
>>> 
>>> Hi Ralph, all,
>>> 
>>> To give some background, I'm part of the RADICAL-Pilot [1] development team.
>>> RADICAL-Pilot is a Pilot System, an implementation of the Pilot (job) 
>>> concept, which is in its most minimal form takes care of the decoupling of 
>>> resource acquisition and workload management.
>>> So instead of launching your real_science.exe through PBS, you submit a 
>>> Pilot, which will allow you to perform application level scheduling.
>>> Most obvious use-case if you want to run many (relatively) small tasks, 
>>> then you really don;t want to go through the batch system every time. That 
>>> is besides the fact that these machines are very bad in managing many tasks 
>>> anyway.
>> 
>> Yeah, we sympathize.
> 
> Thats always good :-)
> 
>> Of course, one obvious solution is to get an allocation and execute a shell 
>> script that runs the tasks within that allocation - yes?
> 
> Not really. Most of our use-cases have dynamic runtime properties, which 
> means that at t=0 the exact workload is not known.
> 
> In addition, I don't think such a script would allow me to work around the 
> aprun bottleneck, as I'm not aware of a way to start MPI tasks that span 
> multiple nodes from a Cray worker node.
> 
>>> I looked a bit better at ORCM and it clearly overlaps with what I want to 
>>> achieve.
>> 
>> Agreed. In ORCM, we allow a user to request a “session” that 

Re: [OMPI users] independent startup of orted and orterun

[Mark Santcroos]

Hi Ralph,

> On 21 Jan 2015, at 21:20 , Ralph Castain  wrote:
> 
> Hi Mark
> 
>> On Jan 21, 2015, at 11:21 AM, Mark Santcroos  
>> wrote:
>> 
>> Hi Ralph, all,
>> 
>> To give some background, I'm part of the RADICAL-Pilot [1] development team.
>> RADICAL-Pilot is a Pilot System, an implementation of the Pilot (job) 
>> concept, which is in its most minimal form takes care of the decoupling of 
>> resource acquisition and workload management.
>> So instead of launching your real_science.exe through PBS, you submit a 
>> Pilot, which will allow you to perform application level scheduling.
>> Most obvious use-case if you want to run many (relatively) small tasks, then 
>> you really don;t want to go through the batch system every time. That is 
>> besides the fact that these machines are very bad in managing many tasks 
>> anyway.
> 
> Yeah, we sympathize.

Thats always good :-)

> Of course, one obvious solution is to get an allocation and execute a shell 
> script that runs the tasks within that allocation - yes?

Not really. Most of our use-cases have dynamic runtime properties, which means 
that at t=0 the exact workload is not known.

In addition, I don't think such a script would allow me to work around the 
aprun bottleneck, as I'm not aware of a way to start MPI tasks that span 
multiple nodes from a Cray worker node.

>> I looked a bit better at ORCM and it clearly overlaps with what I want to 
>> achieve.
> 
> Agreed. In ORCM, we allow a user to request a “session” that results in 
> allocation of resources. Each session is given an “orchestrator” - the ORCM 
> “shepherd” daemon - responsible for executing the individual tasks across the 
> assigned allocation, and a collection of “lamb” daemons (one on each node of 
> the allocation) that forms a distributed VM. The orchestrator can execute the 
> tasks very quickly since it doesn’t have to go back to the scheduler, and we 
> allow it to do so according to any provided precedence requirement. Again, 
> for simplicity, a shell script is the default mechanism for submitting the 
> individual tasks.

Yeah, similar solution to a similar problem.
I noticed that Exascale is also part of the motivation? How does this relate to 
the pmix effort? Different part of the stack I guess.

>> One thing I noticed is that parts of it runs as root, why is that?
> 
> ORCM is a full resource manager, which means it has a scheduler (rudimentary 
> today) and boot-time daemons that must run as root so they can fork/exec the 
> session-level daemons (that run at the user level). The orchestrator and its 
> daemons all run at the user-level.

Ok. Our solution is user-space only, as one of our features is that we are able 
to run across different type of systems. Both approaches come with a tradeoff 
obviously.

>>> We used to have a cmd line option in ORTE for what you propose - it 
>>> wouldn’t be too hard to restore. Is there some reason to do so?
>> 
>> Can you point me to something that I could look for in the repo history, 
>> then I can see if it serves my purpose.
> 
> It would be back in the svn repo, I fear - would take awhile to hunt it down. 
> Basically, it just (a) started all the daemons to create a VM, and (b) told 
> mpirun to stick around as a persistent daemon. All subsequent calls to mpirun 
> would reference back to the persistent one, thus using it to launch the jobs 
> against the standing VM instead of starting a new one every time.

*nod* That's what I tried to do this afternoon actually with the 
"--ompi-server", but that was not meant to be.

> For ORCM, we just took that capability and expressed it as the “shepherd” 
> plus “lamb” daemon architecture described above.

ACK.

> If you don’t want to replace the base RM, then using ORTE to establish a 
> persistent VM is probably the way to go.

Indeed, thats what it sounds like. Plus that ORTE is generic enough that I can 
re-use it on other type of systems too.

> I can probably make it do that again fairly readily. We have a developer’s 
> meeting next week, which usually means I have some free time (during evenings 
> and topics I’m not involved with), so I can take a crack at this then if that 
> would be timely enough.

Happy to accept that offer. At this stage I'm not sure if I would want a CLI or 
would be more interested to be able to do this programmatically though.
Also more than willing to assist in any way I can.

I tried to see how it all worked, but because of the modular nature of ompi 
that was quite daunting. There is some learning curve I guess :-)
So it seems that mpirun is persistent, and opens up a listening port, then some 
orded's get launched that phone home.
From there I got lost in the MCA maze. How do the tasks get unto the compute 
nodes and started?

Thanks a lot again, I appreciate your help.

Cheers,

Mark

Re: [OMPI users] independent startup of orted and orterun

[Ralph Castain]

Hi Mark

> On Jan 21, 2015, at 11:21 AM, Mark Santcroos  
> wrote:
> 
> Hi Ralph, all,
> 
> To give some background, I'm part of the RADICAL-Pilot [1] development team.
> RADICAL-Pilot is a Pilot System, an implementation of the Pilot (job) 
> concept, which is in its most minimal form takes care of the decoupling of 
> resource acquisition and workload management.
> So instead of launching your real_science.exe through PBS, you submit a 
> Pilot, which will allow you to perform application level scheduling.
> Most obvious use-case if you want to run many (relatively) small tasks, then 
> you really don;t want to go through the batch system every time. That is 
> besides the fact that these machines are very bad in managing many tasks 
> anyway.

Yeah, we sympathize. Of course, one obvious solution is to get an allocation 
and execute a shell script that runs the tasks within that allocation - yes?

> 
> The recent discussion we had on Spawn() on Cray's also originates here.
> I want to free myself from having to use aprun for every task, and therefore 
> I am interested to see if ompi and/or orte can be the vehicle for that.
> 
>> On 21 Jan 2015, at 17:16 , Ralph Castain  wrote:
>> Theoretically, yes - see the ORCM project, which basically does what you ask.
>> The launch system in there isn’t fully implemented yet, but the fundamental 
>> idea is valid and supports some range of capability.
> 
> I looked a bit better at ORCM and it clearly overlaps with what I want to 
> achieve.

Agreed. In ORCM, we allow a user to request a “session” that results in 
allocation of resources. Each session is given an “orchestrator” - the ORCM 
“shepherd” daemon - responsible for executing the individual tasks across the 
assigned allocation, and a collection of “lamb” daemons (one on each node of 
the allocation) that forms a distributed VM. The orchestrator can execute the 
tasks very quickly since it doesn’t have to go back to the scheduler, and we 
allow it to do so according to any provided precedence requirement. Again, for 
simplicity, a shell script is the default mechanism for submitting the 
individual tasks.

> One thing I noticed is that parts of it runs as root, why is that?

ORCM is a full resource manager, which means it has a scheduler (rudimentary 
today) and boot-time daemons that must run as root so they can fork/exec the 
session-level daemons (that run at the user level). The orchestrator and its 
daemons all run at the user-level.

> 
>> We used to have a cmd line option in ORTE for what you propose - it wouldn’t 
>> be too hard to restore. Is there some reason to do so?
> 
> Can you point me to something that I could look for in the repo history, then 
> I can see if it serves my purpose.

It would be back in the svn repo, I fear - would take awhile to hunt it down. 
Basically, it just (a) started all the daemons to create a VM, and (b) told 
mpirun to stick around as a persistent daemon. All subsequent calls to mpirun 
would reference back to the persistent one, thus using it to launch the jobs 
against the standing VM instead of starting a new one every time.

For ORCM, we just took that capability and expressed it as the “shepherd” plus 
“lamb” daemon architecture described above. If you don’t want to replace the 
base RM, then using ORTE to establish a persistent VM is probably the way to go.

I can probably make it do that again fairly readily. We have a developer’s 
meeting next week, which usually means I have some free time (during evenings 
and topics I’m not involved with), so I can take a crack at this then if that 
would be timely enough.

Ralph

> 
> If you think there is enough to warrant looking in more detail at ORCM I'm 
> happy to do that too.
> 
> Cheers,
> 
> Mark
> 
> 1. https://github.com/radical-cybertools/radical.pilot
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2015/01/26225.php

Re: [OMPI users] independent startup of orted and orterun

[Mark Santcroos]

Hi Ralph, all,

To give some background, I'm part of the RADICAL-Pilot [1] development team.
RADICAL-Pilot is a Pilot System, an implementation of the Pilot (job) concept, 
which is in its most minimal form takes care of the decoupling of resource 
acquisition and workload management.
So instead of launching your real_science.exe through PBS, you submit a Pilot, 
which will allow you to perform application level scheduling.
Most obvious use-case if you want to run many (relatively) small tasks, then 
you really don;t want to go through the batch system every time. That is 
besides the fact that these machines are very bad in managing many tasks anyway.

The recent discussion we had on Spawn() on Cray's also originates here.
I want to free myself from having to use aprun for every task, and therefore I 
am interested to see if ompi and/or orte can be the vehicle for that.

> On 21 Jan 2015, at 17:16 , Ralph Castain  wrote:
> Theoretically, yes - see the ORCM project, which basically does what you ask.
> The launch system in there isn’t fully implemented yet, but the fundamental 
> idea is valid and supports some range of capability.

I looked a bit better at ORCM and it clearly overlaps with what I want to 
achieve.
One thing I noticed is that parts of it runs as root, why is that?

> We used to have a cmd line option in ORTE for what you propose - it wouldn’t 
> be too hard to restore. Is there some reason to do so?

Can you point me to something that I could look for in the repo history, then I 
can see if it serves my purpose.

If you think there is enough to warrant looking in more detail at ORCM I'm 
happy to do that too.

Cheers,

Mark

1. https://github.com/radical-cybertools/radical.pilot

Re: [OMPI users] independent startup of orted and orterun

[Ralph Castain]

Ah, I see - actually, MR+ is supported under ORTE. You just need to add the 
—staged option to the cmd line.

You then run your job like this:

mpirun —staged -n X my-mapper : -n Y my-reducer

We’ll wire together the output of the mappers to the input of the reducers and 
run them in an appropriate MR staged pattern.


> On Jan 21, 2015, at 8:23 AM, Mark Santcroos  
> wrote:
> 
> Hi Ralph,
> 
> I figured that MR+ had to require something similar too (assuming it actually 
> does use ORCM).
> 
> Let me study that wiki a bit and I will come back with a description of my 
> use-case.
> 
> Thanks!
> 
> Mark
> 
> 
>> On 21 Jan 2015, at 17:18 , Ralph Castain  wrote:
>> 
>> Sorry - should have included the link to the ORCM project:
>> 
>> https://github.com/open-mpi/orcm/wiki
>> 
>> 
>>> On Jan 21, 2015, at 8:16 AM, Ralph Castain  wrote:
>>> 
>>> Theoretically, yes - see the ORCM project, which basically does what you 
>>> ask. The launch system in there isn’t fully implemented yet, but the 
>>> fundamental idea is valid and supports some range of capability.
>>> 
>>> We used to have a cmd line option in ORTE for what you propose - it 
>>> wouldn’t be too hard to restore. Is there some reason to do so?
>>> 
>>> 
 On Jan 21, 2015, at 7:53 AM, Mark Santcroos  
 wrote:
 
 Hi,
 
 Would it be possible to initially run "idle" orted's on my resources and 
 then use orterun to launch my applications to these already running 
 orted's.
 
 Thanks!
 
 Mark
 ___
 users mailing list
 us...@open-mpi.org
 Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
 Link to this post: 
 http://www.open-mpi.org/community/lists/users/2015/01/26220.php
>>> 
>> 
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/users/2015/01/26222.php
> 
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2015/01/26223.php

Re: [OMPI users] independent startup of orted and orterun

[Mark Santcroos]

Hi Ralph,

I figured that MR+ had to require something similar too (assuming it actually 
does use ORCM).

Let me study that wiki a bit and I will come back with a description of my 
use-case.

Thanks!

Mark


> On 21 Jan 2015, at 17:18 , Ralph Castain  wrote:
> 
> Sorry - should have included the link to the ORCM project:
> 
> https://github.com/open-mpi/orcm/wiki
> 
> 
>> On Jan 21, 2015, at 8:16 AM, Ralph Castain  wrote:
>> 
>> Theoretically, yes - see the ORCM project, which basically does what you 
>> ask. The launch system in there isn’t fully implemented yet, but the 
>> fundamental idea is valid and supports some range of capability.
>> 
>> We used to have a cmd line option in ORTE for what you propose - it wouldn’t 
>> be too hard to restore. Is there some reason to do so?
>> 
>> 
>>> On Jan 21, 2015, at 7:53 AM, Mark Santcroos  
>>> wrote:
>>> 
>>> Hi,
>>> 
>>> Would it be possible to initially run "idle" orted's on my resources and 
>>> then use orterun to launch my applications to these already running orted's.
>>> 
>>> Thanks!
>>> 
>>> Mark
>>> ___
>>> users mailing list
>>> us...@open-mpi.org
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>> Link to this post: 
>>> http://www.open-mpi.org/community/lists/users/2015/01/26220.php
>> 
> 
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2015/01/26222.php

Re: [OMPI users] independent startup of orted and orterun

[Ralph Castain]

Sorry - should have included the link to the ORCM project:

https://github.com/open-mpi/orcm/wiki 


> On Jan 21, 2015, at 8:16 AM, Ralph Castain  wrote:
> 
> Theoretically, yes - see the ORCM project, which basically does what you ask. 
> The launch system in there isn’t fully implemented yet, but the fundamental 
> idea is valid and supports some range of capability.
> 
> We used to have a cmd line option in ORTE for what you propose - it wouldn’t 
> be too hard to restore. Is there some reason to do so?
> 
> 
>> On Jan 21, 2015, at 7:53 AM, Mark Santcroos  
>> wrote:
>> 
>> Hi,
>> 
>> Would it be possible to initially run "idle" orted's on my resources and 
>> then use orterun to launch my applications to these already running orted's.
>> 
>> Thanks!
>> 
>> Mark
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/users/2015/01/26220.php
> 

Re: [OMPI users] independent startup of orted and orterun

[Ralph Castain]

Theoretically, yes - see the ORCM project, which basically does what you ask. 
The launch system in there isn’t fully implemented yet, but the fundamental 
idea is valid and supports some range of capability.

We used to have a cmd line option in ORTE for what you propose - it wouldn’t be 
too hard to restore. Is there some reason to do so?


> On Jan 21, 2015, at 7:53 AM, Mark Santcroos  
> wrote:
> 
> Hi,
> 
> Would it be possible to initially run "idle" orted's on my resources and then 
> use orterun to launch my applications to these already running orted's.
> 
> Thanks!
> 
> Mark
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2015/01/26220.php