Hi Mark > On Jan 21, 2015, at 11:21 AM, Mark Santcroos <mark.santcr...@rutgers.edu> > wrote: > > Hi Ralph, all, > > To give some background, I'm part of the RADICAL-Pilot [1] development team. > RADICAL-Pilot is a Pilot System, an implementation of the Pilot (job) > concept, which is in its most minimal form takes care of the decoupling of > resource acquisition and workload management. > So instead of launching your real_science.exe through PBS, you submit a > Pilot, which will allow you to perform application level scheduling. > Most obvious use-case if you want to run many (relatively) small tasks, then > you really don;t want to go through the batch system every time. That is > besides the fact that these machines are very bad in managing many tasks > anyway.
Yeah, we sympathize. Of course, one obvious solution is to get an allocation and execute a shell script that runs the tasks within that allocation - yes? > > The recent discussion we had on Spawn() on Cray's also originates here. > I want to free myself from having to use aprun for every task, and therefore > I am interested to see if ompi and/or orte can be the vehicle for that. > >> On 21 Jan 2015, at 17:16 , Ralph Castain <r...@open-mpi.org> wrote: >> Theoretically, yes - see the ORCM project, which basically does what you ask. >> The launch system in there isn’t fully implemented yet, but the fundamental >> idea is valid and supports some range of capability. > > I looked a bit better at ORCM and it clearly overlaps with what I want to > achieve. Agreed. In ORCM, we allow a user to request a “session” that results in allocation of resources. Each session is given an “orchestrator” - the ORCM “shepherd” daemon - responsible for executing the individual tasks across the assigned allocation, and a collection of “lamb” daemons (one on each node of the allocation) that forms a distributed VM. The orchestrator can execute the tasks very quickly since it doesn’t have to go back to the scheduler, and we allow it to do so according to any provided precedence requirement. Again, for simplicity, a shell script is the default mechanism for submitting the individual tasks. > One thing I noticed is that parts of it runs as root, why is that? ORCM is a full resource manager, which means it has a scheduler (rudimentary today) and boot-time daemons that must run as root so they can fork/exec the session-level daemons (that run at the user level). The orchestrator and its daemons all run at the user-level. > >> We used to have a cmd line option in ORTE for what you propose - it wouldn’t >> be too hard to restore. Is there some reason to do so? > > Can you point me to something that I could look for in the repo history, then > I can see if it serves my purpose. It would be back in the svn repo, I fear - would take awhile to hunt it down. Basically, it just (a) started all the daemons to create a VM, and (b) told mpirun to stick around as a persistent daemon. All subsequent calls to mpirun would reference back to the persistent one, thus using it to launch the jobs against the standing VM instead of starting a new one every time. For ORCM, we just took that capability and expressed it as the “shepherd” plus “lamb” daemon architecture described above. If you don’t want to replace the base RM, then using ORTE to establish a persistent VM is probably the way to go. I can probably make it do that again fairly readily. We have a developer’s meeting next week, which usually means I have some free time (during evenings and topics I’m not involved with), so I can take a crack at this then if that would be timely enough. Ralph > > If you think there is enough to warrant looking in more detail at ORCM I'm > happy to do that too. > > Cheers, > > Mark > > 1. https://github.com/radical-cybertools/radical.pilot > _______________________________________________ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2015/01/26225.php
