Marc, i am not aware of any mpi implementation in which mpirun does the job allocation.
instead, mpirun gets job info from the batch manager (e.g. number of nodes) so the job can be launched seamlessly and be properly killed in case of a job abort (bkill or equivalent) Cheers, Gilles On 2014/12/01 17:47, marc.hoepp...@bils.se wrote: > > > HI, > > sorry for the late reply - I've been traveling with limited email > access. I think you can leave this issue be. I think I was hoping for a > way to just launch mpirun and have it create the allocation by itself. > It's not super important right now, more something I was wondering > about. > > Thanks again for looking into this! > > /Marc > > On 28.11.2014 17:10, Ralph Castain wrote: > >> Hey Marc - just wanted to check to see if you felt this would indeed solve >> the problem for you. I'd rather not invest the time if this isn't going to >> meet the need, and I honestly don't know of a better solution. >> >> On Nov 20, 2014, at 2:13 PM, Ralph Castain <r...@open-mpi.org> wrote: >> >> Here's what I can provide: >> >> * lsrun -n N bash This causes openlava to create an allocation and start you >> off in a bash shell (or pick your shell) >> >> * mpirun ..... Will read the allocation and use openlava to start the >> daemons, and then the application, on the allocated nodes >> >> You can execute as many mpirun's as you like, then release the allocation (I >> believe by simply exiting the shell) when done. >> >> Does that match your expectations? >> Ralph >> >> On Nov 20, 2014, at 2:03 PM, Marc Höppner <marc.hoepp...@bils.se> wrote: >> >> Hi, >> >> yes, lsrun exists under openlava. >> >> Using mpirun is fine, but openlava currently requires that to be launched >> through a bash script (openmpi-mpirun). Would be neater if one could do away >> with that. >> >> Agan, thanks for looking into this! >> >> /Marc >> >> Hold on - was discussing this with a (possibly former) OpenLava developer >> who made some suggestions that would make this work. It all hinges on one >> thing. >> >> Can you please check and see if you have "lsrun" on your system? If you do, >> then I can offer a tight integration in that we would use OpenLava to >> actually launch the OMPI daemons. Still not sure I could support you >> directly launching MPI apps without using mpirun, if that's what you are >> after. >> >> On Nov 18, 2014, at 7:58 AM, Marc Höppner <marc.hoepp...@bils.se> wrote: >> >> Hi Ralph, >> >> I really appreciate you guys looking into this! At least now I know that >> there isn't a better way to run mpi jobs. Probably worth looking into LSF >> again.. >> >> Cheers, >> >> Marc >> I took a brief gander at the OpenLava source code, and a couple of things >> jump out. First, OpenLava is a batch scheduler and only supports batch >> execution - there is no interactive command for "run this job". So you would >> have to "bsub" mpirun regardless. >> >> Once you submit the job, mpirun can certainly read the local allocation via >> the environment. However, we cannot use the OpenLava internal functions to >> launch the daemons or processes as the code is GPL2, and thus has a viral >> incompatible license. Ordinarily, we get around that by just executing the >> interactive job execution command, but OpenLava doesn't have one. >> >> So we'd have no other choice but to use ssh to launch the daemons on the >> remote nodes. This is exactly what the provided openmpi wrapper script that >> comes with OpenLava already does. >> >> Bottom line: I don't see a way to do any deeper integration minus the >> interactive execution command. If OpenLava had a way of getting an >> allocation and then interactively running jobs, we could support what you >> requested. This doesn't seem to be what they are intending, unless I'm >> missing something (the documentation is rather incomplete). >> >> Ralph >> >> On Tue, Nov 18, 2014 at 6:20 AM, Marc Höppner <marc.hoepp...@bils.se> wrote: >> >> Hi, >> >> sure, no problem. And about the C Api, I really don't know more than what I >> was told in the google group post I referred to (i.e. the API is essentially >> identical to LSF 4-6, which should be on the web). >> >> The output of env can be found here: >> https://dl.dropboxusercontent.com/u/1918141/env.txt [6] >> >> /M >> >> Marc P. Hoeppner, PhD >> Team Leader >> BILS Genome Annotation Platform >> Department for Medical Biochemistry and Microbiology >> Uppsala University, Sweden >> marc.hoepp...@bils.se >> >> On 18 Nov 2014, at 15:14, Ralph Castain <r...@open-mpi.org> wrote: >> >> If you could just run a single copy of "env" and send the output along, that >> would help a lot. I'm not interested in the usual path etc, but would like >> to see the envars that OpenLava is setting. >> >> Thanks >> Ralph >> >> On Tue, Nov 18, 2014 at 2:19 AM, Gilles Gouaillardet >> <gilles.gouaillar...@iferc.org> wrote: >> >> Marc, >> >> the reply you pointed is a bit confusing to me : >> >> "There is a native C API which can submit/start/stop/kill/re queue jobs" >> this is not what i am looking for :-( >> >> "you need to make an appropriate call to openlava to start a remote process" >> this is what i am interested in :-) >> could you be more specific (e.g. point me to the functions, since the >> OpenLava doc is pretty minimal ...) >> >> the goal here is to spawn the orted daemons as part of the parallel job, >> so these daemons are accounted within the parallel job. >> /* if we use an API that simply spawns orted, but the orted is not related >> whatsoever to the parallel job, >> then we can simply use ssh */ >> >> Cheers, >> >> Gilles >> >> On 2014/11/18 18:24, Marc Höppner wrote: >> >> Hi Gilles, >> >> thanks for the prompt reply. Yes, as far as I know there is a C API to >> interact with jobs etc. Some mentioning here: >> https://groups.google.com/forum/#!topic/openlava-users/w74cRUe9Y9E [1] >> <https://groups.google.com/forum/#!topic/openlava-users/w74cRUe9Y9E> [1] >> >> /Marc Marc P. Hoeppner, PhD Team Leader BILS Genome Annotation Platform >> Department for Medical Biochemistry and Microbiology Uppsala University, >> Sweden marc.hoepp...@bils.se >> >> On 18 Nov 2014, at 08:40, Gilles Gouaillardet >> <gilles.gouaillar...@iferc.org> wrote: >> >> Hi Marc, >> >> OpenLava is based on a pretty old version of LSF (4.x if i remember >> correctly) >> and i do not think LSF had support for parallel jobs tight integration >> at that time. >> >> my understanding is that basically, there is two kind of direct >> integration : >> - mpirun launch: mpirun spawns orted via the API provided by the batch >> manager >> - direct launch: the mpi tasks are launched directly from the >> script/command line and no mpirun/orted is involved >> at that time, it works with SLURM and possibly other PMI capable batch >> manager >> >> i think OpenLava simply gets a list of hosts from the environment, build >> a machinefile, pass it to mpirun that spawns orted with ssh, so this is >> really loose integration. >> >> OpenMPI is based on plugins, so as long as the queing system provides an >> API to start/stop/kill tasks, mpirun launch should not >> be a huge effort. >> >> Are you aware of such an API provided by OpenLava ? >> >> Cheers, >> >> Gilles >> >> On 2014/11/18 16:31, Marc Höppner wrote: >> >> Hi list, >> >> I have recently started to wonder how hard it would be to add support for >> queuing systems to the tight integration function of OpenMPI (unfortunately, >> I am not a developer myself). Specifically, we are working with OpenLava >> (www.openlava.org [2]), which is based on an early version of Lava/LSF and >> open source. It's proven quite useful in environments where some level of >> LSF compatibility is needed, but without actually paying for a (rather >> pricey) LSF license. >> >> Given that openLava shares quite a bit of DNA with LSF, I was wondering how >> hard it would be to add OL tight integration support to OpenMPI. Currently, >> OL enables OpenMPI jobs through a wrapper script, but that's obviously not >> ideal and doesn't work for some programs that have MPI support built-in (and >> thus expect to be able to just execute mpirun). >> >> Any thoughts on this would be greatly appreciated! >> >> Regards, >> >> Marc >> >> Marc P. Hoeppner, PhD >> Team Leader >> BILS Genome Annotation Platform >> Department for Medical Biochemistry and Microbiology >> Uppsala University, Sweden >> marc.hoepp...@bils.se >> >> _______________________________________________ >> devel mailing list >> de...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel [3] >> Link to this post: >> http://www.open-mpi.org/community/lists/devel/2014/11/16312.php [4] >> >> _______________________________________________ >> devel mailing list >> de...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel [3] >> Link to this post: >> http://www.open-mpi.org/community/lists/devel/2014/11/16313.php [5] > _______________________________________________ devel mailing list > de...@open-mpi.org Subscription: > http://www.open-mpi.org/mailman/listinfo.cgi/devel [3] > > Link to this post: > http://www.open-mpi.org/community/lists/devel/2014/11/16314.php [7] > > _______________________________________________ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel [3] > Link to this post: > http://www.open-mpi.org/community/lists/devel/2014/11/16315.php [11] > _______________________________________________ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel [3] > Link to this post: > http://www.open-mpi.org/community/lists/devel/2014/11/16316.php [12] > _______________________________________________ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel [3] > Link to this post: > http://www.open-mpi.org/community/lists/devel/2014/11/16317.php [13] > > _______________________________________________ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel [3] > Link to this post: > http://www.open-mpi.org/community/lists/devel/2014/11/16318.php [8] > > _______________________________________________ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel [3] > Link to this post: > http://www.open-mpi.org/community/lists/devel/2014/11/16319.php [14] > > _______________________________________________ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel [3] > Link to this post: > http://www.open-mpi.org/community/lists/devel/2014/11/16326.php [9] > > _______________________________________________ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel [3] > Link to this post: > http://www.open-mpi.org/community/lists/devel/2014/11/16327.php [15] > > _______________________________________________ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel [3] > Link to this post: > http://www.open-mpi.org/community/lists/devel/2014/11/16373.php [10] > > > > Links: > ------ > [1] https://groups.google.com/forum/#%21topic/openlava-users/w74cRUe9Y9E > [2] http://www.openlava.org/ > [3] http://www.open-mpi.org/mailman/listinfo.cgi/devel > [4] http://www.open-mpi.org/community/lists/devel/2014/11/16312.php > [5] http://www.open-mpi.org/community/lists/devel/2014/11/16313.php > [6] https://dl.dropboxusercontent.com/u/1918141/env.txt > [7] http://www.open-mpi.org/community/lists/devel/2014/11/16314.php > [8] http://www.open-mpi.org/community/lists/devel/2014/11/16318.php > [9] http://www.open-mpi.org/community/lists/devel/2014/11/16326.php > [10] http://www.open-mpi.org/community/lists/devel/2014/11/16373.php > [11] http://www.open-mpi.org/community/lists/devel/2014/11/16315.php > [12] http://www.open-mpi.org/community/lists/devel/2014/11/16316.php > [13] http://www.open-mpi.org/community/lists/devel/2014/11/16317.php > [14] http://www.open-mpi.org/community/lists/devel/2014/11/16319.php > [15] http://www.open-mpi.org/community/lists/devel/2014/11/16327.php > > > > _______________________________________________ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Searchable archives: > http://www.open-mpi.org/community/lists/devel/2014/12/index.php
