Re: [galaxy-dev] SGE and Galaxy (a different approach)
Ah this is exactly what I was looking for. Thanks! On Wed, Apr 6, 2011 at 2:56 AM, Bram Slabbinck wrote: > Hi andrew > > What you need to do is to add the qsub parameter '-sync y'. This puts a > hold on the qsub command and makes it wait until the SGE job is finished. > > regards > Bram > > On 05/04/2011 18:27, andrew stewart wrote: > > I'm aware of how to configure Galaxy to use SGE in universe_wsgi.ini, > however what I want to do is a little different. Because I only want > certain processes to be submitted to the queue, I'd rather control this at > the tool configuration level (the xml wrapper). For example: > > > qsub myscript.sh > > > This will work, except that the status of the job (in Galaxy) shows as > completed even though the job has simply been submitted to SGE. Basically > Galaxy 'loses track' of the process because the submission process > (myscript.sh) has completed even if the actual job hasn't. > > Has anyone else tried anything like this before, or have anything helpful > to suggest? One thought is to somehow cause the myscript.sh process to > pause until the SGE job has completed... somehow. > > Any advice appreciated. > > Thanks, > Andrew > > > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/ > > > -- > = > Bram Slabbinck, PhD > > Bioinformatics & Systems Biology > VIB Department of Plant Systems Biology, Ghent University > Technologiepark 927, 9052 Gent, BELGIUM > > Tel:+32 (0)9 33 13 822 > Fax:+32 (0)9 33 13 809 > Email: bram.slabbi...@psb.ugent.be > WWW: http://bioinformatics.psb.ugent.be > == > === > Services and consulting in bioinformatics > http://www.arctix.be > = > > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] SGE and Galaxy (a different approach)
Hi andrew What you need to do is to add the qsub parameter '-sync y'. This puts a hold on the qsub command and makes it wait until the SGE job is finished. regards Bram On 05/04/2011 18:27, andrew stewart wrote: I'm aware of how to configure Galaxy to use SGE in universe_wsgi.ini, however what I want to do is a little different. Because I only want certain processes to be submitted to the queue, I'd rather control this at the tool configuration level (the xml wrapper). For example: qsub myscript.sh This will work, except that the status of the job (in Galaxy) shows as completed even though the job has simply been submitted to SGE. Basically Galaxy 'loses track' of the process because the submission process (myscript.sh) has completed even if the actual job hasn't. Has anyone else tried anything like this before, or have anything helpful to suggest? One thought is to somehow cause the myscript.sh process to pause until the SGE job has completed... somehow. Any advice appreciated. Thanks, Andrew ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- = Bram Slabbinck, PhD Bioinformatics& Systems Biology VIB Department of Plant Systems Biology, Ghent University Technologiepark 927, 9052 Gent, BELGIUM Tel:+32 (0)9 33 13 822 Fax:+32 (0)9 33 13 809 Email: bram.slabbi...@psb.ugent.be WWW: http://bioinformatics.psb.ugent.be = Services and consulting in bioinformatics http://www.arctix.be = ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] SGE and Galaxy (a different approach)
On Tue, Apr 5, 2011 at 12:27 PM, andrew stewart wrote: > I'm aware of how to configure Galaxy to use SGE in universe_wsgi.ini, > however what I want to do is a little different. Hi, Andrew. Take a look at this page: https://bitbucket.org/galaxy/galaxy-central/wiki/Config/Cluster In particular, does the last section, "Tool Configuration", describe something like what you want to do? Sean > Because I only want > certain processes to be submitted to the queue, I'd rather control this at > the tool configuration level (the xml wrapper). For example: > > qsub myscript.sh > > This will work, except that the status of the job (in Galaxy) shows as > completed even though the job has simply been submitted to SGE. Basically > Galaxy 'loses track' of the process because the submission process > (myscript.sh) has completed even if the actual job hasn't. > Has anyone else tried anything like this before, or have anything helpful to > suggest? One thought is to somehow cause the myscript.sh process to pause > until the SGE job has completed... somehow. > Any advice appreciated. > Thanks, > Andrew ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] SGE and Galaxy (a different approach)
andrew stewart wrote: > I'm aware of how to configure Galaxy to use SGE in universe_wsgi.ini, > however what I want to do is a little different. Because I only want > certain processes to be submitted to the queue, I'd rather control this at > the tool configuration level (the xml wrapper). For example: Hi Andrew, If you're not aware, it's possible to specify that only certain tools should be run on a specific cluster with certain parameters. See the bottom of the Cluster documentation for detials (under the "Galaxy Configuration" and "Tool Configuration" headers): https://bitbucket.org/galaxy/galaxy-central/wiki/Config/Cluster > > > qsub myscript.sh > > > This will work, except that the status of the job (in Galaxy) shows as > completed even though the job has simply been submitted to SGE. Basically > Galaxy 'loses track' of the process because the submission process > (myscript.sh) has completed even if the actual job hasn't. To avoid needing a wrapper that executes qsub for you, you may be able to just run your command behind qrsh or qlogin instead of qsub - this will remain in the foreground until the job completes. > Has anyone else tried anything like this before, or have anything helpful to > suggest? One thought is to somehow cause the myscript.sh process to pause > until the SGE job has completed... somehow. To use qsub, you'd need to make a wrapper around myscript.sh along the lines of the following: #!/bin/sh jobid=`qsub -terse myscript.sh` while [ 1 ]; do qstat -j $jobid >/dev/null 2>&1 [ $? -eq 1 ] && break; sleep 5; done -- The above script is the most basic option, it would probably be best to add more error handling and reporting. --nate > > Any advice appreciated. > > Thanks, > Andrew > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] SGE and Galaxy (a different approach)
Use qrsh instead of qsub to submit the job should help, I believe. On Tue, Apr 5, 2011 at 10:27 AM, andrew stewart wrote: > I'm aware of how to configure Galaxy to use SGE in universe_wsgi.ini, > however what I want to do is a little different. Because I only want > certain processes to be submitted to the queue, I'd rather control this at > the tool configuration level (the xml wrapper). For example: > > > qsub myscript.sh > > > This will work, except that the status of the job (in Galaxy) shows as > completed even though the job has simply been submitted to SGE. Basically > Galaxy 'loses track' of the process because the submission process > (myscript.sh) has completed even if the actual job hasn't. > > Has anyone else tried anything like this before, or have anything helpful > to suggest? One thought is to somehow cause the myscript.sh process to > pause until the SGE job has completed... somehow. > > Any advice appreciated. > > Thanks, > Andrew > > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/ > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] SGE and Galaxy (a different approach)
Ah, I should have added that this is for a strictly in-house Galaxy implementation with absolutely no intention of publishing these specific tools (just these ones! I promise I'll publish other tools in the future :) On Tue, Apr 5, 2011 at 12:38 PM, Peter Cock wrote: > On Tue, Apr 5, 2011 at 5:27 PM, andrew stewart > wrote: > > I'm aware of how to configure Galaxy to use SGE in universe_wsgi.ini, > > however what I want to do is a little different. Because I only want > > certain processes to be submitted to the queue, I'd rather control this > at > > the tool configuration level (the xml wrapper). > > Hi Andrew, > > That seems like a bad idea for portable XML wrappers - having local > configuration information hard coded within a tool wrapper would > prevent the tool (and any workflow using it) from being shared with > another Galaxy server. > > Peter > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] SGE and Galaxy (a different approach)
On Tue, Apr 5, 2011 at 5:27 PM, andrew stewart wrote: > I'm aware of how to configure Galaxy to use SGE in universe_wsgi.ini, > however what I want to do is a little different. Because I only want > certain processes to be submitted to the queue, I'd rather control this at > the tool configuration level (the xml wrapper). Hi Andrew, That seems like a bad idea for portable XML wrappers - having local configuration information hard coded within a tool wrapper would prevent the tool (and any workflow using it) from being shared with another Galaxy server. Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] SGE and Galaxy (a different approach)
I'm aware of how to configure Galaxy to use SGE in universe_wsgi.ini, however what I want to do is a little different. Because I only want certain processes to be submitted to the queue, I'd rather control this at the tool configuration level (the xml wrapper). For example: qsub myscript.sh This will work, except that the status of the job (in Galaxy) shows as completed even though the job has simply been submitted to SGE. Basically Galaxy 'loses track' of the process because the submission process (myscript.sh) has completed even if the actual job hasn't. Has anyone else tried anything like this before, or have anything helpful to suggest? One thought is to somehow cause the myscript.sh process to pause until the SGE job has completed... somehow. Any advice appreciated. Thanks, Andrew ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
