Re: [galaxy-dev] Persistent jobs in cluster queue even after canceling job in galaxy
Erg... I am pretty ignorant about mercurial so I should probably not respond to this but I will try. It is pretty common practice for the Galaxy team to push bug fixes to the last release to the stable branch of galaxy-central - which is very different than the default branch of galaxy-central which contains active development. These don't go out to galaxy-dist automatically to prevent the need to strip truly egregious stuff out of the stable branch that the galaxy-dev news says to target. A quirk of this however is that the stable branch of galaxy-central is actually a good deal more stable the stable branch of galaxy-dist. It is what usegalaxy.org targets and at least a few other high profile Galaxy maintainers have caught on to this trick as well. I think you can update (or merge) the latest stable branch by doing something like the following: hg pull https://bitbucket.org/galaxy/galaxy-central#stable hg update stable We should probably do a better job of keeping the stable branch of galaxy-dist up-to-date - but right now we just push out updates at releases and for major security issues as far as I know. -John On Fri, Mar 14, 2014 at 2:23 PM, Brian Claywell bclay...@fhcrc.org wrote: On Fri, Mar 14, 2014 at 9:01 AM, John Chilton jmchil...@gmail.com wrote: I believe this problem was fixed by Nate after the latest dist release and pushed to the stable branch of galaxy-central. https://bitbucket.org/galaxy/galaxy-central/commits/1298d3f6aca59825d0eb3d32afd5686c4b1b9294 If you are eager for this bug fix, you can track the latest stable branch of galaxy-central instead of the galaxy-dist tag mentioned in the dev news. Right now it has some other good bug fixes not in the latest release. Ah, got it, thanks! Is it unfeasible to push bug fixes like those back to galaxy-dist/stable so those of us that would prefer stable to bleeding-edge don't have to cherry-pick commits? -- Brian Claywell, Systems Analyst/Programmer Fred Hutchinson Cancer Research Center bclay...@fhcrc.org ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Persistent jobs in cluster queue even after canceling job in galaxy
I believe this problem was fixed by Nate after the latest dist release and pushed to the stable branch of galaxy-central. https://bitbucket.org/galaxy/galaxy-central/commits/1298d3f6aca59825d0eb3d32afd5686c4b1b9294 If you are eager for this bug fix, you can track the latest stable branch of galaxy-central instead of the galaxy-dist tag mentioned in the dev news. Right now it has some other good bug fixes not in the latest release. -John On Tue, Mar 11, 2014 at 9:53 PM, Brian Claywell bclay...@fhcrc.org wrote: We've had the same problem since updating from the April 2013 stable to Feb 2014 stable. Our jobs are going off to a slurm cluster via the SlurmJobRunner plugin (though this was happening with the DRMAAJobRunner plugin too, if I remember right). Removing pending datasets occasionally removes the entry in Admin/Jobs, but not reliably, and regardless the jobs stay queued in slurm with no noise in galaxy.log at DEBUG or higher. We're not using parallelism either. I noticed this around the same time as I noticed the new in-progress animation in the history pane; perhaps there's an ajax-y callback that's not firing? Otherwise I'd expect to see something in the log about Galaxy at least trying to cancel the job. On Tue, Mar 11, 2014 at 10:57 AM, Ravi Alla ravi.a...@berkeley.edu wrote: Hi All, I've been able to submit jobs to the cluster through galaxy, it works great. But when the job is in queue to run (it is gray in the galaxy history pane) and I cancel the job, it still remains in queue on the cluster. Why does this happen? How can I delete the jobs in queue as well? I tried qdel job_id as galaxy user but it says I am not authenticated to delete the job. Any help would be greatly appreciated. Thanks Ravi. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ -- Brian Claywell, Systems Analyst/Programmer Fred Hutchinson Cancer Research Center bclay...@fhcrc.org ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Persistent jobs in cluster queue even after canceling job in galaxy
On Fri, Mar 14, 2014 at 9:01 AM, John Chilton jmchil...@gmail.com wrote: I believe this problem was fixed by Nate after the latest dist release and pushed to the stable branch of galaxy-central. https://bitbucket.org/galaxy/galaxy-central/commits/1298d3f6aca59825d0eb3d32afd5686c4b1b9294 If you are eager for this bug fix, you can track the latest stable branch of galaxy-central instead of the galaxy-dist tag mentioned in the dev news. Right now it has some other good bug fixes not in the latest release. Ah, got it, thanks! Is it unfeasible to push bug fixes like those back to galaxy-dist/stable so those of us that would prefer stable to bleeding-edge don't have to cherry-pick commits? -- Brian Claywell, Systems Analyst/Programmer Fred Hutchinson Cancer Research Center bclay...@fhcrc.org ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Persistent jobs in cluster queue even after canceling job in galaxy
On Tue, Mar 11, 2014 at 5:57 PM, Ravi Alla ravi.a...@berkeley.edu wrote: Hi All, I've been able to submit jobs to the cluster through galaxy, it works great. But when the job is in queue to run (it is gray in the galaxy history pane) and I cancel the job, it still remains in queue on the cluster. Why does this happen? How can I delete the jobs in queue as well? I tried qdel job_id as galaxy user but it says I am not authenticated to delete the job. Any help would be greatly appreciated. Thanks Ravi. What kind of cluster is it? e.g. SGE? Are you using task splitting (parallelization)? Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Persistent jobs in cluster queue even after canceling job in galaxy
Hi Peter, No I am not using that option. It is currently set to false in my universe_wgsi.ini file. It says it is a new feature and not recommended, so I didn't mess with it. Thanks Ravi On Mar 11, 2014, at 11:27 AM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi Ravi, Could you reply to the list? And actually I meant do you have use_tasked_jobs = True in your universe_wsgi.ini file which is linked to the special XML tag parallelism in some Galaxy Tools for splitting jobs. Sorry, that was unclear of me. Peter On Tue, Mar 11, 2014 at 6:24 PM, Ravi Alla ravi.a...@berkeley.edu wrote: Peter, This is a Centos linux cluster using PBS Torque job manager. By parallelization are you referring to multiple job handlers? No I am not. I barely set this server up and it is in its most basic form so far. I will add options for multiple web servers and runners once I get things working soundly. Thanks Ravi On Mar 11, 2014, at 11:07 AM, Peter Cock p.j.a.c...@googlemail.com wrote: On Tue, Mar 11, 2014 at 5:57 PM, Ravi Alla ravi.a...@berkeley.edu wrote: Hi All, I've been able to submit jobs to the cluster through galaxy, it works great. But when the job is in queue to run (it is gray in the galaxy history pane) and I cancel the job, it still remains in queue on the cluster. Why does this happen? How can I delete the jobs in queue as well? I tried qdel job_id as galaxy user but it says I am not authenticated to delete the job. Any help would be greatly appreciated. Thanks Ravi. What kind of cluster is it? e.g. SGE? Are you using task splitting (parallelization)? Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Persistent jobs in cluster queue even after canceling job in galaxy
We've had the same problem since updating from the April 2013 stable to Feb 2014 stable. Our jobs are going off to a slurm cluster via the SlurmJobRunner plugin (though this was happening with the DRMAAJobRunner plugin too, if I remember right). Removing pending datasets occasionally removes the entry in Admin/Jobs, but not reliably, and regardless the jobs stay queued in slurm with no noise in galaxy.log at DEBUG or higher. We're not using parallelism either. I noticed this around the same time as I noticed the new in-progress animation in the history pane; perhaps there's an ajax-y callback that's not firing? Otherwise I'd expect to see something in the log about Galaxy at least trying to cancel the job. On Tue, Mar 11, 2014 at 10:57 AM, Ravi Alla ravi.a...@berkeley.edu wrote: Hi All, I've been able to submit jobs to the cluster through galaxy, it works great. But when the job is in queue to run (it is gray in the galaxy history pane) and I cancel the job, it still remains in queue on the cluster. Why does this happen? How can I delete the jobs in queue as well? I tried qdel job_id as galaxy user but it says I am not authenticated to delete the job. Any help would be greatly appreciated. Thanks Ravi. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ -- Brian Claywell, Systems Analyst/Programmer Fred Hutchinson Cancer Research Center bclay...@fhcrc.org ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
