It seems likely you are correct and a catch for this hasn't been
implemented in Galaxy or more likely this is a limitation of the DRMAA
library for SGE. Ideally the job state would be coming back as FAILED
or something like that:

https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/jobs/runners/drmaa.py#L224

If you can debug the drmaa runner and verify that the SGE drmaa
library is indeed sending back some signal that the job failed with a
memory timeout and what that status code is - then we can catch and
update the drmaa runner. If not, probably the right place to address
this is in the drmaa library (I doubt that is still maintained - but I
honestly have no clue).

We have added special logic to the SLURM runner which extends the
drmaa runner and gives more informative errors for various error
conditions. An SGE runner that does something similar might be
possible:

https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/jobs/runners/slurm.py

-John

On Sun, May 22, 2016 at 3:24 PM, Nick <m...@rzg.mpg.de> wrote:
>
>  Hi,
>
> it looks like bug. When job is terminated by exceeded wallclock Galaxy assumes
> that job finished normaly and has green status in history. What confuses user
> while there are no results or results are broken.
>
>
> Report from SGE:
> 05/22/2016 20:16:26|  main|o3|W|job 1643894.1 exceeded hard wallclock time -
> initiate terminate method
>
> 05/22/2016 20:16:28|worker|ooo|W|job 1643894.1 failed on host o3 qmaster
> enforced h_rt, h_cpu, or h_vmem limit because: job 1643894.1 died through
> signal KILL (9)
>
>
> Logs from Galaxy:
> galaxy.jobs.runners.drmaa DEBUG 2016-05-22 20:16:29,425 (502/1643894) state
> change: job finished normally
>
>
> Job was submited with following native parameters:
> galaxy.jobs.runners.drmaa DEBUG 2016-05-22 20:11:19,837 (502) native
> specification is: -cwd -l h_rt=300
>
> Tested on Galaxy 15.11, 16.01, 16.04
>
> --
> Nick
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Reply via email to