Hi again,
The loop (as explained below), did the job :)
Nikolay
Thank you very much, Nate,
1.
I have put a fix : a loop running the JobStatus check 5 times every 60
secs and only then throwing an exception as the one below.
It happened that all connect failures happen at the same time - at
slurm log rotation time at 3 am. Hopefully it helps :)
2.
Our slurm conf keeps the info about each job for 5 min. But looking at
the code, it seems that in in the case you describe below, there will be
an "Invalid job exception" leading to "Job finished" state. Am I wrong?
Anyway, I'll let you know if the loop does the job.
Thanks again,
Nikolay
On 2013-11-04 15:57, Nate Coraor wrote:
Hi Nikolay,
With slurm, the following change that I backed out should fix the
problem:
https://bitbucket.org/galaxy/galaxy-central/diff/lib/galaxy/jobs/runners/drmaa.py?diff2=d46b64f12c52&at=default
Although I do believe that if Galaxy doesn't read the completion
state before slurm "forgets" about the job (MinJobAge in slurm.conf),
this change could result in the job becoming permanently stuck in the
running state.
I should have some enhancements to the DRMAA runner for slurm coming
soon that would prevent this.
--nate
On Oct 31, 2013, at 5:27 AM, Nikolai Vazov wrote:
Hi,
I discovered a weird issue in the job behaviour : Galaxy is running a
long job on a cluster (more than 24h), about 15 hours later it misses
the connection with SLURM on the cluster and throws the following
message :
[root@galaxy-prod01 galaxy-dist]# grep 3715200 paster.log
galaxy.jobs.runners.drmaa INFO 2013-10-30 10:51:54,149 (555) queued
as 3715200
galaxy.jobs.runners.drmaa DEBUG 2013-10-30 10:51:55,149 (555/3715200)
state change: job is queued and active
galaxy.jobs.runners.drmaa DEBUG 2013-10-30 10:52:13,516 (555/3715200)
state change: job is running
galaxy.jobs.runners.drmaa INFO 2013-10-31 03:29:33,090 (555/3715200)
job left DRM queue with following message: code 1: slurm_load_jobs
error: Unable to contact slurm controller (connect failure),job_id:
3715200
Is there a timeout in Galaxy for contacting slurm? Yet, the job is
still running properly on the cluster ...
Thanks for help, it's really urgent :)
Nikolay
--
Nikolay Vazov, PhD
Research Computing Centre - http://hpc.uio.no
USIT, University of Oslo
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
--
Nikolay Vazov, PhD
Research Computing Centre - http://hpc.uio.no
USIT, University of Oslo
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
