Hi all, I've been running into some sporadic errors on our Cluster while using the latest development Galaxy, and the error handling has made this quite difficult to diagnose.
For a user perspective, the jobs seem to run, get submitted to the cluster, and finish, and the data looks OK via the 'eye' view icon, but is red in the history with: 0 bytes An error occurred running this job: info unavailable Furthermore, the stdout and stderr via the 'info' icon are blank. For watching the log (and adding more diagnosis lines), what is happening is the job is being split and sent out to the cluster fine, and starts running. If one of the tasks fails (and this seems to be happening due to some sort of file system error on our cluster), Galaxy spots this, and kills the rest of the jobs. That's good. The problem is it fails to record any record of why the job died. This is my suggestion for now - it would be nice to go further and fill the info text show in the history peep as well?: $ hg diff diff -r 4de1d566e9f8 lib/galaxy/jobs/__init__.py --- a/lib/galaxy/jobs/__init__.py Fri Sep 21 11:02:50 2012 +0100 +++ b/lib/galaxy/jobs/__init__.py Fri Sep 21 11:59:27 2012 +0100 @@ -1061,6 +1061,14 @@ log.error( "stderr for job %d is greater than 32K, only first part will be logged to database" % task.id ) task.stderr = stderr[:32768] task.command_line = self.command_line + + if task.state == task.states.ERROR: + # If failed, will kill the other tasks in this job. Record this + # task's stdout/stderr as should be useful to explain the failure: + job = self.get_job() + job.stdout = ("(From one sub-task:)\n" +task.stdout)[:32768] + job.stderr = ("(From one sub-task:)\n" +task.stderr)[:32768] + self.sa_session.flush() log.debug( 'task %d ended' % self.task_id ) Regards, Peter ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/