Re: [galaxy-dev] job output not returned from cluster
Hi all, I am using galaxy on the cloud and I keep getting the following error: An error occurred running this job: Job output not returned from cluster Any clues? thanks Alfonso ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] job output not returned from cluster
Hi Alfonso, Is this any particular tool that's failing? What does the state of your cloud cluster look like, are there any failures in the log? (in the cloudman interface) And lastly, when writing a new issue to the mailing list, please create a new email instead of replying to an unrelated thread. This will help us assist you and keep track of your individual issue instead of associating it with someone else. -Dannon On Feb 11, 2013, at 12:40 PM, Alfonso Garrido-Lecca alfonso.garrido-le...@colorado.edu wrote: Hi all, I am using galaxy on the cloud and I keep getting the following error: An error occurred running this job: Job output not returned from cluster Any clues? thanks Alfonso ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Job output not returned from Cluster
Hi Nate, thanks a lot for your hints. I was finally able to fix the problem. Galaxy couldn?t find the .drmerr file because our SGE installation merged .drmerr into .drmout... now everything works fine! Cheers, Sascha Original Message processed by CONSOLIDATE Subject: Re: [galaxy-dev] Job output not returned from Cluster Sent: Freitag, 13. Juli 2012 19:35 From: Nate Coraor (n...@bx.psu.edu) On Jul 9, 2012, at 11:25 AM, Sascha Kastens wrote: Hi! Unfortunately I was not able to fix this problem yet. Is anybody out there who had a similar problem while using Galaxy with SGE or has the knowledge about the things I can look at? Hi Sascha, The error message you are getting is because the SGE job?s stdout and stderr files are not where Galaxy expects to find them. The working directory will be output to the debug log prior to the job?s execution and should resemble: // where: is the absolute path to the value of job_working_directory in universe_wsgi.ini is determined based on the job ID but is most likely ?000? if you are just setting up a new server is the job?s ID as shown in the debug log Upon job completion, this directory should contain files like .drmout and .drmerr. Is it possible that your SGE installation is overriding the stdout/stderr paths, or that job_working_directory is not a shared filesystem? Thanks in advance! Cheers, Sascha ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Job output not returned from Cluster
On Jul 9, 2012, at 11:25 AM, Sascha Kastens wrote: Hi! Unfortunately I was not able to fix this problem yet. Is anybody out there who had a similar problem while using Galaxy with SGE or has the knowledge about the things I can look at? Hi Sascha, The error message you are getting is because the SGE job's stdout and stderr files are not where Galaxy expects to find them. The working directory will be output to the debug log prior to the job's execution and should resemble: job_working_directory/job id hash .../job id where: job_working_directory is the absolute path to the value of job_working_directory in universe_wsgi.ini job id hash is determined based on the job ID but is most likely '000' if you are just setting up a new server job id is the job's ID as shown in the debug log Upon job completion, this directory should contain files like job id.drmout and job id.drmerr. Is it possible that your SGE installation is overriding the stdout/stderr paths, or that job_working_directory is not a shared filesystem? Thanks in advance! Cheers, Sascha ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Job output not returned from Cluster
Hi Hemant, thanks for your help. I have set the value to 100, but this did not fix the problem. I am now trying to get a NFS mounted with -noac. Cheers, Sascha Original Message processed by CONSOLIDATE Subject: RE: [galaxy-dev] Job output not returned from Cluster Sent: Montag, 9. Juli 2012 17:38 From: Kelkar, Hemant (hkel...@unc.edu) Hi Sascha, I am not sure if the error you are seeing is like one I had posted in this thread? http://dev.list.galaxyproject.org/Problem-related-to-a-job-that-quot-failed-quot-td4627406.html#a4632168 --Hemant From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Sascha Kastens Sent: Monday, July 09, 2012 11:26 AM To: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Job output not returned from Cluster Hi! Unfortunately I was not able to fix this problem yet. Is anybody out there who had a similar problem while using Galaxy with SGE or has the knowledge about the things I can look at? Thanks in advance! Cheers, Sascha ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Job output not returned from Cluster
Hi! Unfortunately I was not able to fix this problem yet. Is anybody out there who had a similar problem while using Galaxy with SGE or has the knowledge about the things I can look at? Thanks in advance! Cheers, Sascha ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Job output not returned from Cluster
Hi all, I have exactly the same problem as described here: http://gmod.827538.n3.nabble.com/Job-output-not-returned-from-cluster-td879580.html All the SGE stuff is administrated in our company and unfortunately they know nothing about Galaxy and so it?s my task to get it running. Everything works fine except the problem mentioned above. I?ve contacted Erick but haven?t received a response yet. If anybody can give me some hints where I can look at to solve the problem or at least what I can tell the admins, I would be really grateful. Thanks in advance. If you need further information let me know! Cheers, Sascha ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Job output not returned from cluster
On Nov 29, 2011, at 9:22 PM, Fields, Christopher J wrote: On Nov 29, 2011, at 3:13 AM, Peter Cock wrote: On Monday, November 28, 2011, Joseph Hargitai joseph.hargi...@einstein.yu.edu wrote: Ed, we had the classic goof on our cluster with this. 4 nodes could not see the /home/galaxy folder due to a missing entry in /etc/fstab. When the jobs hit those nodes (which explains the randomness) we got the error message. Bothersome was the lack of good logs to go on. The error message was too generic - however I discovered that Galaxy was depositing the error and our messages in the /pbs folder and you could briefly read them before they got deleted. There the message was the classic SGE input/output message - /home/galaxy file not found. Hence my follow up question - how can I have galaxy NOT to delete these SGE error and out files? best, joe Better yet, Galaxy should read the SGE o and e files and record their contents as it would for a directly executed tools stdout and stderr. Peter ...or at least have the option to do so, maybe a level of verbosity. I have been bitten by lack of stderr output myself, where having it might have saved some manual debugging. Unless I'm misunderstanding, this is what Galaxy already does. stdout/stderr up to 32K are read from .o and .e and stored in job.stdout/job.stderr. We do need to just store them as files and make them accessible for each tool run, this will hopefully happen sometime soonish. --nate chris ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Job output not returned from cluster
On Monday, November 28, 2011, Joseph Hargitai joseph.hargi...@einstein.yu.edu wrote: Ed, we had the classic goof on our cluster with this. 4 nodes could not see the /home/galaxy folder due to a missing entry in /etc/fstab. When the jobs hit those nodes (which explains the randomness) we got the error message. Bothersome was the lack of good logs to go on. The error message was too generic - however I discovered that Galaxy was depositing the error and our messages in the /pbs folder and you could briefly read them before they got deleted. There the message was the classic SGE input/output message - /home/galaxy file not found. Hence my follow up question - how can I have galaxy NOT to delete these SGE error and out files? best, joe Better yet, Galaxy should read the SGE o and e files and record their contents as it would for a directly executed tools stdout and stderr. Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Job output not returned from cluster
On Nov 29, 2011, at 3:13 AM, Peter Cock wrote: On Monday, November 28, 2011, Joseph Hargitai joseph.hargi...@einstein.yu.edu wrote: Ed, we had the classic goof on our cluster with this. 4 nodes could not see the /home/galaxy folder due to a missing entry in /etc/fstab. When the jobs hit those nodes (which explains the randomness) we got the error message. Bothersome was the lack of good logs to go on. The error message was too generic - however I discovered that Galaxy was depositing the error and our messages in the /pbs folder and you could briefly read them before they got deleted. There the message was the classic SGE input/output message - /home/galaxy file not found. Hence my follow up question - how can I have galaxy NOT to delete these SGE error and out files? best, joe Better yet, Galaxy should read the SGE o and e files and record their contents as it would for a directly executed tools stdout and stderr. Peter ...or at least have the option to do so, maybe a level of verbosity. I have been bitten by lack of stderr output myself, where having it might have saved some manual debugging. chris ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Job output not returned from cluster
Ed, we had the classic goof on our cluster with this. 4 nodes could not see the /home/galaxy folder due to a missing entry in /etc/fstab. When the jobs hit those nodes (which explains the randomness) we got the error message. Bothersome was the lack of good logs to go on. The error message was too generic - however I discovered that Galaxy was depositing the error and our messages in the /pbs folder and you could briefly read them before they got deleted. There the message was the classic SGE input/output message - /home/galaxy file not found. Hence my follow up question - how can I have galaxy NOT to delete these SGE error and out files? best, joe From: Edward Kirton [eskir...@lbl.gov] Sent: Monday, November 28, 2011 4:15 PM To: Nate Coraor Cc: Joseph Hargitai; galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Job output not returned from cluster hi, we've had this issue too -- in short, the cluster node(s) finish writing outfiles to disk, but the file system (inode metadata) isn't updated at the galaxy server yet when galaxy checks for the files. turning the metadata caching off (as recommended on the galaxy wiki) isn't an option for me (and the performance hit would be significant), so i added some loops around the file checking (5sec sleep and retry up to 6 times). there were a couple of places this probably should be done (not just .[eo]* log files but also the outfiles). i am testing these hacks now but due to the intermittent nature of these errors, it'll be a few days before i know if this is working as expected. once vetted, i will put these minor edits in a clone of galaxy-central so the changes can be picked up. ed On Mon, Oct 24, 2011 at 10:24 AM, Nate Coraor n...@bx.psu.edumailto:n...@bx.psu.edu wrote: Joseph Hargitai wrote: Nate, this error is intermittent. You resubmit the same job twice or three time and then it works. Once we are over the midterm exams - which use galaxy - we will try to switch the filesystem from autofs to hard mount. We suspect this to be the issue. Ah, I suspect this is attribute caching in NFS. Try mounting with the option 'noac' and see if it solves the problem. Could we suppress e and o SGE style to resolve this issue, or Galaxy wants the o? The filename is unimportant, but I doubt it's the cause. Do you have an idea about the url build for galaxy - ucsc page return when the url is :8080/galaxy and not just /galaxy? Not off the top of my head. I have this message marked, I'll take a look as soon as I have time. --nate thanks, joe From: Nate Coraor [n...@bx.psu.edumailto:n...@bx.psu.edu] Sent: Friday, October 21, 2011 10:26 AM To: Joseph Hargitai Cc: galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Job output not returned from cluster Joseph Hargitai wrote: Hi, i was browsing through the list and found many entries for this issue but not a definite answer. We are actually running into this error for simple file uploads from the internal filesystem. Hi Joe, This error occurs when the job's standard output and error files are not found where Galaxy expects them, namely: cluster_files_directory/job_id.o cluster_files_directory/job_id.e Please check your queueing system to make sure it can correctly deliver these back from the execution hosts to the specified filesystem. --nate thanks, joe ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Job output not returned from cluster
Joseph Hargitai wrote: Nate, this error is intermittent. You resubmit the same job twice or three time and then it works. Once we are over the midterm exams - which use galaxy - we will try to switch the filesystem from autofs to hard mount. We suspect this to be the issue. Ah, I suspect this is attribute caching in NFS. Try mounting with the option 'noac' and see if it solves the problem. Could we suppress e and o SGE style to resolve this issue, or Galaxy wants the o? The filename is unimportant, but I doubt it's the cause. Do you have an idea about the url build for galaxy - ucsc page return when the url is :8080/galaxy and not just /galaxy? Not off the top of my head. I have this message marked, I'll take a look as soon as I have time. --nate thanks, joe From: Nate Coraor [n...@bx.psu.edu] Sent: Friday, October 21, 2011 10:26 AM To: Joseph Hargitai Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Job output not returned from cluster Joseph Hargitai wrote: Hi, i was browsing through the list and found many entries for this issue but not a definite answer. We are actually running into this error for simple file uploads from the internal filesystem. Hi Joe, This error occurs when the job's standard output and error files are not found where Galaxy expects them, namely: cluster_files_directory/job_id.o cluster_files_directory/job_id.e Please check your queueing system to make sure it can correctly deliver these back from the execution hosts to the specified filesystem. --nate thanks, joe ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Job output not returned from cluster
Joseph Hargitai wrote: Hi, i was browsing through the list and found many entries for this issue but not a definite answer. We are actually running into this error for simple file uploads from the internal filesystem. Hi Joe, This error occurs when the job's standard output and error files are not found where Galaxy expects them, namely: cluster_files_directory/job_id.o cluster_files_directory/job_id.e Please check your queueing system to make sure it can correctly deliver these back from the execution hosts to the specified filesystem. --nate thanks, joe ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Job output not returned from cluster
Nate, this error is intermittent. You resubmit the same job twice or three time and then it works. Once we are over the midterm exams - which use galaxy - we will try to switch the filesystem from autofs to hard mount. We suspect this to be the issue. Could we suppress e and o SGE style to resolve this issue, or Galaxy wants the o? Do you have an idea about the url build for galaxy - ucsc page return when the url is :8080/galaxy and not just /galaxy? thanks, joe From: Nate Coraor [n...@bx.psu.edu] Sent: Friday, October 21, 2011 10:26 AM To: Joseph Hargitai Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Job output not returned from cluster Joseph Hargitai wrote: Hi, i was browsing through the list and found many entries for this issue but not a definite answer. We are actually running into this error for simple file uploads from the internal filesystem. Hi Joe, This error occurs when the job's standard output and error files are not found where Galaxy expects them, namely: cluster_files_directory/job_id.o cluster_files_directory/job_id.e Please check your queueing system to make sure it can correctly deliver these back from the execution hosts to the specified filesystem. --nate thanks, joe ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Job output not returned from cluster
Hi, i was browsing through the list and found many entries for this issue but not a definite answer. We are actually running into this error for simple file uploads from the internal filesystem. thanks, joe ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Job output not returned from cluster
On Fri, Jul 29, 2011 at 1:01 AM, Ka Ming Nip km...@bcgsc.ca wrote: My jobs have this problem when the command for the tool is wrapped by the stderr wrapper script. Ka Ming Which stderr wrapper script? I think there is more than one... I've also had this error message (I'm currently working out how to connect our Galaxy to our cluster), and in at least one case it was caused by a file permission problem - the tool appeared to run but could not write the output files. If Galaxy could give more diagnostics rather than just Job output not returned from cluster it would help. For instance, as we use SGE, perhaps the captured stdout/stderr files might be available. Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Job output not returned from cluster
It was the one on the wiki page. Ka Ming From: Peter Cock [p.j.a.c...@googlemail.com] Sent: July 29, 2011 2:42 AM To: Ka Ming Nip Cc: Edward Kirton; galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Job output not returned from cluster On Fri, Jul 29, 2011 at 1:01 AM, Ka Ming Nip km...@bcgsc.ca wrote: My jobs have this problem when the command for the tool is wrapped by the stderr wrapper script. Ka Ming Which stderr wrapper script? I think there is more than one... I've also had this error message (I'm currently working out how to connect our Galaxy to our cluster), and in at least one case it was caused by a file permission problem - the tool appeared to run but could not write the output files. If Galaxy could give more diagnostics rather than just Job output not returned from cluster it would help. For instance, as we use SGE, perhaps the captured stdout/stderr files might be available. Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Job output not returned from cluster
thanks for your comments, fellas. permissions would certainly cause this problem, but that's not the cause for me. most wrappers just serve to redirect stderr, so i don't think it's the wrapper script itself, but the stdout/stderr files are part of the problem. the error message is thrown in the finish_job method when it can't open the source/dest stdout/stderr for reading/writing. i split the try statement to add finer-grained error messages but i already verified the files do exist, so it's seems to be a file system issue. i suspect it's because the storage i'm using as a staging area has flashdrives between the RAM and spinnning disks, so upon close, the file buffers may get flushed out of RAM to the SSDs but not immediately be available from the SCSI drives. Or maybe the (inode) metadata table hasn't finished updating yet. if so, it's not the fact that the cluster is heavily utilized, but the filesystem is. this disk is expressly for staging cluster jobs. i'll see if adding a short sleep and retry once upon error solves this problem... but i won't know immediately as the problem is intermittent. that's the problem with fancy toys; they often come with fancy problems! On Fri, Jul 29, 2011 at 2:42 AM, Peter Cock p.j.a.c...@googlemail.comwrote: also had this error message (I'm currently working out how to connect our Galaxy to our cluster), and in at least one case it was caused by a file permission problem - the tool appeared to run but could not write the output files. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Job output not returned from cluster
I've been getting these errors sometimes lately, particularly when the cluster is heavily loaded. The jobs have completed successfully, as I can see the output if I click the pen icon, but the job is in a failed state. Have any other sites been experiencing this problem? Or can the galaxy developers help shed some light on the issue? FYI, I use the outputs_to_working_directory option in universe_wsgi.ini so that i can use a faster/more reliable filesystem to collect output from the cluster. I'm not using the recently discussed patch to run jobs as the unix user. I'll continue to experiment with different filesystems and software settings. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Job output not returned from cluster
My jobs have this problem when the command for the tool is wrapped by the stderr wrapper script. Ka Ming From: galaxy-dev-boun...@lists.bx.psu.edu [galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Edward Kirton [eskir...@lbl.gov] Sent: July 28, 2011 3:41 PM To: galaxy-dev@lists.bx.psu.edu Subject: [galaxy-dev] Job output not returned from cluster I've been getting these errors sometimes lately, particularly when the cluster is heavily loaded. The jobs have completed successfully, as I can see the output if I click the pen icon, but the job is in a failed state. Have any other sites been experiencing this problem? Or can the galaxy developers help shed some light on the issue? FYI, I use the outputs_to_working_directory option in universe_wsgi.ini so that i can use a faster/more reliable filesystem to collect output from the cluster. I'm not using the recently discussed patch to run jobs as the unix user. I'll continue to experiment with different filesystems and software settings. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
