Re: [galaxy-dev] Re Acutal user code
Chorny, Ilya wrote: So it looks like for some reason galaxy is not picking up the information in the .e file. Not sure why. I even switched it back to writing the files to the pbs directory and it still did not pick up the .e files. BWA was core dumping but I was not given the error message in galaxy. Okay, this is pretty odd, I'll try to see what's going on here. If you figure anything out, please let me know. BTW, I find it nice to be able to see the .o and .e files. The way you have it coded the .o and .e are written out to the job_working_directory and then disappear at the end of the run. But the way I had it coded the pbs directory is world read writable. Is there an issue with having the pbs directory be world read write? I imagine for some environments, it's preferable to avoid world-writable directories. Also, going forward, containing all of a job's filesystem data to a single directory will be important for running on multiple clusters without access to a single shared filesystem mounted at the same path. There's an issue in bitbucket somewhere regarding saving standard out and standard error for all jobs, which would solve the problem of it being in the working directory. For debugging, if you comment out shutil.rmtree( self.working_directory ) in jobs.cleanup(), it'll leave everything behind. --nate Thanks, Ilya -Original Message- From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Monday, October 31, 2011 8:37 PM To: Nate Coraor (n...@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Re Acutal user code It seems to be working but when I run BWA or Bowtie it finishes green but the files are empty. Doesn't give me much information. I also tried fastqc and it ran just fine. -Original Message- From: Nate Coraor (n...@bx.psu.edu) [mailto:n...@bx.psu.edu] Sent: Monday, October 31, 2011 1:21 PM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: Re Acutal user code Chorny, Ilya wrote: Sweet!! Can't wait to see it. Have a look: https://bitbucket.org/natefoo/galaxy-central-ichorny/ Caveats: * job_working_directory has to be in a shared filesystem * outputs_to_working_directory must be True * set_metadata_externally must be True One thing that would be good from a security standpoint would be if the external chown script took a Galaxy Job ID for its argument instead of a path, and then looked up the working directory path from the ID. That way the script couldn't be used maliciously to chown any file. --nate Thanks, Ilya -Original Message- From: Nate Coraor (n...@bx.psu.edu) [mailto:n...@bx.psu.edu] Sent: Thursday, October 27, 2011 1:24 PM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: Re Acutal user code Yup, cleaning up some things that were accidentally lost in merges. I added another bitbucket repo so as not to mess up yours. Chorny, Ilya wrote: Hi Nate, Any updates? Thanks, Ilya -Original Message- From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Friday, October 21, 2011 2:24 PM To: Nate Coraor (n...@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Re Acutal user code One more thing there are some changes in lib/galaxy/tools/__init__.py and lib/galaxy/model/__init__.py that are not related to actual user. They look like they are toolshed related and may have been changed because to version of the file that I copied over was older then galaxy-central. Probably should have done a merge first. I can redo it if you like. Let me know. Ilya -Original Message- From: Chorny, Ilya Sent: Friday, October 21, 2011 2:11 PM To: Chorny, Ilya; Nate Coraor (n...@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: RE: Re Acutal user code Sorry, the link was wrong. https://bitbucket.org/ichorny/galaxy-central-ilya2/ -Original Message- From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Friday, October 21, 2011 2:07 PM To: Nate Coraor (n...@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Re Acutal user code https://bitbucket.org/ichorny/galaxy-central-ilya2/pull-request/new everything but the tophat/cufflinks wrapper. Thanks, Ilya -Original Message- From: Nate Coraor (n...@bx.psu.edu) [mailto:n...@bx.psu.edu] Sent: Friday, October 21, 2011 12:56 PM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: Re Acutal user code Chorny, Ilya wrote: Nate, I have been trying to figure out how to make
Re: [galaxy-dev] Re Acutal user code
Chorny, Ilya wrote: I am pulling and testing as we speak. I did a preliminary test on my VM but it does not have enough memory to run BWA. We pull into my dev version on the cluster and do further testing. Also need to test how it integrates with the splitting code. Should have some results by EOD. Yeah, there is probably a problem with splitting, I was about to test that myself. Ilya -Original Message- From: Nate Coraor (n...@bx.psu.edu) [mailto:n...@bx.psu.edu] Sent: Wednesday, November 02, 2011 9:15 AM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Re Acutal user code Hi Ilya, I made a few more commits this morning. Could you test it out in the present state? If everything works properly in your environment, I think it's pretty close to being ready to integrate. The major outstanding issue is to try to avoid chmods in the upload action, which I have some ideas about. --nate Nate Coraor (n...@bx.psu.edu) wrote: Nate Coraor (n...@bx.psu.edu) wrote: Chorny, Ilya wrote: So it looks like for some reason galaxy is not picking up the information in the .e file. Not sure why. I even switched it back to writing the files to the pbs directory and it still did not pick up the .e files. BWA was core dumping but I was not given the error message in galaxy. Okay, this is pretty odd, I'll try to see what's going on here. If you figure anything out, please let me know. Whoops, variable name collision in finish(). I've just committed a fix. BTW, I find it nice to be able to see the .o and .e files. The way you have it coded the .o and .e are written out to the job_working_directory and then disappear at the end of the run. But the way I had it coded the pbs directory is world read writable. Is there an issue with having the pbs directory be world read write? I imagine for some environments, it's preferable to avoid world-writable directories. Also, going forward, containing all of a job's filesystem data to a single directory will be important for running on multiple clusters without access to a single shared filesystem mounted at the same path. There's an issue in bitbucket somewhere regarding saving standard out and standard error for all jobs, which would solve the problem of it being in the working directory. For debugging, if you comment out shutil.rmtree( self.working_directory ) in jobs.cleanup(), it'll leave everything behind. --nate Thanks, Ilya -Original Message- From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Monday, October 31, 2011 8:37 PM To: Nate Coraor (n...@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Re Acutal user code It seems to be working but when I run BWA or Bowtie it finishes green but the files are empty. Doesn't give me much information. I also tried fastqc and it ran just fine. -Original Message- From: Nate Coraor (n...@bx.psu.edu) [mailto:n...@bx.psu.edu] Sent: Monday, October 31, 2011 1:21 PM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: Re Acutal user code Chorny, Ilya wrote: Sweet!! Can't wait to see it. Have a look: https://bitbucket.org/natefoo/galaxy-central-ichorny/ Caveats: * job_working_directory has to be in a shared filesystem * outputs_to_working_directory must be True * set_metadata_externally must be True One thing that would be good from a security standpoint would be if the external chown script took a Galaxy Job ID for its argument instead of a path, and then looked up the working directory path from the ID. That way the script couldn't be used maliciously to chown any file. --nate Thanks, Ilya -Original Message- From: Nate Coraor (n...@bx.psu.edu) [mailto:n...@bx.psu.edu] Sent: Thursday, October 27, 2011 1:24 PM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: Re Acutal user code Yup, cleaning up some things that were accidentally lost in merges. I added another bitbucket repo so as not to mess up yours. Chorny, Ilya wrote: Hi Nate, Any updates? Thanks, Ilya -Original Message- From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Friday, October 21, 2011 2:24 PM To: Nate Coraor (n...@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Re Acutal user code One more thing there are some changes
Re: [galaxy-dev] Re Acutal user code
It seems to be working but when I run BWA or Bowtie it finishes green but the files are empty. Doesn't give me much information. I also tried fastqc and it ran just fine. -Original Message- From: Nate Coraor (n...@bx.psu.edu) [mailto:n...@bx.psu.edu] Sent: Monday, October 31, 2011 1:21 PM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: Re Acutal user code Chorny, Ilya wrote: Sweet!! Can't wait to see it. Have a look: https://bitbucket.org/natefoo/galaxy-central-ichorny/ Caveats: * job_working_directory has to be in a shared filesystem * outputs_to_working_directory must be True * set_metadata_externally must be True One thing that would be good from a security standpoint would be if the external chown script took a Galaxy Job ID for its argument instead of a path, and then looked up the working directory path from the ID. That way the script couldn't be used maliciously to chown any file. --nate Thanks, Ilya -Original Message- From: Nate Coraor (n...@bx.psu.edu) [mailto:n...@bx.psu.edu] Sent: Thursday, October 27, 2011 1:24 PM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: Re Acutal user code Yup, cleaning up some things that were accidentally lost in merges. I added another bitbucket repo so as not to mess up yours. Chorny, Ilya wrote: Hi Nate, Any updates? Thanks, Ilya -Original Message- From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Friday, October 21, 2011 2:24 PM To: Nate Coraor (n...@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Re Acutal user code One more thing there are some changes in lib/galaxy/tools/__init__.py and lib/galaxy/model/__init__.py that are not related to actual user. They look like they are toolshed related and may have been changed because to version of the file that I copied over was older then galaxy-central. Probably should have done a merge first. I can redo it if you like. Let me know. Ilya -Original Message- From: Chorny, Ilya Sent: Friday, October 21, 2011 2:11 PM To: Chorny, Ilya; Nate Coraor (n...@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: RE: Re Acutal user code Sorry, the link was wrong. https://bitbucket.org/ichorny/galaxy-central-ilya2/ -Original Message- From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Friday, October 21, 2011 2:07 PM To: Nate Coraor (n...@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Re Acutal user code https://bitbucket.org/ichorny/galaxy-central-ilya2/pull-request/new everything but the tophat/cufflinks wrapper. Thanks, Ilya -Original Message- From: Nate Coraor (n...@bx.psu.edu) [mailto:n...@bx.psu.edu] Sent: Friday, October 21, 2011 12:56 PM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: Re Acutal user code Chorny, Ilya wrote: Nate, I have been trying to figure out how to make the changes you suggested (i.e. write output files to the job_working_directory) and then move them over to files but I am having a difficult time navigating the code. If you could point me in the right direction, that would be helpful. I know the lib/galaxy/models/__init__.py initializes the output files but I don't think that is the right place to make the change. This change probably needs to be done in the Job_Wrapper but most of the code is very cryptic to me. Any guidance would be greatly appreciated. These changes should not take very long to implement once I get a handle on the code. Hi Ilya, I'm having a lot of difficulty separating out the actual user code from the other code in your fork. Could you create a fork with just this code? Thanks, --nate Thanks, Ilya Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: icho...@illumina.commailto:icho...@illumina.com Website: www.illumina.comhttp://www.illumina.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client
Re: [galaxy-dev] Re Acutal user code
Hi Nate, Any updates? Thanks, Ilya -Original Message- From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Friday, October 21, 2011 2:24 PM To: Nate Coraor (n...@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Re Acutal user code One more thing there are some changes in lib/galaxy/tools/__init__.py and lib/galaxy/model/__init__.py that are not related to actual user. They look like they are toolshed related and may have been changed because to version of the file that I copied over was older then galaxy-central. Probably should have done a merge first. I can redo it if you like. Let me know. Ilya -Original Message- From: Chorny, Ilya Sent: Friday, October 21, 2011 2:11 PM To: Chorny, Ilya; Nate Coraor (n...@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: RE: Re Acutal user code Sorry, the link was wrong. https://bitbucket.org/ichorny/galaxy-central-ilya2/ -Original Message- From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Friday, October 21, 2011 2:07 PM To: Nate Coraor (n...@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Re Acutal user code https://bitbucket.org/ichorny/galaxy-central-ilya2/pull-request/new everything but the tophat/cufflinks wrapper. Thanks, Ilya -Original Message- From: Nate Coraor (n...@bx.psu.edu) [mailto:n...@bx.psu.edu] Sent: Friday, October 21, 2011 12:56 PM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: Re Acutal user code Chorny, Ilya wrote: Nate, I have been trying to figure out how to make the changes you suggested (i.e. write output files to the job_working_directory) and then move them over to files but I am having a difficult time navigating the code. If you could point me in the right direction, that would be helpful. I know the lib/galaxy/models/__init__.py initializes the output files but I don't think that is the right place to make the change. This change probably needs to be done in the Job_Wrapper but most of the code is very cryptic to me. Any guidance would be greatly appreciated. These changes should not take very long to implement once I get a handle on the code. Hi Ilya, I'm having a lot of difficulty separating out the actual user code from the other code in your fork. Could you create a fork with just this code? Thanks, --nate Thanks, Ilya Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: icho...@illumina.commailto:icho...@illumina.com Website: www.illumina.comhttp://www.illumina.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Re Acutal user code
Chorny, Ilya wrote: Nate, I have been trying to figure out how to make the changes you suggested (i.e. write output files to the job_working_directory) and then move them over to files but I am having a difficult time navigating the code. If you could point me in the right direction, that would be helpful. I know the lib/galaxy/models/__init__.py initializes the output files but I don't think that is the right place to make the change. This change probably needs to be done in the Job_Wrapper but most of the code is very cryptic to me. Any guidance would be greatly appreciated. These changes should not take very long to implement once I get a handle on the code. Hi Ilya, I'm having a lot of difficulty separating out the actual user code from the other code in your fork. Could you create a fork with just this code? Thanks, --nate Thanks, Ilya Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: icho...@illumina.commailto:icho...@illumina.com Website: www.illumina.comhttp://www.illumina.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Re Acutal user code
Sorry, the link was wrong. https://bitbucket.org/ichorny/galaxy-central-ilya2/ -Original Message- From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Friday, October 21, 2011 2:07 PM To: Nate Coraor (n...@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Re Acutal user code https://bitbucket.org/ichorny/galaxy-central-ilya2/pull-request/new everything but the tophat/cufflinks wrapper. Thanks, Ilya -Original Message- From: Nate Coraor (n...@bx.psu.edu) [mailto:n...@bx.psu.edu] Sent: Friday, October 21, 2011 12:56 PM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: Re Acutal user code Chorny, Ilya wrote: Nate, I have been trying to figure out how to make the changes you suggested (i.e. write output files to the job_working_directory) and then move them over to files but I am having a difficult time navigating the code. If you could point me in the right direction, that would be helpful. I know the lib/galaxy/models/__init__.py initializes the output files but I don't think that is the right place to make the change. This change probably needs to be done in the Job_Wrapper but most of the code is very cryptic to me. Any guidance would be greatly appreciated. These changes should not take very long to implement once I get a handle on the code. Hi Ilya, I'm having a lot of difficulty separating out the actual user code from the other code in your fork. Could you create a fork with just this code? Thanks, --nate Thanks, Ilya Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: icho...@illumina.commailto:icho...@illumina.com Website: www.illumina.comhttp://www.illumina.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Re Acutal user code
One more thing there are some changes in lib/galaxy/tools/__init__.py and lib/galaxy/model/__init__.py that are not related to actual user. They look like they are toolshed related and may have been changed because to version of the file that I copied over was older then galaxy-central. Probably should have done a merge first. I can redo it if you like. Let me know. Ilya -Original Message- From: Chorny, Ilya Sent: Friday, October 21, 2011 2:11 PM To: Chorny, Ilya; Nate Coraor (n...@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: RE: Re Acutal user code Sorry, the link was wrong. https://bitbucket.org/ichorny/galaxy-central-ilya2/ -Original Message- From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Friday, October 21, 2011 2:07 PM To: Nate Coraor (n...@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Re Acutal user code https://bitbucket.org/ichorny/galaxy-central-ilya2/pull-request/new everything but the tophat/cufflinks wrapper. Thanks, Ilya -Original Message- From: Nate Coraor (n...@bx.psu.edu) [mailto:n...@bx.psu.edu] Sent: Friday, October 21, 2011 12:56 PM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: Re Acutal user code Chorny, Ilya wrote: Nate, I have been trying to figure out how to make the changes you suggested (i.e. write output files to the job_working_directory) and then move them over to files but I am having a difficult time navigating the code. If you could point me in the right direction, that would be helpful. I know the lib/galaxy/models/__init__.py initializes the output files but I don't think that is the right place to make the change. This change probably needs to be done in the Job_Wrapper but most of the code is very cryptic to me. Any guidance would be greatly appreciated. These changes should not take very long to implement once I get a handle on the code. Hi Ilya, I'm having a lot of difficulty separating out the actual user code from the other code in your fork. Could you create a fork with just this code? Thanks, --nate Thanks, Ilya Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: icho...@illumina.commailto:icho...@illumina.com Website: www.illumina.comhttp://www.illumina.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Re Acutal user code
Hey Ilya, I'm looking at it now, I'll let you know asap (should be tomorrow). --nate Chorny, Ilya wrote: Nate, I have been trying to figure out how to make the changes you suggested (i.e. write output files to the job_working_directory) and then move them over to files but I am having a difficult time navigating the code. If you could point me in the right direction, that would be helpful. I know the lib/galaxy/models/__init__.py initializes the output files but I don't think that is the right place to make the change. This change probably needs to be done in the Job_Wrapper but most of the code is very cryptic to me. Any guidance would be greatly appreciated. These changes should not take very long to implement once I get a handle on the code. Thanks, Ilya Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: icho...@illumina.commailto:icho...@illumina.com Website: www.illumina.comhttp://www.illumina.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
