Chorny, Ilya wrote:

> I am pulling and testing as we speak. I did a preliminary test on my VM but 
> it does not have enough memory to run BWA. We pull into my dev version on the 
> cluster and do further testing. Also need to test how it integrates with the 
> splitting code. Should have some results by EOD.

Yeah, there is probably a problem with splitting, I was about to test
that myself.

> 
> Ilya
> 
> 
> -----Original Message-----
> From: Nate Coraor (n...@bx.psu.edu) [mailto:n...@bx.psu.edu] 
> Sent: Wednesday, November 02, 2011 9:15 AM
> To: Chorny, Ilya
> Cc: galaxy-dev@lists.bx.psu.edu
> Subject: Re: [galaxy-dev] Re Acutal user code
> 
> Hi Ilya,
> 
> I made a few more commits this morning.  Could you test it out in the present 
> state?  If everything works properly in your environment, I think it's pretty 
> close to being ready to integrate.  The major outstanding issue is to try to 
> avoid chmods in the upload action, which I have some ideas about.
> 
> --nate
> 
> Nate Coraor (n...@bx.psu.edu) wrote:
> > Nate Coraor (n...@bx.psu.edu) wrote:
> > > Chorny, Ilya wrote:
> > > > So it looks like for some reason galaxy is not picking up the 
> > > > information in the .e file. Not sure why. I even switched it back to 
> > > > writing the files to the pbs directory and it still did not pick up the 
> > > > .e files. BWA was core dumping but I was not given the error message in 
> > > > galaxy. 
> > > 
> > > Okay, this is pretty odd, I'll try to see what's going on here.  If 
> > > you figure anything out, please let me know.
> > 
> > Whoops, variable name collision in finish().  I've just committed a fix.
> > 
> > > 
> > > > BTW, I find it nice to be able to see the .o and .e files. The way you 
> > > > have it coded the .o and .e are written out to the 
> > > > job_working_directory and then disappear at the end of the run.  But 
> > > > the way I had it coded the pbs directory is world read writable. Is 
> > > > there an issue with having the pbs directory be world read write?
> > > 
> > > I imagine for some environments, it's preferable to avoid 
> > > world-writable directories.  Also, going forward, containing all of 
> > > a job's filesystem data to a single directory will be important for 
> > > running on multiple clusters without access to a single shared 
> > > filesystem mounted at the same path.
> > > 
> > > There's an issue in bitbucket somewhere regarding saving standard 
> > > out and standard error for all jobs, which would solve the problem 
> > > of it being in the working directory.
> > > 
> > > For debugging, if you comment out shutil.rmtree( 
> > > self.working_directory ) in jobs.cleanup(), it'll leave everything behind.
> > > 
> > > --nate
> > > 
> > > > 
> > > > Thanks,
> > > > 
> > > > Ilya
> > > > 
> > > > 
> > > > -----Original Message-----
> > > > From: galaxy-dev-boun...@lists.bx.psu.edu 
> > > > [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, 
> > > > Ilya
> > > > Sent: Monday, October 31, 2011 8:37 PM
> > > > To: Nate Coraor (n...@bx.psu.edu)
> > > > Cc: galaxy-dev@lists.bx.psu.edu
> > > > Subject: Re: [galaxy-dev] Re Acutal user code
> > > > 
> > > > It seems to be working but when I run BWA or Bowtie it finishes green 
> > > > but the files are empty. Doesn't give me much information. I also tried 
> > > > fastqc and it ran just fine.
> > > > 
> > > > -----Original Message-----
> > > > From: Nate Coraor (n...@bx.psu.edu) [mailto:n...@bx.psu.edu]
> > > > Sent: Monday, October 31, 2011 1:21 PM
> > > > To: Chorny, Ilya
> > > > Cc: galaxy-dev@lists.bx.psu.edu
> > > > Subject: Re: Re Acutal user code
> > > > 
> > > > Chorny, Ilya wrote:
> > > > > Sweet!! Can't wait to see it.
> > > > 
> > > > Have a look:
> > > > 
> > > >     https://bitbucket.org/natefoo/galaxy-central-ichorny/
> > > > 
> > > > Caveats:
> > > > 
> > > >     * job_working_directory has to be in a shared filesystem
> > > >     * outputs_to_working_directory must be True
> > > >     * set_metadata_externally must be True
> > > > 
> > > > One thing that would be good from a security standpoint would be if the 
> > > > external chown script took a Galaxy Job ID for its argument instead of 
> > > > a path, and then looked up the working directory path from the ID.  
> > > > That way the script couldn't be used maliciously to chown any file.
> > > > 
> > > > --nate
> > > > 
> > > > > 
> > > > > Thanks,
> > > > > 
> > > > > Ilya
> > > > > 
> > > > > -----Original Message-----
> > > > > From: Nate Coraor (n...@bx.psu.edu) [mailto:n...@bx.psu.edu]
> > > > > Sent: Thursday, October 27, 2011 1:24 PM
> > > > > To: Chorny, Ilya
> > > > > Cc: galaxy-dev@lists.bx.psu.edu
> > > > > Subject: Re: Re Acutal user code
> > > > > 
> > > > > Yup, cleaning up some things that were accidentally lost in merges.  
> > > > > I added another bitbucket repo so as not to mess up yours.
> > > > > 
> > > > > Chorny, Ilya wrote:
> > > > > > Hi Nate,
> > > > > > 
> > > > > > Any updates?
> > > > > > 
> > > > > > Thanks,
> > > > > > 
> > > > > > Ilya
> > > > > > 
> > > > > > 
> > > > > > -----Original Message-----
> > > > > > From: galaxy-dev-boun...@lists.bx.psu.edu
> > > > > > [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of 
> > > > > > Chorny, Ilya
> > > > > > Sent: Friday, October 21, 2011 2:24 PM
> > > > > > To: Nate Coraor (n...@bx.psu.edu)
> > > > > > Cc: galaxy-dev@lists.bx.psu.edu
> > > > > > Subject: Re: [galaxy-dev] Re Acutal user code
> > > > > > 
> > > > > > One more thing there are some changes in 
> > > > > > lib/galaxy/tools/__init__.py and lib/galaxy/model/__init__.py that 
> > > > > > are not related to actual user. They look like they are toolshed 
> > > > > > related and may have been changed because to version of the file 
> > > > > > that I copied over was older then galaxy-central. Probably should 
> > > > > > have done a merge first. I can redo it if you like. Let me know.
> > > > > > 
> > > > > > Ilya
> > > > > > 
> > > > > > 
> > > > > > -----Original Message-----
> > > > > > From: Chorny, Ilya
> > > > > > Sent: Friday, October 21, 2011 2:11 PM
> > > > > > To: Chorny, Ilya; Nate Coraor (n...@bx.psu.edu)
> > > > > > Cc: galaxy-dev@lists.bx.psu.edu
> > > > > > Subject: RE: Re Acutal user code
> > > > > > 
> > > > > > Sorry, the link was wrong.
> > > > > > 
> > > > > > https://bitbucket.org/ichorny/galaxy-central-ilya2/
> > > > > > 
> > > > > > -----Original Message-----
> > > > > > From: galaxy-dev-boun...@lists.bx.psu.edu
> > > > > > [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of 
> > > > > > Chorny, Ilya
> > > > > > Sent: Friday, October 21, 2011 2:07 PM
> > > > > > To: Nate Coraor (n...@bx.psu.edu)
> > > > > > Cc: galaxy-dev@lists.bx.psu.edu
> > > > > > Subject: Re: [galaxy-dev] Re Acutal user code
> > > > > > 
> > > > > > https://bitbucket.org/ichorny/galaxy-central-ilya2/pull-reques
> > > > > > t/new
> > > > > > 
> > > > > > everything but the tophat/cufflinks wrapper.
> > > > > > 
> > > > > > Thanks,
> > > > > > 
> > > > > > Ilya
> > > > > > 
> > > > > > 
> > > > > > -----Original Message-----
> > > > > > From: Nate Coraor (n...@bx.psu.edu) [mailto:n...@bx.psu.edu]
> > > > > > Sent: Friday, October 21, 2011 12:56 PM
> > > > > > To: Chorny, Ilya
> > > > > > Cc: galaxy-dev@lists.bx.psu.edu
> > > > > > Subject: Re: Re Acutal user code
> > > > > > 
> > > > > > Chorny, Ilya wrote:
> > > > > > > Nate,
> > > > > > > 
> > > > > > > I have been trying to figure out how to make the changes you 
> > > > > > > suggested (i.e. write output files to the job_working_directory) 
> > > > > > > and then move them over to files but I am having a difficult time 
> > > > > > > navigating the code.  If you could point me in the right 
> > > > > > > direction, that would be helpful.  I know the 
> > > > > > > lib/galaxy/models/__init__.py initializes the output files but I 
> > > > > > > don't think that is the right place to make the change. This 
> > > > > > > change probably needs to be done in the Job_Wrapper but most of 
> > > > > > > the code is very cryptic to me. Any guidance would be greatly 
> > > > > > > appreciated. These changes should not take very long to implement 
> > > > > > > once I get a handle on the code.
> > > > > > 
> > > > > > Hi Ilya,
> > > > > > 
> > > > > > I'm having a lot of difficulty separating out the actual user code 
> > > > > > from the other code in your fork.  Could you create a fork with 
> > > > > > just this code?
> > > > > > 
> > > > > > Thanks,
> > > > > > --nate
> > > > > > 
> > > > > > > 
> > > > > > > Thanks,
> > > > > > > 
> > > > > > > Ilya
> > > > > > > 
> > > > > > > Ilya Chorny Ph.D.
> > > > > > > Bioinformatics Scientist I
> > > > > > > Illumina, Inc.
> > > > > > > 9885 Towne Centre Drive
> > > > > > > San Diego, CA 92121
> > > > > > > Work: 858.202.4582
> > > > > > > Email: icho...@illumina.com<mailto:icho...@illumina.com>
> > > > > > > Website: www.illumina.com<http://www.illumina.com>
> > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > > ___________________________________________________________
> > > > > > Please keep all replies on the list by using "reply all"
> > > > > > in your mail client.  To manage your subscriptions to this and 
> > > > > > other Galaxy lists, please use the interface at:
> > > > > > 
> > > > > >   http://lists.bx.psu.edu/
> > > > > > 
> > > > > > ___________________________________________________________
> > > > > > Please keep all replies on the list by using "reply all"
> > > > > > in your mail client.  To manage your subscriptions to this and 
> > > > > > other Galaxy lists, please use the interface at:
> > > > > > 
> > > > > >   http://lists.bx.psu.edu/
> > > > > > 
> > > > > 
> > > > 
> > > > ___________________________________________________________
> > > > Please keep all replies on the list by using "reply all"
> > > > in your mail client.  To manage your subscriptions to this and other 
> > > > Galaxy lists, please use the interface at:
> > > > 
> > > >   http://lists.bx.psu.edu/
> > > > 
> > > ___________________________________________________________
> > > Please keep all replies on the list by using "reply all"
> > > in your mail client.  To manage your subscriptions to this and other 
> > > Galaxy lists, please use the interface at:
> > > 
> > >   http://lists.bx.psu.edu/
> > ___________________________________________________________
> > Please keep all replies on the list by using "reply all"
> > in your mail client.  To manage your subscriptions to this and other 
> > Galaxy lists, please use the interface at:
> > 
> >   http://lists.bx.psu.edu/
> 
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Reply via email to