Re: [galaxy-user] Galaxy for gene expression comparison
> > > I have patched galaxy to have cuffdiff handle replicates and to do > normalization, when that gets merged into the main branch your workflow will > be the same except you won't have to merge all of the bam files from each > condition together to use cuffdiff. > Hi all, I merged Rory's changes into galaxy-central, so Cuffdiff now supports replicates. I'll see what I can do for Cuffcompare; in the near-term, repeated merging using Cuffcompare will produce a GTF file that is both correct and usable with Cuffdiff. Thanks, J. ___ galaxy-user mailing list galaxy-user@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-user
Re: [galaxy-user] Galaxy for gene expression comparison
> I am not sure about cuffcompare, but cuffdiff doesn't generate any extra > files if you add more groups and replicates to the command line. It adds > columns to the output files but the number of files remains the same. Hmm. This is a new and welcome change. > For a workflow for Martin for now, I would suggest doing this for making > calls with no novel genes: > > 1) upload your reads > 2) fastq groom them into sanger format > 3) run tophat on each lane individually > 4) run cuffcompare with the gtf file you downloaded from uscs or wherever > against itself, > this puts it in a nice format to use with cuffdiff > 5) merge the bam files from tophat for the 10 lanes from each group into one > file > 6) run cuffdiff using the transcript gtf output file from cuffcompare and the > two merged bam files You can also run this workflow easily enough for de novo transcripts as well; only change is whether Cufflinks is fed a reference GTF. > I have patched galaxy to have cuffdiff handle replicates and to do > normalization, when that gets merged into the main branch your workflow will > be the same except you won't have to merge all of the bam files from each > condition together to use cuffdiff. Yes, looking into this soon. J. ___ galaxy-user mailing list galaxy-user@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-user
Re: [galaxy-user] Galaxy for gene expression comparison
I am not sure about cuffcompare, but cuffdiff doesn't generate any extra files if you add more groups and replicates to the command line. It adds columns to the output files but the number of files remains the same. For a workflow for Martin for now, I would suggest doing this for making calls with no novel genes: 1) upload your reads 2) fastq groom them into sanger format 3) run tophat on each lane individually 4) run cuffcompare with the gtf file you downloaded from uscs or wherever against itself, this puts it in a nice format to use with cuffdiff 5) merge the bam files from tophat for the 10 lanes from each group into one file 6) run cuffdiff using the transcript gtf output file from cuffcompare and the two merged bam files Merging is kind of crappy because you use in-replicate variation information, but its the best you can do now. I have patched galaxy to have cuffdiff handle replicates and to do normalization, when that gets merged into the main branch your workflow will be the same except you won't have to merge all of the bam files from each condition together to use cuffdiff. -rory On Jan 21, 2011, at 9:40 AM, Jeremy Goecks wrote: > Hi David, > > Cuffcompare and Cuffdiff generate many more outputs than most other tools; > specifically, both generate multiple output files for each additional input > given. While Galaxy can handle an arbitrary number of inputs easily, handling > so many outputs is challenging and requires extending the framework to handle > so many output files. ___ galaxy-user mailing list galaxy-user@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-user
Re: [galaxy-user] Galaxy for gene expression comparison
> Thank you for the suggestions. Ultimately, I would like to compare > gene(isoform) expression between two groups of 10 animals with one lane per > animal. I am using the public server to practice with some small data sets > right now, but will be getting the real data very soon and plan on using an > Amazon Cloud account to actually do the analysis. I can see now that this > approach is going to be met with some difficulty with the current state of > the data volume restrictions and limited functionality of Galaxy for > Cuffcompare/diff. Can you comment any further on the timeline of the > availability of the full functionality of these programs? > Hi David, Cuffcompare and Cuffdiff generate many more outputs than most other tools; specifically, both generate multiple output files for each additional input given. While Galaxy can handle an arbitrary number of inputs easily, handling so many outputs is challenging and requires extending the framework to handle so many output files. I did a bit of the necessary work this week, but more work is required and the path forward is a bit murky. I'm still hoping to have it available in a couple weeks, but no guarantees. Also, this is a good time to mention that we welcome code patches/contributions; if you can make something work in Galaxy, we'll review the code and, if it looks good, integrate it into our code base. > You seemed to suggest they will be available on the public server before they > are available on the Cloud? > I did not mean to imply this, only that the Cloud folks have their own process and schedule for rolling out changes, and I do not know their schedule. > Also, for the time being, would you mind clarifying for me what you mean by > repeatedly merging Cufflinks outputs? I imagine using Tophat to map the > reads and find splice junctions and assembling transcripts using Cufflinks > for each of the 20 animals. Are you talking about running the Cufflinks GTF > output through Cuffcompare, which allows two GTF files in Galaxy, and merging > that output(the union file) with the third Cufflinks file and so on for all > ten animals? Then do the same thing for the other group of ten animals, and > then comparing the two for a rough idea of the differences? > Cuffdiff requires a GTF reference file; this file contains the transcripts that will be used for comparing samples/replicates. If you're looking only at existing transcripts, using one from UCSC works fine and no merging is necessary. However, if you're looking for novel transcripts, you'll want to use the combined transcripts file that Cuffcompare produces. In this case, you'll want to iteratively merge the Cufflinks' outputs for all 20 animals so that you have a complete list of transcripts for Cuffdiff. > I guess I'm wondering how far I will be able to get with the analysis as > things stand on the Cloud or the public server > You should be able to get some preliminary results. As it stands now, you can run pairwise comparisons through the Tophat-->Cufflinks-->Cuffcompare-->Cuffdiff pipeline. You might try looking at two different pairwise comparisons and seeing how many similarities/differences there are. > I also need to come up with a strategy to work around the 1000Gb space limit, > as with 20 samples of 25 million reads and repeatedly generating files I > think it will get used up quickly > This is a question that Enis can comment on. Best, J.___ galaxy-user mailing list galaxy-user@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-user
Re: [galaxy-user] Galaxy for gene expression comparison
Jeremy, Thank you for the suggestions. Ultimately, I would like to compare gene(isoform) expression between two groups of 10 animals with one lane per animal. I am using the public server to practice with some small data sets right now, but will be getting the real data very soon and plan on using an Amazon Cloud account to actually do the analysis. I can see now that this approach is going to be met with some difficulty with the current state of the data volume restrictions and limited functionality of Galaxy for Cuffcompare/diff. Can you comment any further on the timeline of the availability of the full functionality of these programs? You seemed to suggest they will be available on the public server before they are available on the Cloud? Also, for the time being, would you mind clarifying for me what you mean by repeatedly merging Cufflinks outputs? I imagine using Tophat to map the reads and find splice junctions and assembling transcripts using Cufflinks for each of the 20 animals. Are you talking about running the Cufflinks GTF output through Cuffcompare, which allows two GTF files in Galaxy, and merging that output(the union file) with the third Cufflinks file and so on for all ten animals? Then do the same thing for the other group of ten animals, and then comparing the two for a rough idea of the differences? I guess I'm wondering how far I will be able to get with the analysis as things stand on the Cloud or the public server I also need to come up with a strategy to work around the 1000Gb space limit, as with 20 samples of 25 million reads and repeatedly generating files I think it will get used up quickly As far as changing the bowtie options through Tophat, I was just going to play around with the bowtie mapping settings to get an idea of which strategy is optimal and use those settings for Tophat, but this is probably unnecessary in the grand scheme of the analysis. I really appreciate your help - Thanks, David -Original Message- From: Jeremy Goecks on behalf of Jeremy Goecks Sent: Tue 1/18/2011 9:29 PM To: Martin, David A. Cc: eaf...@emory.edu; galaxy-u...@bx.psu.edu Subject: Re: [galaxy-user] Galaxy for gene expression comparison > I am comparing RNA expression in two groups of rats, a drug treated group > against a control group. There are 10 biological replicates in each group. I > am unsure of how to flow this analysis through Galaxy using Tophat followed > by Cufflinks/compare/diff. Should the files for each group be merged at any > point? I would think they should be kept separate in order to properly > account for the spread across animals. I am just a little unsure of how to > group the files on galaxy, and where to differentiate biological and > technical replicates. > Hi David, Yes, you're right -- merging along the way will prevent you from quantifying within-group variation; consequently, quantifying across-group variation will be very challenging as well. Here's the right thing to do: (1) map each replicate using Tophat and assemble transcripts using Cufflinks; (2) for all Cufflinks' outputs (assembled transcripts), build a set of comprehensive transcripts using Cuffcompare; (3) for Cuffdiff, group the replicates from each group and let Cuffdiff determine and quantitate within group and across group variation. However, Galaxy's tools currently don't support replicates, so you can't yet perform this analysis. We're working to enhance them, however, and we should have this functionality available on our main server in the next couple weeks. Enis can comment about when this functionality will be available on the cloud. (To be clear, you can perform step 1 using Galaxy now. You can also perform step 2, but you'll have to do so by repeatedly merging Cufflinks' outputs. You cannot perform step 3 right now with Galaxy.) > On a different note, is there a way to control the bowtie mapping parameters > more closely when using tophat? > There's limited control that you can exert over the bowtie commands within Tophat. Looking at the Tophat manual: http://tophat.cbcb.umd.edu/manual.html it looks like max-multihits (Maximum number of alignments) is the only Bowtie parameter you can directly control. There are, however, many Tophat parameters that enable you to control splice junction mapping directly; set Tophat's settings to 'Full parameter list' to see all the parameters you can control. What exactly are you looking to do? Thanks, J. ___ galaxy-user mailing list galaxy-user@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-user
Re: [galaxy-user] Galaxy for gene expression comparison
> I am comparing RNA expression in two groups of rats, a drug treated group > against a control group. There are 10 biological replicates in each group. I > am unsure of how to flow this analysis through Galaxy using Tophat followed > by Cufflinks/compare/diff. Should the files for each group be merged at any > point? I would think they should be kept separate in order to properly > account for the spread across animals. I am just a little unsure of how to > group the files on galaxy, and where to differentiate biological and > technical replicates. > Hi David, Yes, you're right -- merging along the way will prevent you from quantifying within-group variation; consequently, quantifying across-group variation will be very challenging as well. Here's the right thing to do: (1) map each replicate using Tophat and assemble transcripts using Cufflinks; (2) for all Cufflinks' outputs (assembled transcripts), build a set of comprehensive transcripts using Cuffcompare; (3) for Cuffdiff, group the replicates from each group and let Cuffdiff determine and quantitate within group and across group variation. However, Galaxy's tools currently don't support replicates, so you can't yet perform this analysis. We're working to enhance them, however, and we should have this functionality available on our main server in the next couple weeks. Enis can comment about when this functionality will be available on the cloud. (To be clear, you can perform step 1 using Galaxy now. You can also perform step 2, but you'll have to do so by repeatedly merging Cufflinks' outputs. You cannot perform step 3 right now with Galaxy.) > On a different note, is there a way to control the bowtie mapping parameters > more closely when using tophat? > There's limited control that you can exert over the bowtie commands within Tophat. Looking at the Tophat manual: http://tophat.cbcb.umd.edu/manual.html it looks like max-multihits (Maximum number of alignments) is the only Bowtie parameter you can directly control. There are, however, many Tophat parameters that enable you to control splice junction mapping directly; set Tophat's settings to 'Full parameter list' to see all the parameters you can control. What exactly are you looking to do? Thanks, J.___ galaxy-user mailing list galaxy-user@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-user
[galaxy-user] Galaxy for gene expression comparison
Hello, I am comparing RNA expression in two groups of rats, a drug treated group against a control group. There are 10 biological replicates in each group. I am unsure of how to flow this analysis through Galaxy using Tophat followed by Cufflinks/compare/diff. Should the files for each group be merged at any point? I would think they should be kept separate in order to properly account for the spread across animals. I am just a little unsure of how to group the files on galaxy, and where to differentiate biological and technical replicates. On a different note, is there a way to control the bowtie mapping parameters more closely when using tophat? Thank you for any kind of knowledge on these matters! -David Martin ___ galaxy-user mailing list galaxy-user@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-user
