Re: [galaxy-dev] install reference genome Bwa and Bowtie!
. For your test and the tool you mentioned, you will want a BAM file assigned to the database "hg19". We have plans to simplify/restructure this document to better reflect how tools are now distributed (in the primary distribution vs the tool shed) and I am certainly interested in feedback about how to make this as easy as possible for everyone, especially new users, to follow. You can expect to see something newly reorganized within the month or so from us. It will probably be a set of documents, not just one. And as this is an open wiki, our community is welcome to then add in additional info/wikis as it seems appropriate or helpful. You can expect the dependencies wikis to have some changes as well to address the same. Thanks for comments and hope this helps! Jen Galaxy team Can you please rephrase the above so I can follow it? Thank you... -Original Message- From: Jennifer Jackson [mailto:j...@bx.psu.edu] Sent: Friday, February 01, 2013 9:32 AM To: Hakeem Almabrazi Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] install reference genome Bwa and Bowtie! Hi Hakeem, The set up for samtools looks correct. If the problems that you had with BWA/Bowtie were due to an error in our wiki, please send in the suggested changes to me. We want this to be correct. For the SAM->BAM tool - in the UI, there will never be a list of genomes - that is not one of the form options. This tool either interprets the database assigned to the input dataset (option "Choose the source for the reference list: Locally cashed") or allows the user to select a genome custom reference genome (option same, but "History"). So, to give this a test, run a dataset through it that has hg19 assigned as the reference and examine the results. If you are curious about custom genomes, which are used as fasta file datasets from a user's history with tools (instead of installed reference genomes), more is here: http://wiki.galaxyproject.org/Support#Custom_reference_genome Take care, Jen Galaxy team On 2/1/13 6:40 AM, Hakeem Almabrazi wrote: Thank you Jennifer for your help. I already resolved the issues with Bwa and Bowtie. I was misled with some missing instructions but I managed to resolved them by searching online. Now I am stuck at the SAM Tools displaying a reference genome :)... I will give more details at what I am doing hopefully someone can help. My reference fasta (hg10.fasta) file is located at (/usr/local/galaxy_data/hg19/samtools/) I ran the following command to build the samtools index file samtools faidx hg19.fasta This generated the hg19.fasta.fai Here is the content of the above path after running the samtools indexing command: -rwxr-xr-x 1 galaxy galaxy hg19.fasta -rw-rw-r-- 1 galaxy galaxy hg19.fasta.fai For my ~tool-data/sam_fa_indices.loc file here is the info I am using to reference the index file: Indexhg19/usr/local/galaxy_data/hg19/samtools/hg19.fasta ### I restarted my server but I do not see anything when I go to GUI->NGS:SamTools->Sam-To-Bam->Choose the source for the reference list: Is there something I am missing or is there anything else I need to change at this point. Regards, -Original Message- From: Jennifer Jackson [mailto:j...@bx.psu.edu] Sent: Thursday, January 31, 2013 6:45 PM To: Hakeem Almabrazi Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] install reference genome Bwa and Bowtie! Hello Hakeem, It is difficult to troubleshoot without knowing all the exact details of your configuration, but it sounds as if you are on the right path and the last few bits should be easy to reconcile with our documentation. In particular, check that the builds.txt file contains the genomes that you are adding. Also, if you want to view or use our indexes (if we have them available for your genome), or just want to double check your indexes vs ours, the rsync service can be used. Both are described in this wiki: http://wiki.galaxyproject.org/Admin/Data%20Integration I am not sure that I understand step #5, but there is another confusing point here. You want the data and all tools under the primary Galaxy directory hierarchy or have the alternate locations specified in the universe config file. Also, it is important to not us symbolic paths in the .loc files - use the full hard path to the indexed data. Hopefully this helps! If you do need more feedback, please be sure to leave the mailing list on the cc to help us track any reply, Best, Jen Galaxy team On 1/30/13 4:19 PM, Hakeem Almabrazi wrote: Hi, I just installed Galaxy in local Centos Vm. I did my best to get everything working ( I think). I followed the instructions in this Wiki (http://wiki.galaxyproject.org/Admin/NGS%20Local%20Setup) to install reference genomes for the tools Bwa and Bowtie. Here is what I did in details: 1.I installed all the software and added them to the PATH and everything s
Re: [galaxy-dev] install reference genome Bwa and Bowtie!
Hi Jennifer, Thank you again for your help. I am totally newbie to Galaxy so my issues could be due to my experience with it but I am getting some progress slowly though. Now, I am having totally different problem right now. I am trying to load simple file using the Galaxy GUi but it keeps telling me it is working on it but nothing really going on. I sent an email to the list but has not get anything back so far... Where can I look for galaxy logs? I am looking at the paster.log but it does not tell me much? People online keeps talking about output/log but I am not sure where to go about that. To answer your questions about SAM Tools: Following the instructions at http://wiki.galaxyproject.org/Admin/NGS%20Local%20Setup I was confused at this part: Then start adding the entries for each build The file's path should include the first part of the index files' names (the base) So if they are named like reference_in.fasta.amb and are stored in /path/to/base/indices, the entry should be: reference_in /path/to/base/indices/reference_in.fasta a tab separating the two --- It talks about two columns rather than 4 columns. Also, I could not find bwa_index.loc.sample in the tool_data/ folder. Also, I had to add this entry or might be bowtie_indices.loc to too_data_table_conf.xml since it was not there by default and it was not mentioned anywhere in the instructions. " For the SAM->BAM tool - in the UI, there will never be a list of genomes - that is not one of the form options. This tool either interprets the database assigned to the input dataset (option "Choose the source for the reference list: Locally cashed") or allows the user to select a genome custom reference genome (option same, but "History"). " I got lost at the point where it explains how to test it as follow: You can test your file by restarting the server and then opening SAM-to-BAM, Merge BAM files, or Generate pileup. Note that all rely on a file in the history that has a build assigned to it that is in your list of SAM Tools index builds. So you will need to set it (click the pencil icon to be able to change it) if it does not have that assigned to it already. This means that your build should be among the general builds available. - I am not really sure if I followed this. I could not see the pencil icon to change it. Can you please rephrase the above so I can follow it? Thank you... -Original Message- From: Jennifer Jackson [mailto:j...@bx.psu.edu] Sent: Friday, February 01, 2013 9:32 AM To: Hakeem Almabrazi Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] install reference genome Bwa and Bowtie! Hi Hakeem, The set up for samtools looks correct. If the problems that you had with BWA/Bowtie were due to an error in our wiki, please send in the suggested changes to me. We want this to be correct. For the SAM->BAM tool - in the UI, there will never be a list of genomes - that is not one of the form options. This tool either interprets the database assigned to the input dataset (option "Choose the source for the reference list: Locally cashed") or allows the user to select a genome custom reference genome (option same, but "History"). So, to give this a test, run a dataset through it that has hg19 assigned as the reference and examine the results. If you are curious about custom genomes, which are used as fasta file datasets from a user's history with tools (instead of installed reference genomes), more is here: http://wiki.galaxyproject.org/Support#Custom_reference_genome Take care, Jen Galaxy team On 2/1/13 6:40 AM, Hakeem Almabrazi wrote: > Thank you Jennifer for your help. > > I already resolved the issues with Bwa and Bowtie. I was misled with some > missing instructions but I managed to resolved them by searching online. Now > I am stuck at the SAM Tools displaying a reference genome :)... I > will give more details at what I am doing hopefully someone can help. > > My reference fasta (hg10.fasta) file is located at > (/usr/local/galaxy_data/hg19/samtools/) > > I ran the following command to build the samtools index file >> samtools faidx hg19.fasta > This generated the hg19.fasta.fai > > Here is the content of the above path after running the samtools indexing > command: > > -rwxr-xr-x 1 galaxy galaxy hg19.fasta > -rw-rw-r-- 1 galaxy galaxy hg19.fasta.fai > > > For my ~tool-data/sam_fa_indices.loc file here is the info I am using to > reference the index file: > > Indexhg19/usr/local/galaxy_data/hg19/samtools/hg19.fasta > ### > > I restarted my server but I do not see anything when I go to > GUI->NGS:SamTools->Sam-To-Bam->Choose the source for the refere
Re: [galaxy-dev] install reference genome Bwa and Bowtie!
Hi Hakeem, The set up for samtools looks correct. If the problems that you had with BWA/Bowtie were due to an error in our wiki, please send in the suggested changes to me. We want this to be correct. For the SAM->BAM tool - in the UI, there will never be a list of genomes - that is not one of the form options. This tool either interprets the database assigned to the input dataset (option "Choose the source for the reference list: Locally cashed") or allows the user to select a genome custom reference genome (option same, but "History"). So, to give this a test, run a dataset through it that has hg19 assigned as the reference and examine the results. If you are curious about custom genomes, which are used as fasta file datasets from a user's history with tools (instead of installed reference genomes), more is here: http://wiki.galaxyproject.org/Support#Custom_reference_genome Take care, Jen Galaxy team On 2/1/13 6:40 AM, Hakeem Almabrazi wrote: Thank you Jennifer for your help. I already resolved the issues with Bwa and Bowtie. I was misled with some missing instructions but I managed to resolved them by searching online. Now I am stuck at the SAM Tools displaying a reference genome :)... I will give more details at what I am doing hopefully someone can help. My reference fasta (hg10.fasta) file is located at (/usr/local/galaxy_data/hg19/samtools/) I ran the following command to build the samtools index file samtools faidx hg19.fasta This generated the hg19.fasta.fai Here is the content of the above path after running the samtools indexing command: -rwxr-xr-x 1 galaxy galaxy hg19.fasta -rw-rw-r-- 1 galaxy galaxy hg19.fasta.fai For my ~tool-data/sam_fa_indices.loc file here is the info I am using to reference the index file: Indexhg19/usr/local/galaxy_data/hg19/samtools/hg19.fasta ### I restarted my server but I do not see anything when I go to GUI->NGS:SamTools->Sam-To-Bam->Choose the source for the reference list: Is there something I am missing or is there anything else I need to change at this point. Regards, -Original Message- From: Jennifer Jackson [mailto:j...@bx.psu.edu] Sent: Thursday, January 31, 2013 6:45 PM To: Hakeem Almabrazi Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] install reference genome Bwa and Bowtie! Hello Hakeem, It is difficult to troubleshoot without knowing all the exact details of your configuration, but it sounds as if you are on the right path and the last few bits should be easy to reconcile with our documentation. In particular, check that the builds.txt file contains the genomes that you are adding. Also, if you want to view or use our indexes (if we have them available for your genome), or just want to double check your indexes vs ours, the rsync service can be used. Both are described in this wiki: http://wiki.galaxyproject.org/Admin/Data%20Integration I am not sure that I understand step #5, but there is another confusing point here. You want the data and all tools under the primary Galaxy directory hierarchy or have the alternate locations specified in the universe config file. Also, it is important to not us symbolic paths in the .loc files - use the full hard path to the indexed data. Hopefully this helps! If you do need more feedback, please be sure to leave the mailing list on the cc to help us track any reply, Best, Jen Galaxy team On 1/30/13 4:19 PM, Hakeem Almabrazi wrote: Hi, I just installed Galaxy in local Centos Vm. I did my best to get everything working ( I think). I followed the instructions in this Wiki (http://wiki.galaxyproject.org/Admin/NGS%20Local%20Setup) to install reference genomes for the tools Bwa and Bowtie. Here is what I did in details: 1.I installed all the software and added them to the PATH and everything seems to be working. 2.Then I created the indices as explained. 3.I put all the indices in the same recommended directory structure and gave all access rights to the Galaxy user which runs the Galaxy server. 4.I created the loc files (bowtie_indices.loc and bowtie_indices_color.loc and bwa_index.loc) and populate them with the recommended contents) and place them in ~/tool_data/ folder. 5.I copied the xml and py files from this location and put them into ~/tools/sr_mapping/ https://bitbucket.org/galaxy/galaxy-dist/src/da9d740fce31/tools/sr_mapping This step was not mentioned in the instruction but I thought this might be the cause. 6.I restarted the server. Once I select NGS:Mapping tool and Bwa or Bowtie I do not see the reference genome in the list. Can someone tell me what I missed or how to go about debugging it? I appreciate any kind of help ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use
Re: [galaxy-dev] install reference genome Bwa and Bowtie!
Thank you Jennifer for your help. I already resolved the issues with Bwa and Bowtie. I was misled with some missing instructions but I managed to resolved them by searching online. Now I am stuck at the SAM Tools displaying a reference genome :)... I will give more details at what I am doing hopefully someone can help. My reference fasta (hg10.fasta) file is located at (/usr/local/galaxy_data/hg19/samtools/) I ran the following command to build the samtools index file >samtools faidx hg19.fasta This generated the hg19.fasta.fai Here is the content of the above path after running the samtools indexing command: -rwxr-xr-x 1 galaxy galaxy hg19.fasta -rw-rw-r-- 1 galaxy galaxy hg19.fasta.fai For my ~tool-data/sam_fa_indices.loc file here is the info I am using to reference the index file: Indexhg19/usr/local/galaxy_data/hg19/samtools/hg19.fasta ### I restarted my server but I do not see anything when I go to GUI->NGS:SamTools->Sam-To-Bam->Choose the source for the reference list: Is there something I am missing or is there anything else I need to change at this point. Regards, -Original Message- From: Jennifer Jackson [mailto:j...@bx.psu.edu] Sent: Thursday, January 31, 2013 6:45 PM To: Hakeem Almabrazi Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] install reference genome Bwa and Bowtie! Hello Hakeem, It is difficult to troubleshoot without knowing all the exact details of your configuration, but it sounds as if you are on the right path and the last few bits should be easy to reconcile with our documentation. In particular, check that the builds.txt file contains the genomes that you are adding. Also, if you want to view or use our indexes (if we have them available for your genome), or just want to double check your indexes vs ours, the rsync service can be used. Both are described in this wiki: http://wiki.galaxyproject.org/Admin/Data%20Integration I am not sure that I understand step #5, but there is another confusing point here. You want the data and all tools under the primary Galaxy directory hierarchy or have the alternate locations specified in the universe config file. Also, it is important to not us symbolic paths in the .loc files - use the full hard path to the indexed data. Hopefully this helps! If you do need more feedback, please be sure to leave the mailing list on the cc to help us track any reply, Best, Jen Galaxy team On 1/30/13 4:19 PM, Hakeem Almabrazi wrote: > Hi, > > I just installed Galaxy in local Centos Vm. I did my best to get > everything working ( I think). I followed the instructions in this Wiki > (http://wiki.galaxyproject.org/Admin/NGS%20Local%20Setup) to install > reference genomes for the tools Bwa and Bowtie. Here is what I did in > details: > > 1.I installed all the software and added them to the PATH and everything > seems to be working. > > 2.Then I created the indices as explained. > > 3.I put all the indices in the same recommended directory structure and > gave all access rights to the Galaxy user which runs the Galaxy server. > > 4.I created the loc files (bowtie_indices.loc and > bowtie_indices_color.loc and bwa_index.loc) and populate them with the > recommended contents) and place them in ~/tool_data/ folder. > > 5.I copied the xml and py files from this location and put them into > ~/tools/sr_mapping/ > https://bitbucket.org/galaxy/galaxy-dist/src/da9d740fce31/tools/sr_mapping > This step was not mentioned in the instruction but I thought this might > be the cause. > > 6.I restarted the server. > > Once I select NGS:Mapping tool and Bwa or Bowtie I do not see the > reference genome in the list. Can someone tell me what I missed or how > to go about debugging it? > > I appreciate any kind of help > > > > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > >http://lists.bx.psu.edu/ > -- Jennifer Hillman-Jackson Galaxy Support and Training http://galaxyproject.org ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] install reference genome Bwa and Bowtie!
Hello Hakeem, It is difficult to troubleshoot without knowing all the exact details of your configuration, but it sounds as if you are on the right path and the last few bits should be easy to reconcile with our documentation. In particular, check that the builds.txt file contains the genomes that you are adding. Also, if you want to view or use our indexes (if we have them available for your genome), or just want to double check your indexes vs ours, the rsync service can be used. Both are described in this wiki: http://wiki.galaxyproject.org/Admin/Data%20Integration I am not sure that I understand step #5, but there is another confusing point here. You want the data and all tools under the primary Galaxy directory hierarchy or have the alternate locations specified in the universe config file. Also, it is important to not us symbolic paths in the .loc files - use the full hard path to the indexed data. Hopefully this helps! If you do need more feedback, please be sure to leave the mailing list on the cc to help us track any reply, Best, Jen Galaxy team On 1/30/13 4:19 PM, Hakeem Almabrazi wrote: Hi, I just installed Galaxy in local Centos Vm. I did my best to get everything working ( I think). I followed the instructions in this Wiki (http://wiki.galaxyproject.org/Admin/NGS%20Local%20Setup) to install reference genomes for the tools Bwa and Bowtie. Here is what I did in details: 1.I installed all the software and added them to the PATH and everything seems to be working. 2.Then I created the indices as explained. 3.I put all the indices in the same recommended directory structure and gave all access rights to the Galaxy user which runs the Galaxy server. 4.I created the loc files (bowtie_indices.loc and bowtie_indices_color.loc and bwa_index.loc) and populate them with the recommended contents) and place them in ~/tool_data/ folder. 5.I copied the xml and py files from this location and put them into ~/tools/sr_mapping/…. https://bitbucket.org/galaxy/galaxy-dist/src/da9d740fce31/tools/sr_mapping This step was not mentioned in the instruction but I thought this might be the cause. 6.I restarted the server. Once I select NGS:Mapping tool and Bwa or Bowtie I do not see the reference genome in the list. Can someone tell me what I missed or how to go about debugging it? I appreciate any kind of help…. ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Jennifer Hillman-Jackson Galaxy Support and Training http://galaxyproject.org ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] install reference genome Bwa and Bowtie!
Hi, I just installed Galaxy in local Centos Vm. I did my best to get everything working ( I think). I followed the instructions in this Wiki (http://wiki.galaxyproject.org/Admin/NGS%20Local%20Setup) to install reference genomes for the tools Bwa and Bowtie. Here is what I did in details: 1. I installed all the software and added them to the PATH and everything seems to be working. 2. Then I created the indices as explained. 3. I put all the indices in the same recommended directory structure and gave all access rights to the Galaxy user which runs the Galaxy server. 4. I created the loc files (bowtie_indices.loc and bowtie_indices_color.loc and bwa_index.loc) and populate them with the recommended contents) and place them in ~/tool_data/ folder. 5. I copied the xml and py files from this location and put them into ~/tools/sr_mapping/ https://bitbucket.org/galaxy/galaxy-dist/src/da9d740fce31/tools/sr_mapping This step was not mentioned in the instruction but I thought this might be the cause. 6. I restarted the server. Once I select NGS:Mapping tool and Bwa or Bowtie I do not see the reference genome in the list. Can someone tell me what I missed or how to go about debugging it? I appreciate any kind of help ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
