Re: [galaxy-dev] Tophat output
Hello Jianpeng, The output files from TopHat are described on the TopHat tool form: --- quote --- Outputs Tophat produces two output files: junctions -- A UCSC BED track of junctions reported by TopHat. Each junction consists of two connected BED blocks, where each block is as long as the maximal overhang of any read spanning the junction. The score is the number of alignments spanning the junction. accepted_hits -- A list of read alignments in BAM format. Two other possible outputs, depending on the options you choose, are insertions and deletions, both of which are in BED format. - BED format is described in the Galaxy wiki, which includes links to the UCSC BED format description (they authored the format). http://wiki.g2.bx.psu.edu/Learn/Datatypes#Bed Two important rules to remember about BED format: rule #1: coordinate data is already reported with respect to the (+) strand rule #2: start is defined as the smallest coordinate, end is defined as the largest coordinate, due to the rule #1. BED files have a 0-based, fully-closed, start position in data files, but in browsers the data will display as 1-based. This means you'll need to add 1 to any start coordinate in a .bed file to locate it in a display application. The two will not and should not match. The end coordinate is also 0-based, but half-open. This will make it appear to be 1-based for casual users, so it will match between data files and display applications. - Using the first data row as an example and this information, we can tell that: chr20 199821 204701 JUNC0001 17 - 199821 204701 255,0,0 2 87,93 0,4787 * column 5 is 'score', or 'number of alignments spanning the junction'. In this case, 17 alignments. * column 11 is the blockSizes, or 'read maximal overhang' of the junctions (max alignment length). The first is 87 bases, the second 93 bases. * column 12 is the blockStarts, or 'overhang start' of the junctions (alignment start). The first is 0, the second 4787 bases. I am fairly certain that the first is always 0 and the second could be interpreted as the 'intron' length, but someone please correct me if this is wrong! Some calculations can be done with these numbers with respect to the overall position of the junction already defined in columns 1,2,3 (chrom, start, end): chr20:199821-204701 (-) that define the location of the predicted splices, the flanking aligned regions, and the (presumed) 'intron'. This example is a bit tricky because the alignment is on the (-) strand, but for most uses it is enough to simply calculate backwards from the end coordinate to the start. (Consider the end the start, and the start the end). If this sounds confusing, that's because it is! When you visualize the data the concept will make more sense and it is definitely worth learning about. Brief explanation: The first start is 0, which literally means that it starts at the very beginning of the alignment (0-based), which would be at position chr20, base 204,701, on the (-) strand. This alignment would continue for 87 bases, then stop. Then the splice would be present. The second start is at position (204701 - 4787) = 199914 = chr20, base 199,914, on the (-) strand. This is where the second splice would be present. This alignment would continue for 93 bases. The places the end at (199914 - 93) = 199821 = chr20, base 199,821, on the (-) strand. Which is the same as the reported global junction start position, which we are considering our end, because this is a (-) stranded alignment. And, it all adds up. Trackster would be a good place to start for Visualization (use the top menu bar link). The dataset can also be saved as a regular .bed file and loaded as a custom track into the UCSC Genome Browser (If the direct link is not fully configured yet). Hopefully this helps, Jen Galaxy team On 4/17/12 7:20 AM, Xu, Jianpeng wrote: Hi, I have installed local galaxy. I used the Tophat to do the RNA-seq alignment and got a output file: splice junction in bed format. I can not understand it clearly. What does the number 17, 14 ... in the column 5 mean ? What does the 87,93 mean ? What does the 0, 4787 mean ? Can you explain a little bit to me ? Which tool can be used to view this file ? Thanks, track name=junctions description=TopHat junctions chr20 199821204701 JUNC000117-199821 204701 255,0,0 287,93 0,4787 chr20204631 205520 JUNC0002 14-204631 205520255,0,0 296,87 0,802 chr20 205428 205775 JUNC00039- 205428 205775 255,0,0 292,91 0,256 chr20 205699 205958JUNC0004 15 - 205699205958 255,0,0 2 87,920,167 chr20205929 207067 JUNC0005 31-205929 207067
Re: [galaxy-dev] [galaxy-user] Toolshed initial upload errors out
Picking up on this thread, I said:
As per the title, I'm trying to setup a toolshed for our local use. After
creating a repo, I try to upload the initial file only to get:
TypeError: array item must be char
Which seems to happen repeatedly, i.e. it's not a passing error.
And Greg said:
Regarding this issue, your initial problem was regarding uploading a file to
a repository that you successfully created, but your
pater log does not include anything about uploading. Can you clarify what
the problem is? I'll need as much context as you can
provide. For example, have you successfully created a new repository? If
so, is the problem still that you cannot upload a file
to it? If this is the case, can you send me the file you are attempting to
upload? What are the contents of your hgweb.config
file, which is located in you Galaxy install directory?
So, to recap:
* I'm testing out using a local toolshed
* I install (via hg) a brand new Galaxy instance
* I configure and fire up the toolshed
* Create a repository
* This works fine. I go to upload a file and I get:
URL: http://158.119.147.40:9009/upload/upload?repository_id=529fd61ab1c6cc36
Module paste.exceptions.errormiddleware:144 in __call__
app_iter = self.application(environ, sr_checker)
Module paste.debug.prints:106 in __call__
environ, self.app)
Module paste.wsgilib:543 in intercept_output
app_iter = application(environ, replacement_start_response)
Module paste.recursive:84 in __call__
return self.application(environ, start_response)
Module paste.httpexceptions:633 in __call__
return self.application(environ, start_response)
Module galaxy.web.framework.base:160 in __call__
body = method( trans, **kwargs )
Module galaxy.web.framework:86 in decorator
return func( self, trans, *args, **kwargs )
Module galaxy.webapps.community.controllers.upload:126 in upload
repo.commit( user=trans.user.username, text=commit_message )
Module mercurial.localrepo:1195 in commit
ret = self.commitctx(cctx, True)
Module mercurial.localrepo:1264 in commitctx
p2.manifestnode(), (new, drop))
Module mercurial.manifest:147 in add
arraytext = array.array('c', text)
TypeError: array item must be char
* This happens repeatedly on different files (tried with several different text
files, pdfs etc., but not with any archives/zips), or with different
repositories, and after restart.
* hgweb.config looks fine I think, showing two (non-uploadable-to) repos
[paths]
repos/agapow/gfdfd = database/community_files/000/repo_1
repos/agapow/testrepo = database/community_files/000/repo_2
* The paster log (community_webapp.log, see http://pastebin.com/ysgnHwcz),
shows the upload call failing with the array item must be char (lines 98, 124
and elsewhere)
* Other toolshed browsing etc functions see to work fine
* Running CentOS 4
Not sure where to look next. Any ideas?
Paul Agapow (paul-michael.aga...@hpa.org.uk)
Bioinformatics, Health Protection Agency
-
**
The information contained in the EMail and any attachments is
confidential and intended solely and for the attention and use of
the named addressee(s). It may not be disclosed to any other person
without the express authority of the HPA, or the intended
recipient, or both. If you are not the intended recipient, you must
not disclose, copy, distribute or retain this message or any part
of it. This footnote also confirms that this EMail has been swept
for computer viruses, but please re-sweep any attachments before
opening or saving. HTTP://www.HPA.org.uk
**
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
[galaxy-dev] Tool syntax: How to get history name?
Hi everyone, I as wondering if there was a way to get the history name of an input dataset within a tool (i.e. in the cheetah between the command tags), is there? Thanks, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Tool syntax: How to get history name?
Ok, after a lot of testing and searching in the libs, I found the solution: $inputdataset.dataset.history.name Le 18/04/2012 10:22, Louise-Amélie Schmitt a écrit : Hi everyone, I as wondering if there was a way to get the history name of an input dataset within a tool (i.e. in the cheetah between the command tags), is there? Thanks, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Merging BLAST database support into Galaxy?
Hi Edward, We're now running BLAST+ searches on our local Galaxy via our cluster, and some of the cluster nodes have relatively small amounts of RAM. This means I've become more aware of limitations in the NCBI BLAST+ tools' support for using a subject FASTA file (instead of making a local BLAST database), which turns out to be surprisingly RAM hungry. The logical step is to allow users to build a BLAST database as a new datatype in Galaxy - which is what you (Edward) did some time ago as a fork, later posted to the Galaxy Tool Shed. Edward - are you happy for me to merge your work into the main wrappers? I mentioned idea this a couple of months ago: http://lists.bx.psu.edu/pipermail/galaxy-dev/2012-February/008544.html Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Merging BLAST database support into Galaxy?
On Wed, Apr 18, 2012 at 10:53 AM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi Edward, We're now running BLAST+ searches on our local Galaxy via our cluster, and some of the cluster nodes have relatively small amounts of RAM. This means I've become more aware of limitations in the NCBI BLAST+ tools' support for using a subject FASTA file (instead of making a local BLAST database), which turns out to be surprisingly RAM hungry. The logical step is to allow users to build a BLAST database as a new datatype in Galaxy - which is what you (Edward) did some time ago as a fork, later posted to the Galaxy Tool Shed. Edward - are you happy for me to merge your work into the main wrappers? I mentioned idea this a couple of months ago: http://lists.bx.psu.edu/pipermail/galaxy-dev/2012-February/008544.html Note this will take some extra work - we need to support protein BLAST databases as well, not just nucleotide database. Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] [galaxy-user] Toolshed initial upload errors out
Hello Paul,
I apologize for somehow missing the first line of your paster log, which in
fact shows the upload step. Since you've tried uploading various files with no
success, the problem is likely to be caused by something specific to your
environment - possibly the version of the mercurial package you're using. What
version of Python are you running, and what version of the mercurial package do
you have installed with it? Also, what version of Galaxy do you have, and what
database / version are you using?
I recently interacted with a developer who was setting up a local tool shed
using Centos6.2 and was not able to create a repository - a slightly different
issue than you have, but his problem stemmed from the old version of the
mercurial package that was installed. Here's what he did:
- Mercurial has version 1.4. --- THIS IS THE PROBLEM (default in the repo of
Centos6.2) -- see below
So I decided to remove the mercurial version 1.4 and do a source install of the
latest version 2.1.2. After installing, as user Galaxy I confirm a correct
installation with:
[galaxy@galaxy galaxy-dist]$ hg debuginstall
Checking encoding (UTF-8)...
Checking installed modules
(/usr/local/lib64/python2.6/site-packages/mercurial)...
Checking templates
(/usr/local/lib64/python2.6/site-packages/mercurial/templates)...
Checking commit editor...
Checking username...
No problems detected
However, after this I still have to set the PYTHONPATH variable (otherwise,
python does not find mercurial):
export PYTHONPATH=/usr/local/lib64/python2.6/site-packages/:${PYTHONPATH}
And our toolshed runs now, without errors. :-D
===
Can you let me know the versions of the various components I've listed above,
and whether upgrading the version of the mercurial package you are using fixes
this?
Thanks,
Greg Von Kuster
On Apr 18, 2012, at 4:12 AM, Paul-Michael Agapow wrote:
Picking up on this thread, I said:
As per the title, I'm trying to setup a toolshed for our local use. After
creating a repo, I try to upload the initial file only to get:
TypeError: array item must be char
Which seems to happen repeatedly, i.e. it's not a passing error.
And Greg said:
Regarding this issue, your initial problem was regarding uploading a file to
a repository that you successfully created, but your
pater log does not include anything about uploading. Can you clarify what
the problem is? I'll need as much context as you can
provide. For example, have you successfully created a new repository? If
so, is the problem still that you cannot upload a file
to it? If this is the case, can you send me the file you are attempting to
upload? What are the contents of your hgweb.config
file, which is located in you Galaxy install directory?
So, to recap:
* I'm testing out using a local toolshed
* I install (via hg) a brand new Galaxy instance
* I configure and fire up the toolshed
* Create a repository
* This works fine. I go to upload a file and I get:
URL: http://158.119.147.40:9009/upload/upload?repository_id=529fd61ab1c6cc36
Module paste.exceptions.errormiddleware:144 in __call__
app_iter = self.application(environ, sr_checker)
Module paste.debug.prints:106 in __call__
environ, self.app)
Module paste.wsgilib:543 in intercept_output
app_iter = application(environ, replacement_start_response)
Module paste.recursive:84 in __call__
return self.application(environ, start_response)
Module paste.httpexceptions:633 in __call__
return self.application(environ, start_response)
Module galaxy.web.framework.base:160 in __call__
body = method( trans, **kwargs )
Module galaxy.web.framework:86 in decorator
return func( self, trans, *args, **kwargs )
Module galaxy.webapps.community.controllers.upload:126 in upload
repo.commit( user=trans.user.username, text=commit_message )
Module mercurial.localrepo:1195 in commit
ret = self.commitctx(cctx, True)
Module mercurial.localrepo:1264 in commitctx
p2.manifestnode(), (new, drop))
Module mercurial.manifest:147 in add
arraytext = array.array('c', text)
TypeError: array item must be char
* This happens repeatedly on different files (tried with several different
text files, pdfs etc., but not with any archives/zips), or with different
repositories, and after restart.
* hgweb.config looks fine I think, showing two (non-uploadable-to) repos
[paths]
repos/agapow/gfdfd = database/community_files/000/repo_1
repos/agapow/testrepo = database/community_files/000/repo_2
* The paster log (community_webapp.log, see http://pastebin.com/ysgnHwcz),
shows the upload call failing with the array item must be char (lines 98,
124 and elsewhere)
* Other toolshed browsing etc functions see to work fine
* Running CentOS 4
Not sure where to look next. Any ideas?
Paul Agapow (paul-michael.aga...@hpa.org.uk)
Bioinformatics, Health Protection Agency
-
[galaxy-dev] Conditional tool options based on input file format?
Dear all, I've looked over the wiki and as far as I can see, 'when' tags used inside 'conditional' tags only work on another input variable's values. I would like to be able to do is a conditional based on the file format of an input file. For instance, a tool might take XML or tabular files, and need to show additional parameters for one input type but not the other. Does this enhancement idea seem useful? Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Tool development: how to pass parameters stored in an input dataset
Hi, Is it possible to pass the parameters of an input dataset? For example: the user selects a dataset of the history. I want to pass to my wrapperscript the input parameters of that dataset (the information you see when clicking the small 'i' on a dataset in the history). Regards, Joachim -- Joachim Jacob, PhD Rijvisschestraat 120, 9052 Zwijnaarde Tel: +32 9 244.66.34 Bioinformatics Training and Services (BITS) http://www.bits.vib.be @bitsatvib ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Hidden output files
Hi Frank, This should be resolved as of changeset 7057:08fbfeaaf3e1. Thanks! -Dannon On Mar 27, 2012, at 4:37 AM, Frank Sørensen wrote: Hi Dannon, Unfortunately our server is (still) behind a firewall, so I can't share anything, but I hope you can reproduce the error from the following description. If I make a vanilla test workflow say: Input Dataset - Map with BWA for Illumina (single end) - SAM-to-BAM - flagstat and save it. Then all outputs will show if I run it, even if none of the snowflakes are checked which is not logical. It would be better if the output visibility was checked by default, and the user should actively de-select the outputs that should be hidden, but that's another matter. If I edit the workflow that I just created, and select the output from flagstat to be visible, then the two other (BWA and SAM-to-BAM) will be hidden when I run it, which they should be. So far so good... If I edit the workflow again and select the remaining outputs (BWA and SAM-to-BAM) to be visible and save the workflow, then only BWA will be visible after run, thats a problem, SAM-to-BAM should also be visible since that's what I selected. If I want the output from SAM-to-BAM to be visible, I have to edit the workflow again, and de-select visibility of the output from SAM-to-BAM, save the workflow, and re-edit the workflow and select visibility of the output from SAM-to-BAM and save the workflow. It seems that selection/de-selection of output visibility only works under some circumstances and not others. I can conclude the state of the output visibility on the existing or missing post-hide-action, which is apparent if I select run workflow, and press expand all on the Running workflow screen, so I don't actually have to run the workflow, which is a great help. By tweaking my workflows using this editing / re-editing method, it is actually possible to get the visibility of all outputs into the state that I desire, but it is quite cumbersome on larger workflows. Regards - Frank ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Exporting worflow to local instal
Hi Danny, For moving workflows from one instance to another you'll want to click on Download or Export in the workflow context menu and use the URL for Importing to Another Galaxy. It looks something like this: (note the for_direct_import, that's how you'll know you have the right link type). https://main.g2.bx.psu.edu/workflow/for_direct_import?id=5214085ac1d2a9af And, the duplicate headers issue should be resolved on our main server now, please let us know if you find this isn't the case. -Dannon On Mar 14, 2012, at 12:13 PM, Muehlschlegel, Jochen D.,M.D. wrote: Hi, I would like to download/export my workflows from the Galaxy main to a local install at my institution. When I use the URL method, I get the following error message: Data at 'http://main.g2.bx.psu.edu/u/muehlsch12/w/imported-tophat---cuffdiff-paired-end-fastq ' does not appear to be a Galaxy workflow Message: No JSON object could be decoded: line 1 column 0 (char 0) When I want to download a workflow to my local machine, I get a server error (Duplicate headers received from server). The local install does not recognize publicly published workflows, so that step won't work either. Thanks for your help, Danny The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] How to specify RUM output dir?
I’m good with everything except the output dir. I can’t figure out how to pass that directory information from Galaxy to the RUM_runner.pl in such a way that Galaxy will be able to subsequently capture the data. Does anybody have any advice, or can someone point me towards an existing wrapper that does something similar? You have two options: (a) you can set up the tool to report only a subset of outputs from the tool; or (b) you can use a composite datatype to store the complete directory: http://wiki.g2.bx.psu.edu/Admin/Datatypes/Composite%20Datatypes Best, J.___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Hidden output files
Thanks. I have set up Mercurial to fetch updates from galaxy-dist, and it seems that my version of galaxy is up to date. - Should I pull from galaxy-central to get this update (7057:08fbfeaaf3e1)? - If so, could you please tell me how I do that? - Do you know how I permanently change repository from galaxy-dist to galaxy central, without loosing my tools and modifications? Kind Regards - Frank Den 18-04-2012 14:09, Dannon Baker skrev: Hi Frank, This should be resolved as of changeset 7057:08fbfeaaf3e1. Thanks! -Dannon On Mar 27, 2012, at 4:37 AM, Frank Sørensen wrote: Hi Dannon, Unfortunately our server is (still) behind a firewall, so I can't share anything, but I hope you can reproduce the error from the following description. If I make a vanilla test workflow say: Input Dataset - Map with BWA for Illumina (single end) - SAM-to-BAM - flagstat and save it. Then all outputs will show if I run it, even if none of the snowflakes are checked which is not logical. It would be better if the output visibility was checked by default, and the user should actively de-select the outputs that should be hidden, but that's another matter. If I edit the workflow that I just created, and select the output from flagstat to be visible, then the two other (BWA and SAM-to-BAM) will be hidden when I run it, which they should be. So far so good... If I edit the workflow again and select the remaining outputs (BWA and SAM-to-BAM) to be visible and save the workflow, then only BWA will be visible after run, thats a problem, SAM-to-BAM should also be visible since that's what I selected. If I want the output from SAM-to-BAM to be visible, I have to edit the workflow again, and de-select visibility of the output from SAM-to-BAM, save the workflow, and re-edit the workflow and select visibility of the output from SAM-to-BAM and save the workflow. It seems that selection/de-selection of output visibility only works under some circumstances and not others. I can conclude the state of the output visibility on the existing or missing post-hide-action, which is apparent if I select run workflow, and press expand all on the Running workflow screen, so I don't actually have to run the workflow, which is a great help. By tweaking my workflows using this editing / re-editing method, it is actually possible to get the visibility of all outputs into the state that I desire, but it is quite cumbersome on larger workflows. Regards - Frank -- Frank Sørensen, B.Sc., Programmer Molecular Diagnostic Laboratory (MDL) Molekylær Medicinsk Afdeling (MOMA) Århus Universitetshospital Skejby, Brendstrupgårdsvej, 8200 Århus N Tlf. +45 7845 5363 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Hidden output files
On Apr 18, 2012, at 8:44 AM, Frank Sørensen wrote: - Should I pull from galaxy-central to get this update (7057:08fbfeaaf3e1)? - If so, could you please tell me how I do that? Sure, you can pull from galaxy-central by issuing a manual pull with source from your galaxy-dist directory: hg pull -u https://bitbucket.org/galaxy/galaxy-central - Do you know how I permanently change repository from galaxy-dist to galaxy central, without loosing my tools and modifications? If you want to permanently change to galaxy-central, you'll need to manually edit the file galaxydirectory/.hg/hgrc. It should look something like this: [paths] default = https://bitbucket.org/galaxy/galaxy-dist Just change galaxy-dist to galaxy-central, save it, and you're good to go. -Dannon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Displaying genomic sequences in Trackster
Dear all, We have our own galaxy instance and the idea is to have trackster enabled for users to be able to visualize NGS mapping. We were able to configure trackster in our instance and the visualization works fine. We have two questions regarding trackster: 1) We can't display genomic sequences in trackster. As per the tutorial, we set the location of the .2bit file in the twobit.loc file for the trackster to be able to display the genomic sequence but for some reason it doesn't display it. The name of the builds is the same in all places i.e) in ucsc/chrom/builds.txt and also in the .loc files. Any ideas on what else should be done? 2) While saving the visualization, there is always an error message saying could not save visualization and it doesn't seem to be a web browser issue. How do we then save the visualization? Thanks in advance, Naharajan ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] database engine pool size
On Mar 26, 2012, at 12:18 PM, Shantanu Pavgi wrote:
{{{
TimeoutError: QueuePool limit of size 40 overflow 50 reached, connection
timed out, timeout 30
}}}
These limits are not reached for regular (non-workflow) galaxy jobs. Any help
on optimum values for these settings or performance tuning for workflow runs
would be appreciated.
Hi Shantanu,
Two things could help here:
1. Enable database_engine_option_strategy = threadlocal in your
universe_wsgi.ini
2. Run additional Galaxy processes:
http://wiki.g2.bx.psu.edu/Admin/Config/Performance/Web%20Application%20Scaling
-Dannon
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] Exception: Error Setting BAM Metadata: /bin/sh: samtools: not found
Hi Ross,
Thank you for your reply.
It still did not work.
I think SGE knows the path of samtools. Here is a test case I added to Galaxy,
and it creates the index for fasta file. I pasted the samtools faidx wrapper
and xml files below:
ubuntu@master:/mnt/galaxyTools/galaxy-dist/tools/samtools$ more
samtools_faidx.pl
#!/usr/bin/perl
use strict;
use warnings;
if (@ARGV 1) {
die Usage: perl $0 input fasta file\n;
}
`samtools faidx $ARGV[0]`;
`cp ${ARGV[0]}.fai $ARGV[1]`;
exit;
ubuntu@master:/mnt/galaxyTools/galaxy-dist/tools/samtools$ more
samtools_faidx.xml
tool id=samtools_faidx name=samtools faidx version=1.0.3
requirements
requirement type=packagesamtools/requirement
/requirements
descriptioncreates fasta index/description
command interpreter=perl
samtools_faidx.pl $input1 $output1
/command
inputs
param name=input1 type=data format=fasta label=fasta file to be
indexed /
/inputs
outputs
data format=fai name=output1 label=${tool.name} on ${on_string}:
fasta /
/outputs
/tool
It works, which shows that SGE knows the path of samtools.
Cai
--- 12年4月18日，周三, Ross ross.laza...@gmail.com 写道：
发件人: Ross ross.laza...@gmail.com
主题: Re: [galaxy-dev] Exception: Error Setting BAM Metadata: /bin/sh:
samtools: not found
收件人: zhengqiu cai caizhq2...@yahoo.com.cn
抄送: Jennifer Jackson j...@bx.psu.edu, galaxy-dev@lists.bx.psu.edu
日期: 2012年4月18日,周三,上午5:18
Ah. Then you probably have a library
path problem.
Here's how we do it FWIW:
Assume your want your shared cluster apps in
/data/apps/bin
To share the compiled libraries and binaries together,
remember to
compile apps like samtools using the
--prefix /data/app
flag so then you can add the /data/app/lib path to the
galaxy user's
path so the nodes running jobs inherit those paths because
of the -V
switch.
I hope this works!
On Tue, Apr 17, 2012 at 5:10 PM, zhengqiu cai caizhq2...@yahoo.com.cn
wrote:
Hi Ross,
Thank you for your reply.
I tried it, and it was still not working.
Only samtools has this problem, and the other tools
works fine.
samtools was compiled from source correctly. If I
install samtools using sudo apt-get install samtools, this
problem is gone, but this is not a solution for since it
does not work across the cluster.
Cai
--- 12年4月18日，周三, Ross ross.laza...@gmail.com
写道：
发件人: Ross ross.laza...@gmail.com
主题: Re: [galaxy-dev] Exception: Error Setting
BAM Metadata: /bin/sh: samtools: not found
收件人: zhengqiu cai caizhq2...@yahoo.com.cn
抄送: Jennifer Jackson j...@bx.psu.edu,
galaxy-dev@lists.bx.psu.edu
日期: 2012年4月18日,周三,上午4:59
Cai - if you are using SGE you
probably want to pass the Galaxy user's
path to the job using the
-V
switch in the system wide (or Galaxy user's home
directory)
sge_request file
eg I have
rlazarus@iaas1-int:~$ cat
/var/lib/gridengine/default/common/sge_request
-cwd
-V
Without that switch, the job gets no path.
With -V, it inherits the job submitter's full
path.
On Tue, Apr 17, 2012 at 4:26 PM, zhengqiu cai
caizhq2...@yahoo.com.cn
wrote:
Hi Jen,
Thank you for your quick response.
I changed the env.sh file to what you
seggested, and
the problem still existed.
I am pasting more details below:
I submitted a job to convert sam to bam, and
the job
was running forever without outputing the result. I
then
checked the log, and it read:
Traceback (most recent call last):
File
/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py,
line 336, in finish_job
drm_job_state.job_wrapper.finish(
stdout,
stderr )
File
/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/__init__.py,
line 637, in finish
dataset.set_meta( overwrite =
False )
File
/mnt/galaxyTools/galaxy-dist/lib/galaxy/model/__init__.py,
line 875, in set_meta
return self.datatype.set_meta(
self, **kwd
)
File
/mnt/galaxyTools/galaxy-dist/lib/galaxy/datatypes/binary.py,
line 179, in set_meta
raise Exception, Error Setting
BAM
Metadata: %s % stderr
Exception: Error Setting BAM Metadata:
/bin/sh:
samtools: not found
It means that the samtools is not in the PATH.
I tried
to set the PATH in a couple of methods according
the Galaxy
documentation:
1. put the path in the env.sh in the tool
directory and
symbolink default to the tool directory, e.g.
default -
=/mnt/galaxyTools/tools/samtools/0.1.18
2. put -v
PATH=/mnt/galaxyTools/tools/samtools/0.1.18
in ~/.sge_request
3. put -v
PATH=/mnt/galaxyTools/tools/samtools/0.1.18
in /path/sge_request
none of them worked, and I got the above same
problem.
Then I checked the job log file in the
job_working_directory, and it read:
Samtools Version: 0.1.18 (r982:295)
SAM file converted to BAM
which shows that sge knows the PATH of
samtools. To
double check it, I added samtools index to Galaxy,
and it
worked well. I am very
Re: [galaxy-dev] Fetch taxonomic representation DOESN'T WORK
--- 12年4月18日，周三, JIE CHEN jiechenable1...@gmail.com 写道：
发件人: JIE CHEN jiechenable1...@gmail.com
主题: Re: [galaxy-dev] Fetch taxonomic representation DOESN'T WORK
收件人: galaxy-dev@lists.bx.psu.edu
日期: 2012年4月18日,周三,上午5:36
On Tue, Apr 17, 2012 at 1:00 PM, JIE CHEN jiechenable1...@gmail.com wrote:
Thank you Nate. Your advise is very helpful to me. I installed the binaries
like you said. Right now the binaries can be run successfully in the command
line prompt. However i run into other problems :
Failed to open input file:
/usr/local/projects/galaxy-dist/tool-data/taxonomy/names.dmp
What i have done:
1. downloaded the .tar.gz file from the link you told me
2. tar zxvf name.tar.gz
3. then i found the three executables which are: taxBuilder, taxonomy2tree and
tree2PS-fast.
4. made symbolic link of these files to the /usr/local/bin
what else shall i do? Thanks.
Tyler
The file may not exist. To generate the dependent files, you can try the
following steps (change the path to yours):
Go to /mnt/galaxyTools/galaxy-central/scripts/taxonomy
Run processTaxonomy.sh, which will generate the file gi_taxid_sorted.txt and
other files.
Link those files to the directory
/mnt/galaxyTools/galaxy-central/tool-data/taxonomy
mkdir /mnt/galaxyTools/galaxy-central/tool-data/taxonomy
ls|awk '{print ln -s /mnt/galaxyIndices/AdditionalData/taxonomy/$1
/mnt/galaxyTools/galaxy-central/tool-data/taxonomy/$1}'|sh
zhengqiu cai
On Tue, Apr 17, 2012 at 10:36 AM, Nate Coraor n...@bx.psu.edu wrote:
On Apr 17, 2012, at 1:32 PM, JIE CHEN wrote:
Dear all,
I was trying to run the build-in function Fetch taxonomic representation
on my local instance of Galaxy, however it doesn't work as I expect.
The error message is:
/bin/sh: taxBuilder: not found
Anyone has seen this problem before? Please tell me what to do to solve this
problem? A lot of thanks
Hi Tyler,
Some Galaxy tools depend on external binaries not included in the Galaxy
distribution, and this is one of those tools. The full list and where to get
the tools from can be found here:
http://wiki.g2.bx.psu.edu/Admin/Tools/Tool%20Dependencies
This specific tool can be found at:
https://bitbucket.org/natefoo/taxonomy
--nate
Tyler
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
-下面为附件内容-
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] Exception: Error Setting BAM Metadata: /bin/sh: samtools: not found
Hi, Cai - sorry to hear it's still not working right.
You've compiled/installed samtools using the --prefix configure option
and tested that all the samtools based Galaxy wrappers work if you use
the local job runner? That means that both the appropriate lib and bin
directories are on the Galaxy user's path ready to be inherited by SGE
with the -V SGE option.
If the local job runner works, that seems to isolate the problem to
SGE configuration and operation - the steps I've suggested seem to
work for us so I've run out of ideas on why it's not working at your
shop and hope somebody with more experience can help out.
On Wed, Apr 18, 2012 at 10:50 AM, zhengqiu cai caizhq2...@yahoo.com.cn wrote:
Hi Ross,
Thank you for your reply.
It still did not work.
I think SGE knows the path of samtools. Here is a test case I added to
Galaxy, and it creates the index for fasta file. I pasted the samtools faidx
wrapper and xml files below:
ubuntu@master:/mnt/galaxyTools/galaxy-dist/tools/samtools$ more
samtools_faidx.pl
#!/usr/bin/perl
use strict;
use warnings;
if (@ARGV 1) {
die Usage: perl $0 input fasta file\n;
}
`samtools faidx $ARGV[0]`;
`cp ${ARGV[0]}.fai $ARGV[1]`;
exit;
ubuntu@master:/mnt/galaxyTools/galaxy-dist/tools/samtools$ more
samtools_faidx.xml
tool id=samtools_faidx name=samtools faidx version=1.0.3
requirements
requirement type=packagesamtools/requirement
/requirements
descriptioncreates fasta index/description
command interpreter=perl
samtools_faidx.pl $input1 $output1
/command
inputs
param name=input1 type=data format=fasta label=fasta file to be
indexed /
/inputs
outputs
data format=fai name=output1 label=${tool.name} on ${on_string}:
fasta /
/outputs
/tool
It works, which shows that SGE knows the path of samtools.
Cai
--- 12年4月18日，周三, Ross ross.laza...@gmail.com 写道：
发件人: Ross ross.laza...@gmail.com
主题: Re: [galaxy-dev] Exception: Error Setting BAM Metadata: /bin/sh:
samtools: not found
收件人: zhengqiu cai caizhq2...@yahoo.com.cn
抄送: Jennifer Jackson j...@bx.psu.edu, galaxy-dev@lists.bx.psu.edu
日期: 2012年4月18日,周三,上午5:18
Ah. Then you probably have a library
path problem.
Here's how we do it FWIW:
Assume your want your shared cluster apps in
/data/apps/bin
To share the compiled libraries and binaries together,
remember to
compile apps like samtools using the
--prefix /data/app
flag so then you can add the /data/app/lib path to the
galaxy user's
path so the nodes running jobs inherit those paths because
of the -V
switch.
I hope this works!
On Tue, Apr 17, 2012 at 5:10 PM, zhengqiu cai caizhq2...@yahoo.com.cn
wrote:
Hi Ross,
Thank you for your reply.
I tried it, and it was still not working.
Only samtools has this problem, and the other tools
works fine.
samtools was compiled from source correctly. If I
install samtools using sudo apt-get install samtools, this
problem is gone, but this is not a solution for since it
does not work across the cluster.
Cai
--- 12年4月18日，周三, Ross ross.laza...@gmail.com
写道：
发件人: Ross ross.laza...@gmail.com
主题: Re: [galaxy-dev] Exception: Error Setting
BAM Metadata: /bin/sh: samtools: not found
收件人: zhengqiu cai caizhq2...@yahoo.com.cn
抄送: Jennifer Jackson j...@bx.psu.edu,
galaxy-dev@lists.bx.psu.edu
日期: 2012年4月18日,周三,上午4:59
Cai - if you are using SGE you
probably want to pass the Galaxy user's
path to the job using the
-V
switch in the system wide (or Galaxy user's home
directory)
sge_request file
eg I have
rlazarus@iaas1-int:~$ cat
/var/lib/gridengine/default/common/sge_request
-cwd
-V
Without that switch, the job gets no path.
With -V, it inherits the job submitter's full
path.
On Tue, Apr 17, 2012 at 4:26 PM, zhengqiu cai
caizhq2...@yahoo.com.cn
wrote:
Hi Jen,
Thank you for your quick response.
I changed the env.sh file to what you
seggested, and
the problem still existed.
I am pasting more details below:
I submitted a job to convert sam to bam, and
the job
was running forever without outputing the result. I
then
checked the log, and it read:
Traceback (most recent call last):
File
/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py,
line 336, in finish_job
drm_job_state.job_wrapper.finish(
stdout,
stderr )
File
/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/__init__.py,
line 637, in finish
dataset.set_meta( overwrite =
False )
File
/mnt/galaxyTools/galaxy-dist/lib/galaxy/model/__init__.py,
line 875, in set_meta
return self.datatype.set_meta(
self, **kwd
)
File
/mnt/galaxyTools/galaxy-dist/lib/galaxy/datatypes/binary.py,
line 179, in set_meta
raise Exception, Error Setting
BAM
Metadata: %s % stderr
Exception: Error Setting BAM Metadata:
/bin/sh:
samtools: not found
It means that the samtools is not in the PATH.
I tried
to set the PATH in a couple of
Re: [galaxy-dev] Exception: Error Setting BAM Metadata: /bin/sh: samtools: not found
Hi Ross,
I did not compile samtools using the --prefix. The samtools I used comes with
Galaxy AMI image, and I copied it to the same path on my EC2. How to compile
samtools using --prefix? I just typed 'make', and there is not a configuration
file. In the cloudman version of Galaxy (it is working well for me), only two
executables (samtools and maq2sam-long) in the directory
/mnt/galaxyTools/tools/samtools/0.1.12. I copies the same files to my EC2. Is
the only compiled executable 'samtools' fully functional for the cluster(it is
on the local machine)? or should I set some other libraries?
Thank you,
Cai
--- 12年4月18日，周三, Ross ross.laza...@gmail.com 写道：
发件人: Ross ross.laza...@gmail.com
主题: Re: [galaxy-dev] Exception: Error Setting BAM Metadata: /bin/sh:
samtools: not found
收件人: zhengqiu cai caizhq2...@yahoo.com.cn
抄送: Jennifer Jackson j...@bx.psu.edu, galaxy-dev@lists.bx.psu.edu
日期: 2012年4月18日,周三,下午11:07
Hi, Cai - sorry to hear it's still
not working right.
You've compiled/installed samtools using the --prefix
configure option
and tested that all the samtools based Galaxy wrappers work
if you use
the local job runner? That means that both the appropriate
lib and bin
directories are on the Galaxy user's path ready to be
inherited by SGE
with the -V SGE option.
If the local job runner works, that seems to isolate the
problem to
SGE configuration and operation - the steps I've suggested
seem to
work for us so I've run out of ideas on why it's not working
at your
shop and hope somebody with more experience can help out.
On Wed, Apr 18, 2012 at 10:50 AM, zhengqiu cai caizhq2...@yahoo.com.cn
wrote:
Hi Ross,
Thank you for your reply.
It still did not work.
I think SGE knows the path of samtools. Here is a test
case I added to Galaxy, and it creates the index for fasta
file. I pasted the samtools faidx wrapper and xml files
below:
ubuntu@master:/mnt/galaxyTools/galaxy-dist/tools/samtools$
more samtools_faidx.pl
#!/usr/bin/perl
use strict;
use warnings;
if (@ARGV 1) {
die Usage:
perl $0 input fasta file\n;
}
`samtools faidx $ARGV[0]`;
`cp ${ARGV[0]}.fai $ARGV[1]`;
exit;
ubuntu@master:/mnt/galaxyTools/galaxy-dist/tools/samtools$
more samtools_faidx.xml
tool id=samtools_faidx name=samtools faidx
version=1.0.3
requirements
requirement
type=packagesamtools/requirement
/requirements
descriptioncreates fasta
index/description
command interpreter=perl
samtools_faidx.pl $input1 $output1
/command
inputs
param name=input1 type=data
format=fasta label=fasta file to be indexed /
/inputs
outputs
data format=fai name=output1
label=${tool.name} on ${on_string}: fasta /
/outputs
/tool
It works, which shows that SGE knows the path of
samtools.
Cai
--- 12年4月18日，周三, Ross ross.laza...@gmail.com
写道：
发件人: Ross ross.laza...@gmail.com
主题: Re: [galaxy-dev] Exception: Error Setting
BAM Metadata: /bin/sh: samtools: not found
收件人: zhengqiu cai caizhq2...@yahoo.com.cn
抄送: Jennifer Jackson j...@bx.psu.edu,
galaxy-dev@lists.bx.psu.edu
日期: 2012年4月18日,周三,上午5:18
Ah. Then you probably have a library
path problem.
Here's how we do it FWIW:
Assume your want your shared cluster apps in
/data/apps/bin
To share the compiled libraries and binaries
together,
remember to
compile apps like samtools using the
--prefix /data/app
flag so then you can add the /data/app/lib path to
the
galaxy user's
path so the nodes running jobs inherit those paths
because
of the -V
switch.
I hope this works!
On Tue, Apr 17, 2012 at 5:10 PM, zhengqiu cai
caizhq2...@yahoo.com.cn
wrote:
Hi Ross,
Thank you for your reply.
I tried it, and it was still not working.
Only samtools has this problem, and the other
tools
works fine.
samtools was compiled from source correctly.
If I
install samtools using sudo apt-get install
samtools, this
problem is gone, but this is not a solution for
since it
does not work across the cluster.
Cai
--- 12年4月18日，周三, Ross ross.laza...@gmail.com
写道：
发件人: Ross ross.laza...@gmail.com
主题: Re: [galaxy-dev] Exception: Error
Setting
BAM Metadata: /bin/sh: samtools: not found
收件人: zhengqiu cai caizhq2...@yahoo.com.cn
抄送: Jennifer Jackson j...@bx.psu.edu,
galaxy-dev@lists.bx.psu.edu
日期:
2012年4月18日,周三,上午4:59
Cai - if you are using SGE you
probably want to pass the Galaxy user's
path to the job using the
-V
switch in the system wide (or Galaxy
user's home
directory)
sge_request file
eg I have
rlazarus@iaas1-int:~$ cat
/var/lib/gridengine/default/common/sge_request
-cwd
-V
Without that switch, the job gets no
path.
With -V, it inherits the job submitter's
full
path.
On Tue, Apr 17, 2012 at 4:26 PM, zhengqiu
cai
caizhq2...@yahoo.com.cn
wrote:
Hi Jen,
Thank you for your quick response.
Re: [galaxy-dev] Exception: Error Setting BAM Metadata: /bin/sh: samtools: not found
Hi, Cai.
Apologies but I had not noticed any reference to the cloud
distribution before - that changes everything - I was talking about a
normal cluster/SGE install - you should definitely not need to do
anything at all to the AMI - maybe someone with cloudman experience
can be more helpful than I can.
On Wed, Apr 18, 2012 at 11:39 AM, zhengqiu cai caizhq2...@yahoo.com.cn wrote:
Hi Ross,
I did not compile samtools using the --prefix. The samtools I used comes with
Galaxy AMI image, and I copied it to the same path on my EC2. How to compile
samtools using --prefix? I just typed 'make', and there is not a
configuration file. In the cloudman version of Galaxy (it is working well for
me), only two executables (samtools and maq2sam-long) in the directory
/mnt/galaxyTools/tools/samtools/0.1.12. I copies the same files to my EC2. Is
the only compiled executable 'samtools' fully functional for the cluster(it
is on the local machine)? or should I set some other libraries?
Thank you,
Cai
--- 12年4月18日，周三, Ross ross.laza...@gmail.com 写道：
发件人: Ross ross.laza...@gmail.com
主题: Re: [galaxy-dev] Exception: Error Setting BAM Metadata: /bin/sh:
samtools: not found
收件人: zhengqiu cai caizhq2...@yahoo.com.cn
抄送: Jennifer Jackson j...@bx.psu.edu, galaxy-dev@lists.bx.psu.edu
日期: 2012年4月18日,周三,下午11:07
Hi, Cai - sorry to hear it's still
not working right.
You've compiled/installed samtools using the --prefix
configure option
and tested that all the samtools based Galaxy wrappers work
if you use
the local job runner? That means that both the appropriate
lib and bin
directories are on the Galaxy user's path ready to be
inherited by SGE
with the -V SGE option.
If the local job runner works, that seems to isolate the
problem to
SGE configuration and operation - the steps I've suggested
seem to
work for us so I've run out of ideas on why it's not working
at your
shop and hope somebody with more experience can help out.
On Wed, Apr 18, 2012 at 10:50 AM, zhengqiu cai caizhq2...@yahoo.com.cn
wrote:
Hi Ross,
Thank you for your reply.
It still did not work.
I think SGE knows the path of samtools. Here is a test
case I added to Galaxy, and it creates the index for fasta
file. I pasted the samtools faidx wrapper and xml files
below:
ubuntu@master:/mnt/galaxyTools/galaxy-dist/tools/samtools$
more samtools_faidx.pl
#!/usr/bin/perl
use strict;
use warnings;
if (@ARGV 1) {
die Usage:
perl $0 input fasta file\n;
}
`samtools faidx $ARGV[0]`;
`cp ${ARGV[0]}.fai $ARGV[1]`;
exit;
ubuntu@master:/mnt/galaxyTools/galaxy-dist/tools/samtools$
more samtools_faidx.xml
tool id=samtools_faidx name=samtools faidx
version=1.0.3
requirements
requirement
type=packagesamtools/requirement
/requirements
descriptioncreates fasta
index/description
command interpreter=perl
samtools_faidx.pl $input1 $output1
/command
inputs
param name=input1 type=data
format=fasta label=fasta file to be indexed /
/inputs
outputs
data format=fai name=output1
label=${tool.name} on ${on_string}: fasta /
/outputs
/tool
It works, which shows that SGE knows the path of
samtools.
Cai
--- 12年4月18日，周三, Ross ross.laza...@gmail.com
写道：
发件人: Ross ross.laza...@gmail.com
主题: Re: [galaxy-dev] Exception: Error Setting
BAM Metadata: /bin/sh: samtools: not found
收件人: zhengqiu cai caizhq2...@yahoo.com.cn
抄送: Jennifer Jackson j...@bx.psu.edu,
galaxy-dev@lists.bx.psu.edu
日期: 2012年4月18日,周三,上午5:18
Ah. Then you probably have a library
path problem.
Here's how we do it FWIW:
Assume your want your shared cluster apps in
/data/apps/bin
To share the compiled libraries and binaries
together,
remember to
compile apps like samtools using the
--prefix /data/app
flag so then you can add the /data/app/lib path to
the
galaxy user's
path so the nodes running jobs inherit those paths
because
of the -V
switch.
I hope this works!
On Tue, Apr 17, 2012 at 5:10 PM, zhengqiu cai
caizhq2...@yahoo.com.cn
wrote:
Hi Ross,
Thank you for your reply.
I tried it, and it was still not working.
Only samtools has this problem, and the other
tools
works fine.
samtools was compiled from source correctly.
If I
install samtools using sudo apt-get install
samtools, this
problem is gone, but this is not a solution for
since it
does not work across the cluster.
Cai
--- 12年4月18日，周三, Ross ross.laza...@gmail.com
写道：
发件人: Ross ross.laza...@gmail.com
主题: Re: [galaxy-dev] Exception: Error
Setting
BAM Metadata: /bin/sh: samtools: not found
收件人: zhengqiu cai caizhq2...@yahoo.com.cn
抄送: Jennifer Jackson j...@bx.psu.edu,
galaxy-dev@lists.bx.psu.edu
日期:
2012年4月18日,周三,上午4:59
Cai - if you are using SGE you
probably want to pass the Galaxy user's
path to the job using the
-V
switch in the system wide (or Galaxy
user's home
Re: [galaxy-dev] Exception: Error Setting BAM Metadata: /bin/sh: samtools: not found
Hi Ross,
I have a cloudman version of Galaxy working well. What I am doing now is trying
to install Galaxy on starcluster, which is a SGE cluster. I should followed the
install method for the normal cluster/SGE install, not for the cloud.
Thank you for your quick reply.
Have a nice day.
Cai
--- 12年4月18日，周三, Ross ross.laza...@gmail.com 写道：
发件人: Ross ross.laza...@gmail.com
主题: Re: [galaxy-dev] Exception: Error Setting BAM Metadata: /bin/sh:
samtools: not found
收件人: zhengqiu cai caizhq2...@yahoo.com.cn
抄送: Jennifer Jackson j...@bx.psu.edu, galaxy-dev@lists.bx.psu.edu
日期: 2012年4月18日,周三,下午11:46
Hi, Cai.
Apologies but I had not noticed any reference to the cloud
distribution before - that changes everything - I was
talking about a
normal cluster/SGE install - you should definitely not need
to do
anything at all to the AMI - maybe someone with cloudman
experience
can be more helpful than I can.
On Wed, Apr 18, 2012 at 11:39 AM, zhengqiu cai caizhq2...@yahoo.com.cn
wrote:
Hi Ross,
I did not compile samtools using the --prefix. The
samtools I used comes with Galaxy AMI image, and I copied it
to the same path on my EC2. How to compile samtools using
--prefix? I just typed 'make', and there is not a
configuration file. In the cloudman version of Galaxy (it is
working well for me), only two executables (samtools and
maq2sam-long) in the directory
/mnt/galaxyTools/tools/samtools/0.1.12. I copies the same
files to my EC2. Is the only compiled executable 'samtools'
fully functional for the cluster(it is on the local
machine)? or should I set some other libraries?
Thank you,
Cai
--- 12年4月18日，周三, Ross ross.laza...@gmail.com
写道：
发件人: Ross ross.laza...@gmail.com
主题: Re: [galaxy-dev] Exception: Error Setting
BAM Metadata: /bin/sh: samtools: not found
收件人: zhengqiu cai caizhq2...@yahoo.com.cn
抄送: Jennifer Jackson j...@bx.psu.edu,
galaxy-dev@lists.bx.psu.edu
日期: 2012年4月18日,周三,下午11:07
Hi, Cai - sorry to hear it's still
not working right.
You've compiled/installed samtools using the
--prefix
configure option
and tested that all the samtools based Galaxy
wrappers work
if you use
the local job runner? That means that both the
appropriate
lib and bin
directories are on the Galaxy user's path ready to
be
inherited by SGE
with the -V SGE option.
If the local job runner works, that seems to
isolate the
problem to
SGE configuration and operation - the steps I've
suggested
seem to
work for us so I've run out of ideas on why it's
not working
at your
shop and hope somebody with more experience can
help out.
On Wed, Apr 18, 2012 at 10:50 AM, zhengqiu cai
caizhq2...@yahoo.com.cn
wrote:
Hi Ross,
Thank you for your reply.
It still did not work.
I think SGE knows the path of samtools. Here
is a test
case I added to Galaxy, and it creates the index
for fasta
file. I pasted the samtools faidx wrapper and xml
files
below:
ubuntu@master:/mnt/galaxyTools/galaxy-dist/tools/samtools$
more samtools_faidx.pl
#!/usr/bin/perl
use strict;
use warnings;
if (@ARGV 1) {
die Usage:
perl $0 input fasta file\n;
}
`samtools faidx $ARGV[0]`;
`cp ${ARGV[0]}.fai $ARGV[1]`;
exit;
ubuntu@master:/mnt/galaxyTools/galaxy-dist/tools/samtools$
more samtools_faidx.xml
tool id=samtools_faidx name=samtools
faidx
version=1.0.3
requirements
requirement
type=packagesamtools/requirement
/requirements
descriptioncreates fasta
index/description
command interpreter=perl
samtools_faidx.pl $input1
$output1
/command
inputs
param name=input1
type=data
format=fasta label=fasta file to be indexed
/
/inputs
outputs
data format=fai
name=output1
label=${tool.name} on ${on_string}: fasta /
/outputs
/tool
It works, which shows that SGE knows the path
of
samtools.
Cai
--- 12年4月18日，周三, Ross ross.laza...@gmail.com
写道：
发件人: Ross ross.laza...@gmail.com
主题: Re: [galaxy-dev] Exception: Error
Setting
BAM Metadata: /bin/sh: samtools: not found
收件人: zhengqiu cai caizhq2...@yahoo.com.cn
抄送: Jennifer Jackson j...@bx.psu.edu,
galaxy-dev@lists.bx.psu.edu
日期:
2012年4月18日,周三,上午5:18
Ah. Then you probably have a library
path problem.
Here's how we do it FWIW:
Assume your want your shared cluster apps
in
/data/apps/bin
To share the compiled libraries and
binaries
together,
remember to
compile apps like samtools using the
--prefix /data/app
flag so then you can add the /data/app/lib
path to
the
galaxy user's
path so the nodes running jobs inherit
those paths
because
of the -V
switch.
I hope this works!
On Tue, Apr 17, 2012 at 5:10 PM, zhengqiu
cai
caizhq2...@yahoo.com.cn
wrote:
Hi Ross,
Thank you for your reply.
I tried it, and it was still not
working.
Only samtools has this
Re: [galaxy-dev] Displaying genomic sequences in Trackster
Hi Naharajan, My best guess is that you've found some bugs. Fortunately, they are likely fixed in galaxy-central and will soon be available in galaxy-dist (we're planning an update in the next couple days). Once galaxy-dist is updated, please try updating your Galaxy instance and seeing if that fixes the problems. If not, let us know and we can take a closer look. J. On Apr 18, 2012, at 9:05 AM, Naharajan Lakshmanaperumal wrote: Dear all, We have our own galaxy instance and the idea is to have trackster enabled for users to be able to visualize NGS mapping. We were able to configure trackster in our instance and the visualization works fine. We have two questions regarding trackster: 1) We can't display genomic sequences in trackster. As per the tutorial, we set the location of the .2bit file in the twobit.loc file for the trackster to be able to display the genomic sequence but for some reason it doesn't display it. The name of the builds is the same in all places i.e) in ucsc/chrom/builds.txt and also in the .loc files. Any ideas on what else should be done? 2) While saving the visualization, there is always an error message saying could not save visualization and it doesn't seem to be a web browser issue. How do we then save the visualization? Thanks in advance, Naharajan ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Potential database corruption with local galaxy instance
% sh manage_db.sh downgrade 92 % sh manage_db.sh upgrade The downgrade to 92 and upgrade created job.params. This is progress. You should be able to run jobs again, yes? Unfortunately, we are still getting errors about duplicate key values. The debug output when I try to export a history to a file is shown below my signature. Is there anything that was updated recently that would change primary keys? Primary keys are handled by SQLAlchemy, not Galaxy, so that's not the problem. I would guess the issue arose due to the missing 'params' column in job. This can likely be fixed by deleting all the rows in the job_history_export_table: DELETE FROM job_export_history_archive; The only downside to this operation is that existing history archives won't be found and will have to be recreated. Best, J. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Merging BLAST database support into Galaxy?
sounds great, thanks peter. i granted you access to my toolshed repo, but perhaps we want only one tool in the toolshed when all done. On Wed, Apr 18, 2012 at 3:20 AM, Peter Cock p.j.a.c...@googlemail.comwrote: On Wed, Apr 18, 2012 at 10:53 AM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi Edward, We're now running BLAST+ searches on our local Galaxy via our cluster, and some of the cluster nodes have relatively small amounts of RAM. This means I've become more aware of limitations in the NCBI BLAST+ tools' support for using a subject FASTA file (instead of making a local BLAST database), which turns out to be surprisingly RAM hungry. The logical step is to allow users to build a BLAST database as a new datatype in Galaxy - which is what you (Edward) did some time ago as a fork, later posted to the Galaxy Tool Shed. Edward - are you happy for me to merge your work into the main wrappers? I mentioned idea this a couple of months ago: http://lists.bx.psu.edu/pipermail/galaxy-dev/2012-February/008544.html Note this will take some extra work - we need to support protein BLAST databases as well, not just nucleotide database. Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Merging BLAST database support into Galaxy?
On Wed, Apr 18, 2012 at 6:10 PM, Edward Kirton eskir...@lbl.gov wrote: sounds great, thanks peter. i granted you access to my toolshed repo, but perhaps we want only one tool in the toolshed when all done. Thanks - but given the tools currently live in the main repository with Galaxy itself, I'll be focussing my efforts there for now - and adding bits of code from your work as appropriate (with a credit). Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Private key file format issue (for connecting EC2 instance with EBS volume)
Hello, I am trying to connect an EC2 instance to an EBS volume. I have attached the EBS volume to the EC2 instance (running Galaxy). I now need to mount the EBS volume to the EC2 instance. To do this I will use Putty to make the connection. But first I have to convert the private key from .pem format to .ppk format. I am trying to use Puttygen for the conversion, but the .pem file is not recognized and I get this error message: Could't load private key (key file does not begin with OpenSSH key header) Does anyone have experience with this? BTW, I am using windows. Thanks! Cheers, Mo Heydarian ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Output files from Galaxy
I have installed local galaxy and am using it to run some programs. The output files in the history seems to be stored in the directory .../galaxy-dist/database/files/000/, there are files dataset_100.dat, dataset_120.dat etc. My question is that: If I run a program, is there a way to check which dataset is the output file for a running ? I think the number after underscore of dataset should be able to identify it ? right ? Where can I find this number ? Thanks a lot. Jianpeng ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Mira-Assembler: DOESN'T WORK ON GALAXY
Dear all, I just installed Mira-Assembler via Galaxy tool sheds. As expected, it downloaded the wrappers into the shed-tools directory. Then I downloaded and installed Mira binaries on the system and add it to $PATH. Right now, mira can run successfully in the command line prompt. However when i run it in the Galaxy UI, i run into the below error: Return error code 1 from command: mira --job=denovo,genome,accurate SANGER_SETTINGS -LR:lsd=1:mxti=0:ft=fastq -FN:fqi=/media/partition2_/galaxydb_data/000/dataset_290.dat SOLEXA_SETTINGS -LR:lsd=1:ft=fastq -FN:fqi=/media/partition2_/galaxydb_data/000/dataset_290.dat COMMON_SETTINGS -OUT:orf=1:orc=1:ora=1:orw=1:orm=0:org=0:ors=0 -OUT:rrot=1:rtd=1 Anyone know why? Thanks in advance. Tyler ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Mira-Assembler: DOESN'T WORK ON GALAXY
On Wed, Apr 18, 2012 at 11:21 PM, JIE CHEN jiechenable1...@gmail.com wrote: Dear all, I just installed Mira-Assembler via Galaxy tool sheds. As expected, it downloaded the wrappers into the shed-tools directory. Then I downloaded and installed Mira binaries on the system and add it to $PATH. Right now, mira can run successfully in the command line prompt. Which version of MIRA did you install? However when i run it in the Galaxy UI, i run into the below error: Return error code 1 from command: mira --job=denovo,genome,accurate SANGER_SETTINGS -LR:lsd=1:mxti=0:ft=fastq -FN:fqi=/media/partition2_/galaxydb_data/000/dataset_290.dat SOLEXA_SETTINGS -LR:lsd=1:ft=fastq -FN:fqi=/media/partition2_/galaxydb_data/000/dataset_290.dat COMMON_SETTINGS -OUT:orf=1:orc=1:ora=1:orw=1:orm=0:org=0:ors=0 -OUT:rrot=1:rtd=1 Anyone know why? Thanks in advance. Did you get anything in the MIRA log file (it should be in your history with text data even though it will be red as a failed job)? Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Moving data about the EC2/EBS/S3 spaces on the Amazon cloud.
Hi Mo, On Wed, Apr 18, 2012 at 4:53 AM, Mohammad Heydarian mheyd...@jhmi.eduwrote: Hello, I am new to cloud computing and am trying to use the Galaxy cloudman service through Amazon to analyse NGS data. I have a couple of questions regarding data transfer: If I am running an EC2 instance, with an EBS volume attached, how can I retrieve the data from the EBS volume? If the data is not accessible via Galaxy (in which case you can use the Save icon for a given dataset in your History), you will need to scp the files over to your local instance. A command like the following should work: [local] $ scp -i path to private key ubuntu@ec2-rest of instance DNS:path to files (see below) path on your local machine In addition, how do I view what files are in the EBS volume? If using CloudMan to setup a data volume, all the user data will be stored under /mnt/galaxyData/. There will be more directories there with the Galaxy ones under the 'files' subdir. Can I send the data to an S3 bucket? And how would I move data from the S3 bucket to a fresh EC2 instance? This needs to be done manually via ec2 tools, a combnation of boto and pyhton or a combination of scp and a GUI tools such as cyberduck. Assuming I can store my files on EBS (or S3), what is the best way to download them? From EBS, you must use scp; from S3 you can use the AWS console or a 3rd party tool (cyberduck) Can I set up a ftp transfer to (quickly) download my potentially very large files? CloudMan sets up an FTP server on the instance for use with Galaxy ( http://wiki.g2.bx.psu.edu/FTPUpload). For downloading files, you could probably leverage that server but it's likely to take some playing around with (here are instructions mirrored on the cloud instance that may be useful http://wiki.g2.bx.psu.edu/Admin/Config/Upload%20via%20FTP). Hope this clears up some of your questions, Enis Many thanks for any insight and direction! Cheers, Mo Heydarian ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Mira-Assembler: DOESN'T WORK ON GALAXY
The version I installed is : mira_3.4.0_prod_linux-gnu_x86_64_static On Wed, Apr 18, 2012 at 3:31 PM, Peter Cock p.j.a.c...@googlemail.comwrote: On Wed, Apr 18, 2012 at 11:21 PM, JIE CHEN jiechenable1...@gmail.com wrote: Dear all, I just installed Mira-Assembler via Galaxy tool sheds. As expected, it downloaded the wrappers into the shed-tools directory. Then I downloaded and installed Mira binaries on the system and add it to $PATH. Right now, mira can run successfully in the command line prompt. Which version of MIRA did you install? However when i run it in the Galaxy UI, i run into the below error: Return error code 1 from command: mira --job=denovo,genome,accurate SANGER_SETTINGS -LR:lsd=1:mxti=0:ft=fastq -FN:fqi=/media/partition2_/galaxydb_data/000/dataset_290.dat SOLEXA_SETTINGS -LR:lsd=1:ft=fastq -FN:fqi=/media/partition2_/galaxydb_data/000/dataset_290.dat COMMON_SETTINGS -OUT:orf=1:orc=1:ora=1:orw=1:orm=0:org=0:ors=0 -OUT:rrot=1:rtd=1 Anyone know why? Thanks in advance. Did you get anything in the MIRA log file (it should be in your history with text data even though it will be red as a failed job)? Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Private key file format issue (for connecting EC2 instance with EBS volume)
Mo, I just tested one of my amazon keys with PuTTYgen and the default settings and it worked. Can you verify the contents of the .pem file you're using for me? On windows I'd open it in http://notepad-plus-plus.org/ or a similar utility to see it correctly. The file should look something like this, with both start and end lines. -BEGIN RSA PRIVATE KEY- about 20 lines of random characters -END RSA PRIVATE KEY- And, lastly, the amazon documentation at http://docs.amazonwebservices.com/AmazonEC2/gsg/2006-06-26/ says to make sure the file ends with a newline character or it won't work with PuTTYgen, so that's something to look for. -Dannon On Apr 18, 2012, at 3:29 PM, Mohammad Heydarian wrote: Hello, I am trying to connect an EC2 instance to an EBS volume. I have attached the EBS volume to the EC2 instance (running Galaxy). I now need to mount the EBS volume to the EC2 instance. To do this I will use Putty to make the connection. But first I have to convert the private key from .pem format to .ppk format. I am trying to use Puttygen for the conversion, but the .pem file is not recognized and I get this error message: Could't load private key (key file does not begin with OpenSSH key header) Does anyone have experience with this? BTW, I am using windows. Thanks! Cheers, Mo Heydarian ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Cuffdiff output
Hi, I have ran Cuffdiff and got output files: transcript differential expression testing, gene differential expression testing. Could you please tell me how to get the significant differential expression gene and transcript ? Thanks, This e-mail message (including any attachments) is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this message (including any attachments) is strictly prohibited. If you have received this message in error, please contact the sender by reply e-mail message and destroy all copies of the original message (including attachments). ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Deleting datasets from server
Of course. I deleted lots of times. But be careful if some of the files are important. Once you deleted them, all the files will disappear from the user's history(or when you open them, you will run into errors). You should know the files you are deleting. On Wed, Apr 18, 2012 at 8:40 PM, diana michelle magbanua dmdrmagba...@gmail.com wrote: Hi there, I would just like to ask if I could just directly delete the .dat files from the database folder of Galaxy (or even just the tmp files). I am not allowed to delete any other file from the computer other than the ones in galaxy-dist. I have tried running the cleanup scripts, but since the disk is already full, I am experiencing some problems with the cleanup. I'm just new to using this application, so any help would greatly be appreciated. Cheers, Michelle ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Deleting datasets from server
This would work for both .dat and tmp files? I was thinking about deleting them manually but the idea of them being connected to a database was bothering me. Other than minor errors, will there be no other serious repercussions? Thanks for the reply! DM On Thu, Apr 19, 2012 at 1:34 PM, JIE CHEN jiechenable1...@gmail.com wrote: Of course. I deleted lots of times. But be careful if some of the files are important. Once you deleted them, all the files will disappear from the user's history(or when you open them, you will run into errors). You should know the files you are deleting. On Wed, Apr 18, 2012 at 8:40 PM, diana michelle magbanua dmdrmagba...@gmail.com wrote: Hi there, I would just like to ask if I could just directly delete the .dat files from the database folder of Galaxy (or even just the tmp files). I am not allowed to delete any other file from the computer other than the ones in galaxy-dist. I have tried running the cleanup scripts, but since the disk is already full, I am experiencing some problems with the cleanup. I'm just new to using this application, so any help would greatly be appreciated. Cheers, Michelle ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Diana Michelle DR Magbanua Professional Service C4 Rice Center International Rice Research Institute ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
