Re: [galaxy-dev] Variable number of inputs in Workflow (or optional inputs)
Hi Jennifer, Thanks for the reply. However, my question was not about building tools. My question was about building a workflow (using the workflow editor with existing tools), as a user of galaxy. Using the workflow editor, when I create a workflow using a tool that allows a variable number of inputs, it seems that I have to define the number of inputs during workflow creation and not when I run the workflow. I was just asking if it would be feasible to have a variable number of inputs (only defined when a running instance of a workflow is created). It feels to me that the answer is no. Thanks again, Daniel On Mon 05 Mar 2012 05:56:16 PM WET, Jennifer Jackson wrote: Hello Daniel, Examining existing tools that have multiple/optional inputs can be a good way to see how this is done. One example is the tool: NGS: QC and manipulation - Manipulate FASTQ. The 'Manipulate FASTQ' tool source is here: http://bitbucket.org/galaxy/galaxy-central/src/e58a87c91bc4/tools/fastq/ fastq_manipulation.py fastq_manipulation.xml The tool tag set is defined in this wiki; http://wiki.g2.bx.psu.edu/Admin/Tools/Tool%20Config%20Syntax See repeat, conditional, and when for more details and examples. Questions about tools are best sent to the galaxy-...@bx.psu.edu mailing list (the galaxy-user list is primarily for questions about data/tools usages on the main public instance). I am going to forward your question over there so that the development community can add to my reply in case they have a simpler way of doing this or other advice. Best, Jen Galaxy team On 3/5/12 7:32 AM, Daniel Sobral wrote: Hi, I wanted to build a workflow where the first step would be a tool that can have a variable number of inputs (e.g. a series). It seems that I need to predefine apriori the number of inputs to give to the tool. Is there a way to define the number of inputs at runtime? E.g. a series-like input? Thanks, Daniel ___ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using reply all in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Variable number of inputs in Workflow (or optional inputs)
Hello Daniel, I had the same problem when I started to work with Galaxy workflows. My workaround is: 1) Modify the first tool of the workflow Make it work on a simple text file that contains a list of input file paths (one by line) instead of several files. 2) Create a galaxy tools that can create this list from a galaxy history 3) Build a workflow with only one input slot for the first tool (the list) Hope this helps :) Have a nice day, Aurélien Le 06/03/2012 10:01, Daniel Sobral a écrit : Hi Jennifer, Thanks for the reply. However, my question was not about building tools. My question was about building a workflow (using the workflow editor with existing tools), as a user of galaxy. Using the workflow editor, when I create a workflow using a tool that allows a variable number of inputs, it seems that I have to define the number of inputs during workflow creation and not when I run the workflow. I was just asking if it would be feasible to have a variable number of inputs (only defined when a running instance of a workflow is created). It feels to me that the answer is no. Thanks again, Daniel On Mon 05 Mar 2012 05:56:16 PM WET, Jennifer Jackson wrote: Hello Daniel, Examining existing tools that have multiple/optional inputs can be a good way to see how this is done. One example is the tool: NGS: QC and manipulation - Manipulate FASTQ. The 'Manipulate FASTQ' tool source is here: http://bitbucket.org/galaxy/galaxy-central/src/e58a87c91bc4/tools/fastq/ fastq_manipulation.py fastq_manipulation.xml The tool tag set is defined in this wiki; http://wiki.g2.bx.psu.edu/Admin/Tools/Tool%20Config%20Syntax Seerepeat,conditional, andwhen for more details and examples. Questions about tools are best sent to the galaxy-...@bx.psu.edu mailing list (the galaxy-user list is primarily for questions about data/tools usages on the main public instance). I am going to forward your question over there so that the development community can add to my reply in case they have a simpler way of doing this or other advice. Best, Jen Galaxy team On 3/5/12 7:32 AM, Daniel Sobral wrote: Hi, I wanted to build a workflow where the first step would be a tool that can have a variable number of inputs (e.g. a series). It seems that I need to predefine apriori the number of inputs to give to the tool. Is there a way to define the number of inputs at runtime? E.g. a series-like input? Thanks, Daniel ___ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using reply all in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Aurélien Bernard IE Bioprogrammeur - CNRS Université des sciences Montpellier II Institut des Sciences de l'Evolution Bâtiment 22 - CC 064 Place Eugène Bataillon 34095 Montpellier cedex 5 France Tel : 04 67 14 32 61 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Variable number of inputs in Workflow (or optional inputs)
Hi Daniel, My apologies! I misunderstood what your question was about. You are correct, the required inputs are defined when the workflow is created. For tools that accept a variable number of inputs, when that tool is added to a workflow, the run-time parameters, including expected inputs, are defined. Did you have a particular tool in mind? We can ask Dannon for some suggestions/comments if you want to share your ideas about how this sort of processing would flow. I am glad you wrote back. Please send more details if you want and we can try to offer more (better!) help, Best, Jen Galaxy team On 3/6/12 1:01 AM, Daniel Sobral wrote: Hi Jennifer, Thanks for the reply. However, my question was not about building tools. My question was about building a workflow (using the workflow editor with existing tools), as a user of galaxy. Using the workflow editor, when I create a workflow using a tool that allows a variable number of inputs, it seems that I have to define the number of inputs during workflow creation and not when I run the workflow. I was just asking if it would be feasible to have a variable number of inputs (only defined when a running instance of a workflow is created). It feels to me that the answer is no. Thanks again, Daniel On Mon 05 Mar 2012 05:56:16 PM WET, Jennifer Jackson wrote: Hello Daniel, Examining existing tools that have multiple/optional inputs can be a good way to see how this is done. One example is the tool: NGS: QC and manipulation - Manipulate FASTQ. The 'Manipulate FASTQ' tool source is here: http://bitbucket.org/galaxy/galaxy-central/src/e58a87c91bc4/tools/fastq/ fastq_manipulation.py fastq_manipulation.xml The tool tag set is defined in this wiki; http://wiki.g2.bx.psu.edu/Admin/Tools/Tool%20Config%20Syntax Seerepeat,conditional, andwhen for more details and examples. Questions about tools are best sent to the galaxy-...@bx.psu.edu mailing list (the galaxy-user list is primarily for questions about data/tools usages on the main public instance). I am going to forward your question over there so that the development community can add to my reply in case they have a simpler way of doing this or other advice. Best, Jen Galaxy team On 3/5/12 7:32 AM, Daniel Sobral wrote: Hi, I wanted to build a workflow where the first step would be a tool that can have a variable number of inputs (e.g. a series). It seems that I need to predefine apriori the number of inputs to give to the tool. Is there a way to define the number of inputs at runtime? E.g. a series-like input? Thanks, Daniel ___ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using reply all in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Jennifer Jackson http://usegalaxy.org http://galaxyproject.org/wiki/Support ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] HMMER wrappers
Hi Ed, Do you want the hmmscan repository itself deleted? It's been downloaded / cloned 86 times, although it is never been automatically installed into a local Galaxy since the contained tool does not properly load into Galaxy. We generally do not like to delete things like this because doing do prevents reproducibility. I'm looking for feedback from the community on this one - does eliminating this repository affect anyone? Thanks for the new contributions tot he tool shed! Greg Von Kuster On Mar 6, 2012, at 1:47 AM, Edward Kirton wrote: i created a new toolshed repo, hmmer since i couldn't rename it. as suggested, it has the hmmscan/hmmsearch as one tool, plus hmmpress. will add hmmbuild, hmmalign asap; others upon request. dave, is there a way to delete an old tool? (hmmscan) On Wed, Feb 29, 2012 at 10:12 AM, Edward Kirton eskir...@lbl.gov wrote: great suggestion; i'll make those changes On Wed, Feb 29, 2012 at 10:05 AM, Peter Cock p.j.a.c...@googlemail.com wrote: On Wed, Feb 29, 2012 at 5:56 PM, Edward Kirton eskir...@lbl.gov wrote: hi, peter - i will fix the description and upload hmmsearch and infernal today. Great. Assuming hmmscan and hmmsearch have (almost) the same command line API, there is something to be said for presenting them as one tool in Galaxy, with a drop down selection between them (with help text about which is recommend adapted from the HMMER blog post). One could even have an automatic selection by a wrapper script based on the number of query sequences and the number of HMMs. My thinking here is the detail of hmmscan vs hmmsearch is purely an implementation detail that the end user shouldn't have to worry about. Or just duplicate most of the XML code and have two wrappers. But as far as I know there isn't (yet) a nice way of reusing XML snippets between tool wrappers... which would be handy. Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Using complete workflow in another workflow
Hello - I would also like the +1 this feature. I currently use the clone feature of a workflow to make a copy and modify, but if I change the original workflow, I think have to remember which workflows I cloned into and hand modify them as well. If we had the ability to re-use workflows, then we could develop modules that could be re-used in larger workflows. Thanks much for the consideration, Ann -- Message: 3 Date: Thu, 01 Mar 2012 20:14:25 + (GMT) From: thondeb...@me.com To: Galaxy-dev galaxy-dev@lists.bx.psu.edu Subject: [galaxy-dev] Using complete workflow in another workflow Message-ID: 7d9b3791-1822-e517-b5b9-e1ba1579b...@me.com Content-Type: text/plain; charset=utf-8; Format=flowed Hi, How far along are we about thinking about being able to re-use a complete workflow as a workflow step in another workflow? This would really allow us to modularize certain aspects of the analyses and would allow us to re-use a workflow in another. Barring that, a simple copy/paste from one workflow into another would also really help... Any plans in that direction? Thanks Thon? -- next part -- An HTML attachment was scrubbed... URL: http://lists.bx.psu.edu/pipermail/galaxy-dev/attachments/20120301/20f0e47 3/attachment-0001.html * ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Using complete workflow in another workflow
This is definitely something we're looking to implement in Galaxy. I started working on a proof of concept a while back for boxing workflows up as tools, but I have not had a chance to finish it yet. What Thon suggests in terms of a simple copy/paste is interesting, and probably a far simpler first step that I'll look into. I could imagine a configurable set of 'component' workflows that you could click and have the steps plopped into your existing workflow. -Dannon On Mar 6, 2012, at 10:25 AM, Ann Black wrote: Hello - I would also like the +1 this feature. I currently use the clone feature of a workflow to make a copy and modify, but if I change the original workflow, I think have to remember which workflows I cloned into and hand modify them as well. If we had the ability to re-use workflows, then we could develop modules that could be re-used in larger workflows. Thanks much for the consideration, Ann -- Message: 3 Date: Thu, 01 Mar 2012 20:14:25 + (GMT) From: thondeb...@me.com To: Galaxy-dev galaxy-dev@lists.bx.psu.edu Subject: [galaxy-dev] Using complete workflow in another workflow Message-ID: 7d9b3791-1822-e517-b5b9-e1ba1579b...@me.com Content-Type: text/plain; charset=utf-8; Format=flowed Hi, How far along are we about thinking about being able to re-use a complete workflow as a workflow step in another workflow? This would really allow us to modularize certain aspects of the analyses and would allow us to re-use a workflow in another. Barring that, a simple copy/paste from one workflow into another would also really help... Any plans in that direction? Thanks Thon? -- next part -- An HTML attachment was scrubbed... URL: http://lists.bx.psu.edu/pipermail/galaxy-dev/attachments/20120301/20f0e47 3/attachment-0001.html * ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] large ftp uploads to galaxy cloud
I'm having difficulty uploading files 6GB or so to an EC2 instantiation of galaxy cloud. The size seems to correlate with the amount of space available on the root volume. I can upload a 5.06 GB file but any larger and it never shows up in the galaxy upload history. I'm using a Large EC2 instance but the root volume is only 15GB and only has 5.9GB free. I'm using the 861460482541/galaxy-cloudman-2011-03-22 = Please note that this e-mail and any files transmitted from Memorial Sloan-Kettering Cancer Center may be privileged, confidential, and protected from disclosure under applicable law. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any reading, dissemination, distribution, copying, or other use of this communication or any of its attachments is strictly prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and deleting this message, any attachments, and all copies and backups from your computer. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Problems getting galaxy to start
Good evening, I am having difficulty getting galaxy to startup over the last couple of days, I initially suspected that it was an issue with the eggs, but have replaced these and the following errors remain, galaxy@jic55119: $ python --version Python 2.6.6 galaxy@jic55119:galaxy-central$ ./run.sh Traceback (most recent call last): File ./scripts/paster.py, line 34, in module command.run() File /home/home/galaxy/software/galaxy-central/eggs/PasteScript-1.7.3-py2.6.egg/paste/script/command.py, line 84, in run invoke(command, command_name, options, args[1:]) File /home/home/galaxy/software/galaxy-central/eggs/PasteScript-1.7.3-py2.6.egg/paste/script/command.py, line 123, in invoke exit_code = runner.run(args) File /home/home/galaxy/software/galaxy-central/eggs/PasteScript-1.7.3-py2.6.egg/paste/script/command.py, line 218, in run result = self.command() File /home/home/galaxy/software/galaxy-central/eggs/PasteScript-1.7.3-py2.6.egg/paste/script/serve.py, line 276, in command relative_to=base, global_conf=vars) File /home/home/galaxy/software/galaxy-central/eggs/PasteScript-1.7.3-py2.6.egg/paste/script/serve.py, line 313, in loadapp **kw) File /home/home/galaxy/software/galaxy-central/eggs/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py, line 204, in loadapp return loadobj(APP, uri, name=name, **kw) File /home/home/galaxy/software/galaxy-central/eggs/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py, line 224, in loadobj global_conf=global_conf) File /home/home/galaxy/software/galaxy-central/eggs/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py, line 248, in loadcontext global_conf=global_conf) File /home/home/galaxy/software/galaxy-central/eggs/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py, line 278, in _loadconfig return loader.get_context(object_type, name, global_conf) File /home/home/galaxy/software/galaxy-central/eggs/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py, line 413, in get_context section) File /home/home/galaxy/software/galaxy-central/eggs/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py, line 458, in _context_from_explicit value = import_string(found_expr) File /home/home/galaxy/software/galaxy-central/eggs/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py, line 18, in import_string return pkg_resources.EntryPoint.parse(x=+s).load(False) File /home/home/galaxy/software/galaxy-central/lib/pkg_resources.py, line 1954, in load entry = __import__(self.module_name, globals(),globals(), ['__name__']) File /home/home/galaxy/software/galaxy-central/lib/galaxy/web/buildapp.py, line 19, in module from galaxy import config, jobs, util, tools File /home/home/galaxy/software/galaxy-central/lib/galaxy/jobs/__init__.py, line 4, in module from galaxy import util, model EOFError: EOF read where object expected Any suggestions? Thanks in advance Michael. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Variable number of inputs in Workflow (or optional inputs)
Good Morning, We are also interested in this capability. To give a concrete example, we sometimes receive multiple sequence runs 1* for the same sample. We would like to be able to process each run of the sample through BWA and then merge them together, post process it a bit, and then send the merged bam file through the rest of our standard pipeline. Ideally this would be automated. Aurélien, I am interested in your workaround this might get us part of the way there, as we could concatenate the fastq files together and run BWA once. Would you be willing to share some of your custom tools for us to iterate on? But ideally, for performance, we would run the bwa steps in parallel. Is the galaxy team looking into these types of features or do other people have custom solutions they are using? Thanks so much, Ann ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Variable number of inputs in Workflow (or optional inputs)
This seems to fall within the recent discussion of map/reduce operators for workflows, e.g. breaking large jobs up for embarrassingly parallel tasks, then merging them back at some later point. Dannon mentioned some basic functionality does exist within Galaxy to do this, but it's at an early stage of development. Peter Cock did get some things to work with it. See this thread: http://thread.gmane.org/gmane.science.biology.galaxy.devel/4502/focus=4502 chris On Mar 6, 2012, at 9:56 AM, Ann Black wrote: Good Morning, We are also interested in this capability. To give a concrete example, we sometimes receive multiple sequence runs 1…* for the same sample. We would like to be able to process each run of the sample through BWA and then merge them together, post process it a bit, and then send the merged bam file through the rest of our standard pipeline. Ideally this would be automated. Aurélien, I am interested in your workaround – this might get us part of the way there, as we could concatenate the fastq files together and run BWA once. Would you be willing to share some of your custom tools for us to iterate on? But ideally, for performance, we would run the bwa steps in parallel. Is the galaxy team looking into these types of features or do other people have custom solutions they are using? Thanks so much, Ann ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] uploading multiple files into one dataset
On Feb 29, 2012, at 11:34 AM, Jorrit Boekel wrote: Dear list, Our lab's proteomics data is frequently outputted into 50 files containing different fractions of proteins. The files are locally stored and not present on the Galaxy server. We've planned to somehow (inside galaxy) merge these files and split them into tasks so they can be run on the cluster. We would either merge/split the files by concatenation, or untar/tar files at every job, depending on filetype and tool. No problems so far. However, I have been looking around for a way to upload 50 files simultaneously to galaxy and convert to one dataset, and this does not seem to be supported. Before starting to create a hack to make this work, which doesn't seem especially trivial to me, I'd like to know if I should instead use libraries. From what I've seen, libraries are not treated as datasets in Galaxy but rather contain datasets. If there was a tar all sets in library and import to history I'd be using that, but I've only encountered tar/zip sets and download locally which would be a bit of a workaround. Hi Jorrit, It's not possible to do this all in one step, but you can definitely upload them all simultaneously and then concatenate them using the concatenate tool (or write a simple tool to tar them). --nate I haven't found much on this subject in the mailing list, has this functionality been requested before? cheers, jorrit boekel ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Possible Galaxy memory leak?
There is also a memory leak that occurs when using the pbs job runner. I believe the leak is somewhere in libtorque or pbs_python but I have not yet tracked it down. --nate On Feb 29, 2012, at 3:28 PM, John Chilton wrote: We had intermittent issues like this for an embarrassingly long time at MSI, but since we applied this patch https://bitbucket.org/galaxy/galaxy-central/pull-request/36/fix-for-dynamic_optionspy Galaxy has been running smoothly for weeks. Seems unlikely this is the problem in your case (unless you have pulled in tools with make use of dynamic options), but it something to consider. -John On Wed, Feb 29, 2012 at 2:08 PM, Joshua Gross gros...@students.wwu.edu wrote: Hi all-- Our group is running a local instance of Galaxy on a Linux box. We have been noticing a memory issue, where if we leave the server running for a few days, soon all available memory (~8gb) will be used up to the point where the tools fail because they are unable to allocate new memory. While we haven't done any rigorous testing, this issue seems to be independent of the specific tools we run, and actually seems to still appear if we do not run any tools at all. The process python is what is taking up all the memory, and the issue is not instantaneous but happens over time. Restarting the server frees the memory immediately. Is this a known issue? If so, is there a known fix for it? Any insight on this would be much appreciated, as it is complicating our efforts to automate long work-flows. Thank you so much for your time Josh Gross, Western Washington University. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Potential issue with eggs ?
On Feb 29, 2012, at 11:50 AM, Michael Burrell (TOC) wrote: Good evening all, I am seeing the following issue when attempting to start galaxy. Traceback (most recent call last): File ./scripts/paster.py, line 34, in module command.run() File /home/home/galaxy/software/galaxy-central/eggs/PasteScript-1.7.3-py2.6.egg/paste/script/command.py, line 84, in run invoke(command, command_name, options, args[1:]) File /home/home/galaxy/software/galaxy-central/eggs/PasteScript-1.7.3-py2.6.egg/paste/script/command.py, line 123, in invoke exit_code = runner.run(args) File /home/home/galaxy/software/galaxy-central/eggs/PasteScript-1.7.3-py2.6.egg/paste/script/command.py, line 218, in run result = self.command() File /home/home/galaxy/software/galaxy-central/eggs/PasteScript-1.7.3-py2.6.egg/paste/script/serve.py, line 276, in command relative_to=base, global_conf=vars) File /home/home/galaxy/software/galaxy-central/eggs/PasteScript-1.7.3-py2.6.egg/paste/script/serve.py, line 313, in loadapp **kw) File /home/home/galaxy/software/galaxy-central/eggs/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py, line 204, in loadapp return loadobj(APP, uri, name=name, **kw) File /home/home/galaxy/software/galaxy-central/eggs/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py, line 224, in loadobj global_conf=global_conf) File /home/home/galaxy/software/galaxy-central/eggs/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py, line 248, in loadcontext global_conf=global_conf) File /home/home/galaxy/software/galaxy-central/eggs/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py, line 278, in _loadconfig return loader.get_context(object_type, name, global_conf) File /home/home/galaxy/software/galaxy-central/eggs/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py, line 413, in get_context section) File /home/home/galaxy/software/galaxy-central/eggs/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py, line 458, in _context_from_explicit value = import_string(found_expr) File /home/home/galaxy/software/galaxy-central/eggs/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py, line 18, in import_string return pkg_resources.EntryPoint.parse(x=+s).load(False) File /home/home/galaxy/software/galaxy-central/lib/pkg_resources.py, line 1954, in load entry = __import__(self.module_name, globals(),globals(), ['__name__']) File /home/home/galaxy/software/galaxy-central/lib/galaxy/web/buildapp.py, line 19, in module from galaxy import config, jobs, util, tools File /home/home/galaxy/software/galaxy-central/lib/galaxy/jobs/__init__.py, line 4, in module from galaxy import util, model EOFError: EOF read where object expected Removing PID file paster.pid Would this suggest that the eggs are bad? What would be the best way to remedy the situation Hi Michael, I think your eggs are okay, the occurrence of egg in the traceback is simply because the problem occurs beneath a call in the PasteDeploy egg. I think your copy of the source may be corrupt. Could you verify that things are okay by using 'hg status'? I'd also suggest finding and removing all .pyc files and letting Python recreate them. --nate Thanks Michael. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] [galaxy-user] Getting errors trying to enable samtools_mpileup
On Mar 1, 2012, at 8:49 AM, Waldron, Michael H wrote: I am running a local copy of Galaxy, last ran 'hg pull -u' on 2/20/12. I am trying to enable use of Mpileup for SAM Tools, and have added the entry for the samtools_mpileup.xml file in tool_conf.xml. However, when I startup Galaxy, the link for Mpileup does not appear in the Tools pane, and I see the following errors from the Galaxy startup: galaxy.tools ERROR 2012-02-29 16:28:51,919 error reading tool from path: samtools/samtools_mpileup.xml Traceback (most recent call last): File /nas02/apps/galaxy-prod/galaxy-dist/lib/galaxy/tools/__init__.py, line 212, in load_tool_tag_set tool = self.load_tool( os.path.join( tool_path, path ), guid=guid ) File /nas02/apps/galaxy-prod/galaxy-dist/lib/galaxy/tools/__init__.py, line 305, in load_tool return ToolClass( config_file, root, self.app, guid=guid ) File /nas02/apps/galaxy-prod/galaxy-dist/lib/galaxy/tools/__init__.py, line 531, in __init__ self.parse( root, guid=guid ) File /nas02/apps/galaxy-prod/galaxy-dist/lib/galaxy/tools/__init__.py, line 641, in parse self.parse_inputs( root ) File /nas02/apps/galaxy-prod/galaxy-dist/lib/galaxy/tools/__init__.py, line 723, in parse_inputs display, inputs = self.parse_input_page( page, enctypes ) File /nas02/apps/galaxy-prod/galaxy-dist/lib/galaxy/tools/__init__.py, line 922, in parse_input_page inputs = self.parse_input_elem( input_elem, enctypes ) File /nas02/apps/galaxy-prod/galaxy-dist/lib/galaxy/tools/__init__.py, line 989, in parse_input_elem case.inputs = self.parse_input_elem( case_elem, enctypes, context ) File /nas02/apps/galaxy-prod/galaxy-dist/lib/galaxy/tools/__init__.py, line 945, in parse_input_elem group.inputs = self.parse_input_elem( elem, enctypes, context ) File /nas02/apps/galaxy-prod/galaxy-dist/lib/galaxy/tools/__init__.py, line 1015, in parse_input_elem param = self.parse_param_elem( elem, enctypes, context ) File /nas02/apps/galaxy-prod/galaxy-dist/lib/galaxy/tools/__init__.py, line 1027, in parse_param_elem param = ToolParameter.build( self, input_elem ) File /nas02/apps/galaxy-prod/galaxy-dist/lib/galaxy/tools/parameters/basic.py, line 176, in build return parameter_types[param_type]( tool, param ) File /nas02/apps/galaxy-prod/galaxy-dist/lib/galaxy/tools/parameters/basic.py, line 1330, in __init__ ToolParameter.__init__( self, tool, elem ) File /nas02/apps/galaxy-prod/galaxy-dist/lib/galaxy/tools/parameters/basic.py, line 43, in __init__ self.validators.append( validation.Validator.from_element( self, elem ) ) File /nas02/apps/galaxy-prod/galaxy-dist/lib/galaxy/tools/parameters/validation.py, line 23, in from_element return validator_types[type].from_element( param, elem ) File /nas02/apps/galaxy-prod/galaxy-dist/lib/galaxy/tools/parameters/validation.py, line 279, in from_element tool_data_table = param.tool.app.tool_data_tables[ table_name ] File /nas02/apps/galaxy-prod/galaxy-dist/lib/galaxy/tools/data/__init__.py, line 21, in __getitem__ return self.data_tables.__getitem__( key ) KeyError: 'sam_fa_indexes' Can someone tell me what is wrong here? Am I missing something in the database? Hi Mike, I moved this over to the galaxy-dev list since it pertains to a local installation. You'll need to update your tool_data_table_conf.xml from the sample. If you haven't made any modifications, the simplest solution is to just `cp tool_data_table_conf.xml.sample tool_data_table_conf.xml`. --nate Thanks, Mike Waldron ___ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using reply all in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Possible Galaxy memory leak?
We've experience memory-leak symptoms and we also use the PBS job runner here. We did not try to locate the leak, but instead installed monit as a work-around http://mmonit.com/monit/ Todd Oakley On 3/6/2012 11:10 AM, Nate Coraor wrote: There is also a memory leak that occurs when using the pbs job runner. I believe the leak is somewhere in libtorque or pbs_python but I have not yet tracked it down. --nate On Feb 29, 2012, at 3:28 PM, John Chilton wrote: We had intermittent issues like this for an embarrassingly long time at MSI, but since we applied this patch https://bitbucket.org/galaxy/galaxy-central/pull-request/36/fix-for-dynamic_optionspy Galaxy has been running smoothly for weeks. Seems unlikely this is the problem in your case (unless you have pulled in tools with make use of dynamic options), but it something to consider. -John On Wed, Feb 29, 2012 at 2:08 PM, Joshua Grossgros...@students.wwu.edu wrote: Hi all-- Our group is running a local instance of Galaxy on a Linux box. We have been noticing a memory issue, where if we leave the server running for a few days, soon all available memory (~8gb) will be used up to the point where the tools fail because they are unable to allocate new memory. While we haven't done any rigorous testing, this issue seems to be independent of the specific tools we run, and actually seems to still appear if we do not run any tools at all. The process python is what is taking up all the memory, and the issue is not instantaneous but happens over time. Restarting the server frees the memory immediately. Is this a known issue? If so, is there a known fix for it? Any insight on this would be much appreciated, as it is complicating our efforts to automate long work-flows. Thank you so much for your time Josh Gross, Western Washington University. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- * Todd Oakley, Professor Ecology Evolution and Marine Biology University of California, Santa Barbara Santa Barbara, CA 93106 USA NEWLY UPDATED LAB WEBSITE: http://labs.eemb.ucsb.edu/oakley/todd/ ** ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] HMMER wrappers
i gave it a bad name previously and peter didn't find it in a search at first, but i couldn't change the name, so i created a new repository. i put a note in the old repo (hmmscan), referring to the new repo (hmmer). that may suffice. however you folks how you want to handle it is fine. On Tue, Mar 6, 2012 at 3:38 AM, Greg Von Kuster g...@bx.psu.edu wrote: Hi Ed, Do you want the hmmscan repository itself deleted? It's been downloaded / cloned 86 times, although it is never been automatically installed into a local Galaxy since the contained tool does not properly load into Galaxy. We generally do not like to delete things like this because doing do prevents reproducibility. I'm looking for feedback from the community on this one - does eliminating this repository affect anyone? Thanks for the new contributions tot he tool shed! Greg Von Kuster On Mar 6, 2012, at 1:47 AM, Edward Kirton wrote: i created a new toolshed repo, hmmer since i couldn't rename it. as suggested, it has the hmmscan/hmmsearch as one tool, plus hmmpress. will add hmmbuild, hmmalign asap; others upon request. dave, is there a way to delete an old tool? (hmmscan) On Wed, Feb 29, 2012 at 10:12 AM, Edward Kirton eskir...@lbl.gov wrote: great suggestion; i'll make those changes On Wed, Feb 29, 2012 at 10:05 AM, Peter Cock p.j.a.c...@googlemail.comwrote: On Wed, Feb 29, 2012 at 5:56 PM, Edward Kirton eskir...@lbl.gov wrote: hi, peter - i will fix the description and upload hmmsearch and infernal today. Great. Assuming hmmscan and hmmsearch have (almost) the same command line API, there is something to be said for presenting them as one tool in Galaxy, with a drop down selection between them (with help text about which is recommend adapted from the HMMER blog post). One could even have an automatic selection by a wrapper script based on the number of query sequences and the number of HMMs. My thinking here is the detail of hmmscan vs hmmsearch is purely an implementation detail that the end user shouldn't have to worry about. Or just duplicate most of the XML code and have two wrappers. But as far as I know there isn't (yet) a nice way of reusing XML snippets between tool wrappers... which would be handy. Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] bacterial genomes missing?
Hi Brad, I just ran the updateucsc.sh script and it seemed to work correctly. Can you be more specific as to which builds you are missing and could you try running the script again and if it doesn't work, copy the log output here? Thanks for using Galaxy, Dan On Mar 6, 2012, at 12:15 PM, Langhorst, Brad wrote: It seems that the bacterial genomes that used to come from ucsc via update_ucsc.sh are now gone... Has anybody else observed this? Fixed it? Brad -- Brad Langhorst langho...@neb.com 978-380-7564 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] bacterial genomes missing?
Hi Dan: When I look at builds.txt I see only the eukaryotic genomes... the canary that triggered my investigation was eschColi_K12. It used to be present, but is now not in builds.txt it is present in ucsc_build_sites.txt When i go to the xml at ucsc's test server http://genome-test.cse.ucsc.edu/cgi-bin/das/dsn i don't see eschColi_K12 I've looked in my backups and I find that the builds.txt from 3 weeks ago did have eschColi_K12 (and many other now missing items). I think we'll have to fetch these from the other sites now... Brad On Mar 6, 2012, at 4:59 PM, Daniel Blankenberg wrote: Hi Brad, I just ran the updateucsc.sh script and it seemed to work correctly. Can you be more specific as to which builds you are missing and could you try running the script again and if it doesn't work, copy the log output here? Thanks for using Galaxy, Dan On Mar 6, 2012, at 12:15 PM, Langhorst, Brad wrote: It seems that the bacterial genomes that used to come from ucsc via update_ucsc.sh are now gone... Has anybody else observed this? Fixed it? Brad -- Brad Langhorst langho...@neb.commailto:langho...@neb.com 978-380-7564 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Brad Langhorst langho...@neb.commailto:langho...@neb.com 978-380-7564 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] [galaxy-user] Advice on disabling SGE queue on galaxy-cloudman
Hi John, Since this question is about a Cloud installation, I am going to forward your question over to the galaxy-...@bx.psu.edu mailing list so that the development community will have a better chance of seeing it and providing feedback. http://wiki.g2.bx.psu.edu/Support#Mailing_Lists Thanks! Jen Galaxy team On 3/6/12 11:23 AM, John Major wrote: Hello All- I'd like to launch a galaxy-cloudman head node which does not accept SGE jobs, but as jobs are submitted go to compute nodes (or cause compute node to be added when auto-scale is on). Primarily, this is b/c I'd like to have the head node be a cheaper instance which can run long term, and only fire up more expensive compute nodes as they are actually needed. How would I enable this? Thanks- John ___ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using reply all in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Jennifer Jackson http://usegalaxy.org http://galaxyproject.org/wiki/Support ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] [galaxy-user] Problem using Galaxy on the cloud
Hi Jan, Since this question is about a Cloud installation, I am going to forward your question over to the galaxy-...@bx.psu.edu mailing list so that the development community will have a better chance of seeing it and providing feedback. http://wiki.g2.bx.psu.edu/Support#Mailing_Lists Thanks! Jen Galaxy team On 3/6/12 1:44 PM, Jan R McDowell wrote: Hi all, I have been trying to get an instance of Galaxy going on the EC2. I have no problem going through BioCloudCentral and getting an instance going. I can also successfully load my data from an S2 bucket into Galaxy. The problem occurs when I try to use velveth. I always says 'job waiting to run'. As a matter of curiosity, I then used SSH to get into CloudBioLinux, which worked. However, when I try to use NX to get the virtual desktop going I get the message usr/bin/nxserver: line 381: echo: write error: No space left on device. Using the df -h command, I get: FilesystemSize Used Avail Use% Mounted on /dev/xvda1 20G 19G 0 100% / udev 8.4G 4.0K 8.4G 1% /dev tmpfs 3.4G 660K 3.4G 1% /run none 5.0M 0 5.0M 0% /run/lock none 8.4G 0 8.4G 0% /run/shm /dev/xvdb 404G 202M 383G 1% /mnt /dev/xvdg1700G 654G 47G 94% /mnt/galaxyIndices /dev/xvdg2 10G 1.7G 8.4G 17% /mnt/galaxyTools /dev/xvdg3200G 11G 190G 6% /mnt/galaxyData So, I guess my question as a new user is: How do I point Galaxy and CloudBioLinux to all of this unused space? I assume the problem is with the /dev/xvda1 that is 100% full. I am obviously doing something silly and/or missing a really big step. Any help would be greatly appreciated. Many thanks in advance, Jan ___ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using reply all in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Jennifer Jackson http://usegalaxy.org http://galaxyproject.org/wiki/Support ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] HMMER wrappers
On Tuesday, March 6, 2012, Edward Kirton eskir...@lbl.gov wrote: i gave it a bad name previously and peter didn't find it in a search at first, but i couldn't change the name, so i created a new repository. i put a note in the old repo (hmmscan), referring to the new repo (hmmer). that may suffice. however you folks how you want to handle it is fine. Surely once the description was updated a search on hmmer would find the old repository? Well anyway, now there are two there should be a way to hide or deprecate one. On a related note, I have a couple of old tools in the ToolShed which I have just labeled as deprecated in the description (and if memory serves, marked as hidden in the XML, so existing workflows and history's can still run the tool, but no new usage should be possible). Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] [galaxy-user] Problem using Galaxy on the cloud
Jan; Glad to hear you got Galaxy running successfully. It sounds like everything is good to go once we sort out the disk space issue. However, when I try to use NX to get the virtual desktop going I get the message usr/bin/nxserver: line 381: echo: write error: No space left on device. Using the df -h command, I get: FilesystemSize Used Avail Use% Mounted on /dev/xvda1 20G 19G 0 100% / udev 8.4G 4.0K 8.4G 1% /dev tmpfs 3.4G 660K 3.4G 1% /run none 5.0M 0 5.0M 0% /run/lock none 8.4G 0 8.4G 0% /run/shm /dev/xvdb 404G 202M 383G 1% /mnt /dev/xvdg1700G 654G 47G 94% /mnt/galaxyIndices /dev/xvdg2 10G 1.7G 8.4G 17% /mnt/galaxyTools /dev/xvdg3200G 11G 190G 6% /mnt/galaxyData So, I guess my question as a new user is: How do I point Galaxy and CloudBioLinux to all of this unused space? By default CloudMan will put files into /mnt/galaxyData. However, as you noticed the main filesystem got filled up at some point. Could this have happened while transferring files over from S3? Are there files in your home directory that you could delete or move to /mnt/galaxyData to free up space? CloudBioLinux and CloudMan shouldn't put a large number of files in the root directory, but when the root filesystem is full it's going to be very unhappy. Once you manually clear up some room there hopefully things will run smoother. If that doesn't help let us know and we can dig into it further. Thanks, Brad ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Gal-Cloudman AMI Fails
Hello GalFolks- I saw a new galaxy-cloudman AMI appear ( 072133624695/galaxy-cloudman-2012-02-26 ) and was interested in giving it a whirl. But when I tried to launch it got the following error: The requested instance type's architecture (x86_64) does not match the architecture in the manifest for aki-407d9529 (i386) Is this a dev-AMI? Or should I be expecting this to run? Thanks- John ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Gal-Cloudman AMI Fails
John, I'm still working on figuring out who created the AMI you're running across (it isn't under the main Galaxy cloud account, see the '072133624695' in the name string), but for future reference we'll always keep the recommended AMI to use listed on usegalaxy.org/cloud. At this point, it's still ami-da58aab3 (861460482541/galaxy-cloudman-2011-03-22). -Dannon On Mar 6, 2012, at 10:47 PM, John Major wrote: Hello GalFolks- I saw a new galaxy-cloudman AMI appear ( 072133624695/galaxy-cloudman-2012-02-26 ) and was interested in giving it a whirl. But when I tried to launch it got the following error: The requested instance type's architecture (x86_64) does not match the architecture in the manifest for aki-407d9529 (i386) Is this a dev-AMI? Or should I be expecting this to run? Thanks- John ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
