[galaxy-dev] Error starting Galaxy after update of March 2012
Hi,
I run upgrade: hg pull -u -r 40f1816d6857 on our local instance, and I'm
now unable to start Galaxy..
Here is the content of Galaxy_log:
Traceback (most recent call last):
File galaxy_dir/galaxy-dist/lib/galaxy/web/buildapp.py, line 81,
in app_factory
from galaxy.app import UniverseApplication
File galaxy_dir/galaxy-dist/lib/galaxy/app.py, line 11, in module
from galaxy.objectstore import build_object_store_from_config
File galaxy_dir/galaxy-dist/lib/galaxy/objectstore/__init__.py,
line 27, in module
from galaxy.objectstore.s3_multipart_upload import multipart_upload
File
galaxy_dir/galaxy-dist/lib/galaxy/objectstore/s3_multipart_upload.py,
line 22, in module
eggs.require('boto')
File galaxy_dir/galaxy-dist/lib/galaxy/eggs/__init__.py, line 415,
in require
raise EggNotFetchable( str( [ egg.name for egg in e.eggs ] ) )
EggNotFetchable: ['boto']
Removing PID file galaxy_dir/galaxy-dist/galaxy.pid
Is that an error specific to our instance?
Is there a way to downgrade Galaxy?
Thanks by advance for your answer..
Alban
--
Alban Lermine
Unité 900 : Inserm - Mines ParisTech - Institut Curie
« Bioinformatics and Computational Systems Biology of Cancer »
11-13 rue Pierre et Marie Curie (1er étage) - 75005 Paris - France
Tel : +33 (0) 1 56 24 69 84
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
[galaxy-dev] Memory leaks with pbs job runner
Hi, We emailed previously about possible memory leaks in our installation of Galaxy here on the HPC at Bristol. We can run Galaxy just fine on our login node but when we integrate into the cluster using pbs job runner the whole thing falls over - almost certainly due to a memory leak. In essence, every attempt to submit a TopHat job (with 2x5GB paired end reads to the full human genome) always results in the whole thing falling over - but not when Galaxy is restricted to the login node. We saw that Nate responded to Todd Oakley about a week ago saying that there is a memory leak in libtorque or pbs_python when using the pbs job runner. Have there been any developments on this ? Best Wishes, David. __ Dr David A. Matthews Senior Lecturer in Virology Room E49 Department of Cellular and Molecular Medicine, School of Medical Sciences University Walk, University of Bristol Bristol. BS8 1TD U.K. Tel. +44 117 3312058 Fax. +44 117 3312091 d.a.matth...@bristol.ac.uk ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] bug with Galaxy form validation and dynamic_options population of drop-down menus where option values are integers
Hi all, Thanks Hans for giving a hint as to what we might need to try, we tried pasing True and False for the is selected field and the Galaxy form validation still doesn't like it, turns out it's the option value fields that have to be stringified in your JSON encoder so that you JSON reads: [One, 1, True ], [Two, 2, False ], Hope this helps others with the same issue, if you want to pass numbers as the option values make sure to stringify them. regards, Leandro On Thu, Mar 1, 2012 at 10:34 AM, Hans-Rudolf Hotz h...@fmi.ch wrote: Hi Leandro Although we use 'dynamic_options' successfully in many tools, it is still a little mystery to me sometimes - probably due to my non-existing python knowledge. Hence just double checking: Does [ One,1,True ], [ Two,2,False ], [ Three,3,False] work? Regards, Hans On 02/29/2012 04:44 PM, Leandro Hermida wrote: Hello, If you have a form with an input like this: param name=input1 type=select dynamic_options=get_options() / And the options dynamically generated have integers as values, e.g. options tuples like this: [ One, 1, 1 ], [ Two, 2, 0 ], [ Three, 3, 0 ] When you try to submit the galaxy form you always get the validation error An invalid option was selected, please verify. regards, Leandro ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Error starting Galaxy after update of March 2012
On Mar 13, 2012, at 11:08 AM, Alban Lermine wrote: Hi Alban, Is your Galaxy server behind a firewall or without network access such that it cannot access http://eggs.g2.bx.psu.edu/ ? What happens if you run `python scripts/fetch_eggs.py` by hand? Is there a way to downgrade Galaxy? Yes, in most cases you can downgrade with `hg update -r old changeset id`. --nate Thanks by advance for your answer.. Alban -- Alban Lermine Unité 900 : Inserm - Mines ParisTech - Institut Curie « Bioinformatics and Computational Systems Biology of Cancer » 11-13 rue Pierre et Marie Curie (1er étage) - 75005 Paris - France Tel : +33 (0) 1 56 24 69 84 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ Re, Thanks for your fast answer.. I run fetch_eggs.py by hand, here is the return: Fetched http://eggs.g2.bx.psu.edu/boto/boto-2.2.2-py2.6.egg Removed conflicting egg: galaxy_dir/galaxy-dist/eggs/boto-1.8d-py2.6.egg I tried then to start Galaxy, and I also have a new error message: Traceback (most recent call last): File /bioinfo/http/prod/hosted/galaxy.curie.fr/galaxy-dist/lib/galaxy/web/buildapp.py, line 82, in app_factory app = UniverseApplication( global_conf = global_conf, **kwargs ) File /bioinfo/http/prod/hosted/galaxy.curie.fr/galaxy-dist/lib/galaxy/app.py, line 24, in __init__ self.config.check() File /bioinfo/http/prod/hosted/galaxy.curie.fr/galaxy-dist/lib/galaxy/config.py, line 254, in check raise ConfigurationError( Unable to create missing directory: %s\n%s % ( path, e ) ) ConfigurationError: Unable to create missing directory: ./../shed_tools [Errno 13] Permission denied: './../shed_tools' It seems it's now only a permission problem.. (I only have rights on galaxy-dist directory), I will ask our system engineer to rerun update.. Hi Alban, You can fix this one by changing the path to the shed_tools directory in shed_tool_conf.xml. It can be any directory you can write to, but should not be underneath the galaxy-dist directory. --nate Alban -- Alban Lermine Unité 900 : Inserm - Mines ParisTech - Institut Curie « Bioinformatics and Computational Systems Biology of Cancer » 11-13 rue Pierre et Marie Curie (1er étage) - 75005 Paris - France Tel : +33 (0) 1 56 24 69 84 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Memory leaks with pbs job runner
On Mar 13, 2012, at 6:59 AM, David Matthews wrote: Hi, We emailed previously about possible memory leaks in our installation of Galaxy here on the HPC at Bristol. We can run Galaxy just fine on our login node but when we integrate into the cluster using pbs job runner the whole thing falls over - almost certainly due to a memory leak. In essence, every attempt to submit a TopHat job (with 2x5GB paired end reads to the full human genome) always results in the whole thing falling over - but not when Galaxy is restricted to the login node. We saw that Nate responded to Todd Oakley about a week ago saying that there is a memory leak in libtorque or pbs_python when using the pbs job runner. Have there been any developments on this ? Best Wishes, David. Hi David, I am almost certain that the problem you have with tophat is not due to the same leak, since it's a slow leak, not an immediate spike. Before we go any further, in reading back over our past conversation about this problem, I noticed that I never asked whether you've set `set_metadata_externally = True` in your Galaxy config. If not, this is almost certainly the cause of the problem. If you're already setting metadata externally, answers to a few of the questions I asked last time (or perhaps any findings of your HPC guys) and a few new things to try would be helpful in figuring out why your tophat jobs still crash: 1. Create a separate job runner and web frontend so we can be sure that the job running portion is the memory culprit: http://wiki.g2.bx.psu.edu/Admin/Config/Performance/Web%20Application%20Scaling You would not need any of the load balancing config, just start a single web process and a single runner process. From reading your prior email I believe you have a proxy server, and so as long as you start the web process on the same port as your previous Galaxy server, no change would be needed to your proxy server. 2. Set use_heartbeat = True in the config file of whichever process is consuming all of the memory. 3. Does the MemoryError appear in the log after Galaxy has noticed that the job has finished on the cluster (`(id/pbs id) PBS job has left queue`), but before the job post-processing is finished (`job id ended`)? 4. Does the MemoryError appear regardless of whether anyone accesses the web interface? There is another memory consumption problem we'll look at soon, which occurs when the job runner reads the metadata files written by the external set_metadata tool. If the output dataset(s) have an extremely large number of columns, this can cause a very large, nearly immediate memory spike when job post-processing begins, even if the output file itself is relatively small. --nate __ Dr David A. Matthews Senior Lecturer in Virology Room E49 Department of Cellular and Molecular Medicine, School of Medical Sciences University Walk, University of Bristol Bristol. BS8 1TD U.K. Tel. +44 117 3312058 Fax. +44 117 3312091 d.a.matth...@bristol.ac.uk ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] HMMER wrappers
Edward, I've marked your hmmscan repository as deleted so it will not longer be displayed. I can undelete it if requested by anyone. Peter, I've left your deprecated repositories alone, so they are still accessible. I'll mark them deleted as well if you want. On Mar 6, 2012, at 3:37 PM, Edward Kirton wrote: i gave it a bad name previously and peter didn't find it in a search at first, but i couldn't change the name, so i created a new repository. i put a note in the old repo (hmmscan), referring to the new repo (hmmer). that may suffice. however you folks how you want to handle it is fine. On Tue, Mar 6, 2012 at 3:38 AM, Greg Von Kuster g...@bx.psu.edu wrote: Hi Ed, Do you want the hmmscan repository itself deleted? It's been downloaded / cloned 86 times, although it is never been automatically installed into a local Galaxy since the contained tool does not properly load into Galaxy. We generally do not like to delete things like this because doing do prevents reproducibility. I'm looking for feedback from the community on this one - does eliminating this repository affect anyone? Thanks for the new contributions tot he tool shed! Greg Von Kuster On Mar 6, 2012, at 1:47 AM, Edward Kirton wrote: i created a new toolshed repo, hmmer since i couldn't rename it. as suggested, it has the hmmscan/hmmsearch as one tool, plus hmmpress. will add hmmbuild, hmmalign asap; others upon request. dave, is there a way to delete an old tool? (hmmscan) On Wed, Feb 29, 2012 at 10:12 AM, Edward Kirton eskir...@lbl.gov wrote: great suggestion; i'll make those changes On Wed, Feb 29, 2012 at 10:05 AM, Peter Cock p.j.a.c...@googlemail.com wrote: On Wed, Feb 29, 2012 at 5:56 PM, Edward Kirton eskir...@lbl.gov wrote: hi, peter - i will fix the description and upload hmmsearch and infernal today. Great. Assuming hmmscan and hmmsearch have (almost) the same command line API, there is something to be said for presenting them as one tool in Galaxy, with a drop down selection between them (with help text about which is recommend adapted from the HMMER blog post). One could even have an automatic selection by a wrapper script based on the number of query sequences and the number of HMMs. My thinking here is the detail of hmmscan vs hmmsearch is purely an implementation detail that the end user shouldn't have to worry about. Or just duplicate most of the XML code and have two wrappers. But as far as I know there isn't (yet) a nice way of reusing XML snippets between tool wrappers... which would be handy. Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Error starting Galaxy after update of March 2012
On Mar 13, 2012, at 10:36 AM, Alban Lermine wrote:
Hi,
I run upgrade: hg pull -u -r 40f1816d6857 on our local instance, and I'm
now unable to start Galaxy..
Here is the content of Galaxy_log:
Traceback (most recent call last):
File galaxy_dir/galaxy-dist/lib/galaxy/web/buildapp.py, line 81,
in app_factory
from galaxy.app import UniverseApplication
File galaxy_dir/galaxy-dist/lib/galaxy/app.py, line 11, in module
from galaxy.objectstore import build_object_store_from_config
File galaxy_dir/galaxy-dist/lib/galaxy/objectstore/__init__.py,
line 27, in module
from galaxy.objectstore.s3_multipart_upload import multipart_upload
File
galaxy_dir/galaxy-dist/lib/galaxy/objectstore/s3_multipart_upload.py,
line 22, in module
eggs.require('boto')
File galaxy_dir/galaxy-dist/lib/galaxy/eggs/__init__.py, line 415,
in require
raise EggNotFetchable( str( [ egg.name for egg in e.eggs ] ) )
EggNotFetchable: ['boto']
Removing PID file galaxy_dir/galaxy-dist/galaxy.pid
Is that an error specific to our instance?
Hi Alban,
Is your Galaxy server behind a firewall or without network access such that it
cannot access http://eggs.g2.bx.psu.edu/ ?
What happens if you run `python scripts/fetch_eggs.py` by hand?
Is there a way to downgrade Galaxy?
Yes, in most cases you can downgrade with `hg update -r old changeset id`.
--nate
Thanks by advance for your answer..
Alban
--
Alban Lermine
Unité 900 : Inserm - Mines ParisTech - Institut Curie
« Bioinformatics and Computational Systems Biology of Cancer »
11-13 rue Pierre et Marie Curie (1er étage) - 75005 Paris - France
Tel : +33 (0) 1 56 24 69 84
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
