Re: [galaxy-dev] Using a Galaxy image on a local system
Hi Alex Enis, Thanks very much! I've only recently got VMware installed and am having a look at the NBIC VM now. This is just the sort of thing I was looking for for quick start-up. I've heard of the afgane project - it has been suggested to us that this would be a good path to take for our local cloud. Is it specifically designed for creating AMIs, or is it really for automated configuration and deployment anywhere? Thanks, Clare On Fri, Apr 29, 2011 at 8:48 AM, Enis Afgan eaf...@emory.edu wrote: Hi Clare, Once you have a VM setup and accessible via ssh, you can also use our scripts for automated configuration and deployment of dependencies and tools. These scripts are used to setup Galaxy Cloud and they're targeted for Ubuntu 10.04 but should be applicable to other distributions as well. The scripts are available here: https://bitbucket.org/afgane/mi-deployment/overview Good luck, Enis On Thu, Apr 28, 2011 at 3:16 AM, Clare Sloggett s...@unimelb.edu.auwrote: Hi all, I would like to set up Galaxy locally. At the moment, I'm just trying to use it on my desktop (a Mac, OSX 10.6.7), but later we will want a local server to play with. Rather than install galaxy and then install all the tools it can use (and deal with OSX issues for some of them), it seems simpler just to use a virtual machine, since there are images which get regularly updated and come with pretty much everything. Is there anything wrong with this approach? I know there are Amazon EC2 images for Galaxy. So far as I know there are not other kinds of images? So for using it on my desktop, I think my options are either to run an EC2-compatible system locally, or to try to convert the AMI to a VMWare or VirtualBox image. I was just wondering if anyone has already tried either of these approaches? Also, is it possible to get hold of the galaxy AMI files themselves? Any advice welcome! Thanks, Clare ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Workflow steps stuck in job queue - since cleanup_datasets.py
Hi, Yesterday I ran the cleanup_datasets.py scripts as follows.. Deleting Userless Histories python cleanup_datasets.py universe_wsgi.ini -d 10 -1 Purging Deleted Histories python cleanup_datasets.py universe_wsgi.ini -d 10 -2 -r Purging Deleted Datasets python cleanup_datasets.py universe_wsgi.ini -d 10 -3 -r Purging Library Folders python cleanup_datasets.py universe_wsgi.ini -d 10 -5 -r Purging Libraries python cleanup_datasets.py universe_wsgi.ini -d 10 -4 -r Deleting Datasets / Purging Dataset Instances python cleanup_datasets.py universe_wsgi.ini -d 10 -6 -r This morning I noticed a number of workflows were either stuck at a certain step (ie..job running) or the step was grey (waiting in queue) but our cluster has free nodes. If I start a new workflow...it completes fine...just the 19 histories that were running yesterday are stuck. Did I do something wrong with the cleanup. Is there a way to restart these stuck histories without having to restart the entire workflow? Thanks in advance, Liisa ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Fresh galaxy installation tips/requests
Hi Assaf, Sorry for the delay in reply. Did you get help/resolve the problems you were having? Or do you still need help? Thanks! Jen Galaxy team On 3/30/11 3:12 PM, Assaf Gordon wrote: Hi all, It's been a long while since I had to install a fresh production galaxy server, and I can offer some tips or requests for minor improvements (but to give credit were it's due - installation is much smoother now, and DRMAA works great with SGE out of the box). These are just minor annoyances, but they can help make new installations easier. --- 1. Debugging REMOTE_USER issues. The mix of apache + authentication + non-root URL + load balancing is a killer. googling mod_rewrite + mod_auth shows that I'm not the only one having problems with it... Adding those two lines in ./lib/galaxy/web/framework/middleware/remoteuser.py, line 79 is a wonderful debugging tool, which enables you to see what Apache is passing on to Galaxy: for k,v in environ.items(): sys.stderr.write ( %s:\t%s\n % ( k, v ) ) I needed to print all environment variables (not just HTTP_REMOTE_USER) so I could debug mod_rewrite's variables setup. If there was an easy way to enable this debug from the INI file it would be a great help. --- 2. The file name universe_wsgi.ini is hard-coded in several places. Most of them are python scripts (like cleanup, ampq, etc.), but one is ./galaxy/eggs/init.py. Not a real problem, except that in the WebApplicationScaling wiki page it's kind of implied that universe_wsgi.ini is not used any more after creating the two separated ini files (universe_wsgi.runner.ini and universe_wsgi.webapp.ini). If you rename universe_wsgi.ini instead of copying it, galaxy will simply not start, even if given a different INI file name. So the best recommendation (in the documents) would probably be to maintain three identical INI files, except for the [server:main] part of each (and the enable_job_running = False in the runner). Most importantly, make sure that the database connection string is identical in all of them. As a side node, the cp command in the WebApplicationScaling wiki page is incorrect (you can't copy one file into two files). It'll also be good to mention that the cleanup scripts will always use universe_wsgi.ini regardless of what the galaxy server is using. --- 3. Saving SGE/PBS/DRMAA shell scripts if debug=TRUE in the jobrunner's INI file - this one is a gem, too bad I discovered it too late. If only it was documented :) --- 4. INI keys that accept multiple comma separated values (e.g. admin_users) are sensitive to white space. Example: admin_users = gor...@cshl.edu,foo...@cshl.edu works fine, but: admin_users = gor...@cshl.edu , foo...@cshl.edu will not work (spaces around the comma). Same thing with ucsc_display_sites: ucsc_display_sites = cshl,main will work (and display both cshl and main), but: ucsc_display_sites = cshl, main will display only cshl (because main is main with a space). It's probably easier to document it than to fix it, but either will do. --- 5. There's no ensembl_display_sites in the INI, so disabling the display at ensembl link (for BAM files, etc.) requires commenting out the corresponding display tag in datatypes_conf.xml, whereas for other servers (gbrowse,ucsc) it's enough to comment out the ucsc_display_sites in the INI. --- 6. For galaxy servers running inside an intranet without access to the main/test UCSC Genome browser server, and want to use a local mirror (this might be an obscure setup but that's what I have): disabling both main and test in ucsc_display_sites will disable the authenticated user bypass, because remouteuser.py have them hard-coded before setting self.allow_ucsc_main = True. I needed to patch and add cshl as a third option - this is kind of ugly. --- Two development issues not related to production server installation: 1. Debugging complex cheetah templates is very hard. If you have good tricks (beyond #slient and #breakpoint) - examples would be appriciated. If there's a way to see the compiled cheetah code which produced the error - it would be great. One problem is that the error is always reported without a line number (I guess because it's first converted into a string) and without the offending tag. 2. I only half-understand the new tool_data_table_conf.xml mechanism - is it explained somewhere ? Looks quite complicated, especially adding in-lined lamba for reading the __app__ global variable inside the cheetah code of each tool :( --- And user interface quirks: 1. when managing workflows, the configure your workflow menu page is a bit uninformative: there's no title, header, instruction or anything else, and the button says Submit Query which doesn't mean much (Update Workflow Menu would be better). If you click on configure your workflow menu and you don't have any workflows, the page is down right mysterious . 2. JQuery Tipsy balloons - please make
[galaxy-dev] Question on setting metadata on upload via API
I'm looking at extending the metadata fields for one of the supported file types. The files can get VERY large, and since I'm creating those files, I'd like to save as metadata some of the information I have on the contents. Specifically, I'd like to tag the files with information about the sample as well as the byte offsets in the file for every millionth record (to facilitate fast file splitting). I saw the sample script for uploading a file to a library via the API. It looks to me like the most direct approach would be to modify LibrariesController.create() to handle arbitrary key-value pairs to be set on the data file. However, that might not support things like having metadata values be arrays. So, does my understanding of the problem and the areas involved seem reasonable? Is there something already out there that does what I need? I saw the set_metadata.py script, but that looks like it is meant to operate directly on the files, which might not make a running instance of Galaxy too happy. Is there an API version of that (or something in one of the web controllers I can repurpose to my needs)? Thanks! John Duddy Illumina, Inc. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] quick question: how can i supply the user's email address to a tool?
is there a variable i can use in the tool config xml file? thanks! ed ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
