Re: [galaxy-dev] substitute or replace a value?
Hi David Have you looked into the excellent Galaxy Unix-Tools package created by Gordon Assaf? http://hannonlab.cshl.edu/galaxy_unix_tools/index.html The Find Replace text in line/column tool is probably what you are looking for: http://hannonlab.cshl.edu/galaxy_unix_tools/galaxy.html#find_and_replace regards, Hans On 04/25/2011 11:20 PM, Jennifer Jackson wrote: Hello David, There are tools to select data based on defined attributes or to perform calculations on existing data (Text Manipulation, Filter and Sort), but as such no direct user-defined substitution tools. Very sorry for any inconvenience, Best, Jen Galaxy team On 2/23/11 9:26 AM, David Hoover wrote: Is there a tool for doing a global or column-based substitution in a text or tabular file? For example, replace all instances of the number 2 in a column to a number 1? David Hoover Helix Systems Staff http://helix.nih.gov ___ To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Staged Method for cluster running SGE?
Hi all, So far we've been running our local Galaxy instance on a single machine, but I would like to be able to offload (some) jobs onto our local SGE cluster. I've been reading https://bitbucket.org/galaxy/galaxy-central/wiki/Config/Cluster Unfortunately in our setup the SGE cluster head node is a different machine to the Galaxy server, and they do not (currently) have a shared file system. Once on the cluster, the head node and the compute nodes do have a shared file system. Therefore we will need some way of copying input data from the Galaxy server to the cluster, running the job, and once the job is done, copying the results back to the Galaxy server. The Staged Method on the wiki sounds relevant, but appears to be for TORQUE only (via pbs_python), not any of the other back ends (via DRMAA). Have I overlooked anything on the Cluster wiki page? Has anyone attempted anything similar, and could you offer any guidance or tips? Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] user and dataset management with LDAP: some questions
Hello everyone I'm updating the remaining questions: 1) We use LDAP for user registration and logging, would it be possible to retrieve automatically the LDAP users' groups and create the groups in Galaxy accordigly? (and of course put the user in their respective groups) 2) Is it possible, still using LDAP, to delete a user and all his datasets? 3) Is it possible to automatically suppress, for instance, any dataset that was added more than a couple of months ago? 4) Is there a not-so-intrusive way to add a column in the user list showing the disk space they use respectively? Thanks in advance for your help Regards, L-A Le mardi 12 avril 2011 à 16:41 +0200, Louise-Amélie Schmitt a écrit : Hello everyone I have a couple of questions regarding user and dataset management. 1) We use LDAP for user registration and logging, would it be possible to retrieve automatically the LDAP users' groups and create the groups in Galaxy accordigly? (and of course put the user in their respective groups) 2) Is it possible, still using LDAP, to delete a user and all his datasets? 3) Is it possible to automatically suppress, for instance, any dataset that was added more than a couple of months ago? 4) Is there a not-so-intrusive way to add a column in the user list showing the disk space they use respectively? 5) I tried to see how the API works but i have to admit I didn't get a thing. I read the scripts/api/README file and there I saw one needs to access the user preferences to generate an API key. What is its purpose? Is there a way to do it when you use LDAP (therefore no access to that key generator) ? Sorry, this is a bit random but I'm kinda drowning here, since I'm not used to manipulating apps this huge. Thanks for your help and patience. Cheers, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Staged Method for cluster running SGE?
On Tue, Apr 26, 2011 at 5:11 AM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi all, So far we've been running our local Galaxy instance on a single machine, but I would like to be able to offload (some) jobs onto our local SGE cluster. I've been reading https://bitbucket.org/galaxy/galaxy-central/wiki/Config/Cluster Unfortunately in our setup the SGE cluster head node is a different machine to the Galaxy server, and they do not (currently) have a shared file system. Once on the cluster, the head node and the compute nodes do have a shared file system. Therefore we will need some way of copying input data from the Galaxy server to the cluster, running the job, and once the job is done, copying the results back to the Galaxy server. The Staged Method on the wiki sounds relevant, but appears to be for TORQUE only (via pbs_python), not any of the other back ends (via DRMAA). Have I overlooked anything on the Cluster wiki page? Has anyone attempted anything similar, and could you offer any guidance or tips? Hi, Peter. You might consider setting up a separate queue for SGE jobs. Then, you could specify a prolog and epilog script that will copy files from the galaxy machine into the cluster (in the prolog) and back to galaxy (in the epilog). This assumes that there is a way to map from one file system to the other, but for Galaxy, that is probably the case (galaxy files on the galaxy server are under the galaxy instance and galaxy files on the cluster will probably be run as a single user in that home directory). I have not done this myself, but the advantage to using prolog and epilog scripts is that galaxy jobs then do not need any special configuration--all the work is done transparently by SGE. Sean ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Staged Method for cluster running SGE?
On Tue, Apr 26, 2011 at 12:10 PM, Sean Davis sdav...@mail.nih.gov wrote: On Tue, Apr 26, 2011 at 5:11 AM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi all, So far we've been running our local Galaxy instance on a single machine, but I would like to be able to offload (some) jobs onto our local SGE cluster. I've been reading https://bitbucket.org/galaxy/galaxy-central/wiki/Config/Cluster Unfortunately in our setup the SGE cluster head node is a different machine to the Galaxy server, and they do not (currently) have a shared file system. Once on the cluster, the head node and the compute nodes do have a shared file system. Therefore we will need some way of copying input data from the Galaxy server to the cluster, running the job, and once the job is done, copying the results back to the Galaxy server. The Staged Method on the wiki sounds relevant, but appears to be for TORQUE only (via pbs_python), not any of the other back ends (via DRMAA). Have I overlooked anything on the Cluster wiki page? Has anyone attempted anything similar, and could you offer any guidance or tips? Hi, Peter. You might consider setting up a separate queue for SGE jobs. Then, you could specify a prolog and epilog script that will copy files from the galaxy machine into the cluster (in the prolog) and back to galaxy (in the epilog). I see - the prolog and epilog scripts are per SGE queue, so in order to have Galaxy specific scripts, the simplest solution is a Galaxy specific queue. This assumes that there is a way to map from one file system to the other, but for Galaxy, that is probably the case (galaxy files on the galaxy server are under the galaxy instance and galaxy files on the cluster will probably be run as a single user in that home directory). We have a user account galaxy on the Galaxy Server, and it would make sense to have a matching user account galaxy on the Cluster which would submit the SGE jobs and own their data files. I have not done this myself, but the advantage to using prolog and epilog scripts is that galaxy jobs then do not need any special configuration--all the work is done transparently by SGE. Sean Thanks for the pointers - I have some reading ahead of me... Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] bwa failure preparing job
Does anybody have any idea why I would be getting this error before the tool runs? -- Branden Timm Great Lakes Bioenergy Research Center bt...@wisc.edu On 4/19/2011 10:45 AM, Branden Timm wrote: Hi All, I'm having issues running BWA for Illumina with the latest version of Galaxy (5433:c1aeb2f33b4a). It seems that the error is a python list error while preparing the job: Traceback (most recent call last): File /home/galaxy/galaxy-central/lib/galaxy/jobs/runners/local.py, line 58, in run_job job_wrapper.prepare() File /home/galaxy/galaxy-central/lib/galaxy/jobs/__init__.py, line 371, in prepare self.command_line = self.tool.build_command_line( param_dict ) File /home/galaxy/galaxy-central/lib/galaxy/tools/__init__.py, line 1575, in build_command_line command_line = fill_template( self.command, context=param_dict ) File /home/galaxy/galaxy-central/lib/galaxy/util/template.py, line 9, in fill_template return str( Template( source=template_text, searchList=[context] ) ) File /home/galaxy/galaxy-central/eggs/Cheetah-2.2.2-py2.6-linux-x86_64-ucs4.egg/Cheetah/Template.py, line 1004, in __str__ return getattr(self, mainMethName)() File DynamicallyCompiledCheetahTemplate.py, line 106, in respond IndexError: list index out of range I checked the bwa_index.loc file for errors, it seems that the line for the reference genome I'm trying to map against is correct (all whitespace is tab characters): synpcc7002 synpcc7002 Synechococcus /home/galaxy/galaxy-central/bwa_ indices/SYNPCC7002 I'm not sure what the next troubleshooting step is, any ideas? -- Branden Timm bt...@glbrc.wisc.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Problem with Map with Bowtie for Illumina on a local Galaxy
Hello Leon, Are you still experiencing problems with your local install? Could you confirm that you have attempted this again with the current distribution (April 8th) and have completed the NGS Tool set up as described in our wiki? Once we know everything is current it will be easier to offer more help. https://bitbucket.org/galaxy/galaxy-central/wiki/NGSLocalSetup galaxy-dist% hg pull -u -r 50e249442c5a https://bitbucket.org/galaxy/galaxy-central/wiki/Features/DevNewsBriefs Best! Jen Galaxy team On 3/19/11 10:30 AM, Leon Mei wrote: Dear colleagues, I have a local installed Galaxy. When I try to run bowtie to align a small FASTQ file, I got the following message and the result was empty. empty, format: sam, database: Project-Id-Version: PACKAGE VERSION Report-Msgid-Bugs-To: POT-Creation-Date: 2008-09-21 18:33+0900 PO-Revision-Date: 2009-03-17 03:55-0400 Last-Translator: FULL NAME Language-Team: en Plural-Forms: nplurals=2; plural=(n != 1) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Generated-By: Babel 0.9.4 Info: It seems bowtie_wrapper.py wasn't executed at all. Has someone seen this message before and what could be the cause? Thanks, Leon -- Jennifer Jackson http://usegalaxy.org http://galaxyproject.org ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] bwa failure preparing job
Hi Branden, I find a wiki documentation here, https://bitbucket.org/galaxy/galaxy-central/wiki/DataTables Hope this will help you to experiment a bit around. regards, Vipin Does anybody have any idea why I would be getting this error before the tool runs? -- Branden Timm Great Lakes Bioenergy Research Center bt...@wisc.edu On 4/19/2011 10:45 AM, Branden Timm wrote: Hi All, I'm having issues running BWA for Illumina with the latest version of Galaxy (5433:c1aeb2f33b4a). It seems that the error is a python list error while preparing the job: Traceback (most recent call last): File /home/galaxy/galaxy-central/lib/galaxy/jobs/runners/local.py, line 58, in run_job job_wrapper.prepare() File /home/galaxy/galaxy-central/lib/galaxy/jobs/__init__.py, line 371, in prepare self.command_line = self.tool.build_command_line( param_dict ) File /home/galaxy/galaxy-central/lib/galaxy/tools/__init__.py, line 1575, in build_command_line command_line = fill_template( self.command, context=param_dict ) File /home/galaxy/galaxy-central/lib/galaxy/util/template.py, line 9, in fill_template return str( Template( source=template_text, searchList=[context] ) ) File /home/galaxy/galaxy-central/eggs/Cheetah-2.2.2-py2.6-linux-x86_64-ucs4.egg/Cheetah/Template.py, line 1004, in __str__ return getattr(self, mainMethName)() File DynamicallyCompiledCheetahTemplate.py, line 106, in respond IndexError: list index out of range I checked the bwa_index.loc file for errors, it seems that the line for the reference genome I'm trying to map against is correct (all whitespace is tab characters): synpcc7002 synpcc7002 Synechococcus /home/galaxy/galaxy-central/bwa_ indices/SYNPCC7002 I'm not sure what the next troubleshooting step is, any ideas? -- Branden Timm bt...@glbrc.wisc.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] galaxy after logout
After logging out and then using the back button, what is it that you're able to do or that you're worried about? Any action on pages you click 'back' to get to should now present a You must be logged in to Galaxy... error message. To your question about disabling it, back button functionality is implemented on the client side, in your browser. The easiest solution here on your end would be to close the browser or individual browser tab after logging out. -Dannon On Apr 21, 2011, at 12:27 AM, hari krishna wrote: hi, In galaxy after logout by using browser back buttons i can able to go my home page .How to disable this facility can any one help -- Thanks Regards, Hari Krishna .M ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] sqlalchemy Timeout Error
You can certainly adjust the two values as needed for your particular environment. Also check in your universe_wsgi.ini for: set_metadata_externally = True Something else you might want to consider, and what we do to handle the load on the server at http://usegalaxy.org, is starting more instances to balance the load. For reference, our settings for pool_size and max_overflow are 10 and 20, respectively. See https://bitbucket.org/galaxy/galaxy-central/wiki/Config/ProductionServer for a good reference with more details on setting up a high performance galaxy server in a production environment suitable for multiple users. -Dannon On Apr 21, 2011, at 1:47 PM, Liisa Koski wrote: Hi, We are running many NGS workflows at the same time on our local instance of Galaxy. They are crashing with the following error Error - class 'sqlalchemy.exc.TimeoutError': QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30 In the universe_wsgi.ini we made the following adjustments: # If the server logs errors about not having enough database pool connections, # you will want to increase these values, or consider running more Galaxy # processes. database_engine_option_pool_size = 50 # this used to be 5 database_engine_option_max_overflow = 100 # this used to be 100 Those numbers were pulled out of a hat so I wanted to make sure what we were doing was correct. Is there a limit on the values? It doesn't appear to be crashing anymore but I still want to make sure. Thanks, Liisa___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Galaxy Community Conference Early Registration Deadline extended to May 1
Hello all, The 2011 Galaxy Community Conference registration site is now back online. See http://galaxy.psu.edu/gcc2011/Register.html for details. Early registration ends May 1. *May 24 Lodging:* As expected, we *were not able* to secure additional hotel rooms at the conference centre for May 24, the evening before the meeting. We encourage people who need a room on May 24 to stay in Amsterdam that night, and then take the train to Lunteren the next morning. There is ample time to arrive before the 9:30am start of the meeting on the 25th. We regret the inconvenience that this will cause for people who will not be able to stay at the conference centre on May 24. *May 25 and 26 Lodging: *We *were able* to reserve additional rooms for the night of May 26. We now have about 35 open hotel rooms reserved at the conference center on the nights of May 25 and May 26 (the night after the meeting). * However, we expect the rooms for May 25 to sell out, possibly in advance of the May 1 early registration deadline. If you are planning on staying at the conference centre, we strongly recommend that you register as soon as possible. *Once those rooms are full, the whole centre will be full.* * Thank you all for your patience while we worked on this. I apologize for and regret any inconvenience this has caused. As always, please let me know if you have any questions. Thanks, Dave C On Thu, Apr 21, 2011 at 11:12 PM, Dave Clements cleme...@galaxyproject.orgwrote: Hello all, The early registration deadline for the 2011 Galaxy Community Conference has been extended by a week, from April 24 to May 1. The conference is being held in Lunteren, the Netherlands, 25-26 May, with a special Introduction to Galaxy session on the evening of May 24. That's the good news. The bad news is that online registration has been taken offline, starting today, until April 26 or 27. We took this step because of unexpectedly high demand for conference centre hotel rooms on May 24 and 26. We still have ample rooms for the night of May 25,* but we are now sold out on the other nights*. We are working with the conference centre to increase our room allotment on May 24 and 26. We are optimistic of getting more rooms on May 26, but we are not as hopeful about getting additional rooms for May 24. We will reopen registration next Tuesday or Wednesday and post an update on the room situation, including alternatives, at that time. As always, please let me know if you have any questions. Thanks for your patience, Dave C. -- http://galaxy.psu.edu/gcc2011/ http://getgalaxy.org http://usegalaxy.org/ -- http://galaxy.psu.edu/gcc2011/ http://getgalaxy.org http://usegalaxy.org/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Question on finding external_filename and _extra_files_path from psql DB.
Hi Dannon, Thanks for pointing the same. Vipin, The numerical portion of the file path is not saved in the database, rather it is generated on the fly by the function directory_hash_id in lib/galaxy/model/__init__.py. The filename itself should be named in the format dataset_dataset id.dat. So, using that directory_hash_id function, you should be able to reconstruct the full path to the file on disk with just the unencoded dataset id and the root galaxy files directory prefix. -Dannon On Apr 25, 2011, at 4:16 PM, Vipin TS wrote: Dear Galaxy Team, As a part of in-house automated cleaning of data set from our production galaxy service, I am finding difficulty to get the galaxy assigned file name (galaxy-root/database/001/dataset_001.dat) for each data set from the corresponding database. We are using PostgreSQL and when I queried through the table name called 'dataset', couldn't figure out the values for external_filename or _extra_files_path. For example: select * from dataset order by id DESC limit 10; id |create_time |update_time | state | deleted | purged | purgable | external_filename | _extra_files_path | file_size 70805 | 2011-04-22 20:49:55.319709 | 2011-04-22 20:50:26.643807 | ok | f | f | t| | | 421593 Could you please let me know where I can locate the file name from other tables or do I need to set up any configuration. Thanks in advance, Vipin Friedrich Miescher Laboratory of the Max Planck Society Spemannstrasse 39, 72076 Tuebingen, Germany ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] bwa failure preparing job
Vipin, thanks for the tip. I was not aware of data tables at all. I checked bwa_wrapper.xml and it is still using the from_file attribute for the reference genome parameter, not from_data_table. It would appear then that BWA is not using data tables? Additionally, I have now noticed that our bowtie runs are failing as well with the error: AssertionError: Requested 'path' column missing from column def I looked at bowtie_wrapper.xml, and it too seems to still be using from_file instead of from_data_table for the reference genome drop-down. There is a line there using data tables, but it is commented out: !--options from_data_table=bowtie_indexes/-- I'm really confused as to what is going on here, but it seems like when I updated recently (first time since January probably) it broke all of my location files and I'm not sure how to fix them. I'm also confused because it seems that even using data tables, the format of my .loc files shouldn't need to change because they both use four columns separated by tabs. As always, any help is greatly appreciated. -- Branden Timm bt...@glbrc.wisc.edu On 4/26/2011 12:29 PM, Vipin TS wrote: Hi Branden, I find a wiki documentation here, https://bitbucket.org/galaxy/galaxy-central/wiki/DataTables Hope this will help you to experiment a bit around. regards, Vipin Does anybody have any idea why I would be getting this error before the tool runs? -- Branden Timm Great Lakes Bioenergy Research Center bt...@wisc.edu mailto:bt...@wisc.edu On 4/19/2011 10:45 AM, Branden Timm wrote: Hi All, I'm having issues running BWA for Illumina with the latest version of Galaxy (5433:c1aeb2f33b4a). It seems that the error is a python list error while preparing the job: Traceback (most recent call last): File /home/galaxy/galaxy-central/lib/galaxy/jobs/runners/local.py, line 58, in run_job job_wrapper.prepare() File /home/galaxy/galaxy-central/lib/galaxy/jobs/__init__.py, line 371, in prepare self.command_line = self.tool.build_command_line( param_dict ) File /home/galaxy/galaxy-central/lib/galaxy/tools/__init__.py, line 1575, in build_command_line command_line = fill_template( self.command, context=param_dict ) File /home/galaxy/galaxy-central/lib/galaxy/util/template.py, line 9, in fill_template return str( Template( source=template_text, searchList=[context] ) ) File /home/galaxy/galaxy-central/eggs/Cheetah-2.2.2-py2.6-linux-x86_64-ucs4.egg/Cheetah/Template.py, line 1004, in __str__ return getattr(self, mainMethName)() File DynamicallyCompiledCheetahTemplate.py, line 106, in respond IndexError: list index out of range I checked the bwa_index.loc file for errors, it seems that the line for the reference genome I'm trying to map against is correct (all whitespace is tab characters): synpcc7002 synpcc7002 Synechococcus /home/galaxy/galaxy-central/bwa_ indices/SYNPCC7002 I'm not sure what the next troubleshooting step is, any ideas? -- Branden Timm bt...@glbrc.wisc.edu mailto:bt...@glbrc.wisc.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
