Re: [galaxy-dev] Manage local data admin tool failed with dbkey KeyError
Dave, I upgraded the galaxy to Sep 7 release. The KeyError error has gone as expected. But the deferred_job_queue error persists. We are running galaxy in the cluster (SGE) here. Is it possible that Manage local data need to be configured in the cluster env? Regards, Derrick On Mon, Sep 10, 2012 at 10:59 PM, Dave Bouvier d...@bx.psu.edu wrote: Derrick, The deferred_job_queue instance in the job manager object is only instantiated when the enable_beta_job_managers variable is set to True. Is it possible that your previous setting of that variable was somehow reverted? The KeyError error you mentioned in your previous should be fixed in the latest -dist and -central revisions. --Dave B. On 9/9/12 21:20:44.000, Derrick Lin wrote: Hi guys, I found that the problematic loc files were created by CloudBioLinux and point to the reference indexes downloaded by CloudBioLinux script. I suspect that Manage Local Data cannot recognize the file structure created by CloudBioLinux? Anyhow, I removed all the loc files from CloudBioLinux, then Manage Local Data can run now. Soon after I tried to download the bowtie hg18 from ucsc, I was given another error caused by deferred job queue: URL: http://129.94.136.11:8080/**galaxy_dev/data_admin/**download_buildhttp://129.94.136.11:8080/galaxy_dev/data_admin/download_build File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/eggs/** WebError-0.8a-py2.6.egg/**weberror/evalexception/**middleware.py', line 364 in respond app_iter = self.application(environ, detect_start_response) File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/eggs/** Paste-1.6-py2.6.egg/paste/**debug/prints.py', line 98 in __call__ environ, self.app) File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/eggs/** Paste-1.6-py2.6.egg/paste/**wsgilib.py', line 539 in intercept_output app_iter = application(environ, replacement_start_response) File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/eggs/** Paste-1.6-py2.6.egg/paste/**recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/eggs/** Paste-1.6-py2.6.egg/paste/**httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/lib/** galaxy/web/framework/base.py', line 160 in __call__ body = method( trans, **kwargs ) File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/lib/** galaxy/web/framework/__init__.**py', line 184 in decorator return func( self, trans, *args, **kwargs ) File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/lib/** galaxy/web/controllers/data_**admin.py', line 199 in download_build jobid = trans.app.job_manager.**deferred_job_queue.plugins['** GenomeTransferPlugin'].create_**job( trans, url, dbkey, longname, indexers ) AttributeError: 'JobManager' object has no attribute 'deferred_job_queue' Anyone can give some hints about what was going on? Regards, Derrick On Fri, Sep 7, 2012 at 4:26 PM, Derrick Lin klin...@gmail.com mailto:klin...@gmail.com wrote: Hi guys, I enabled the tool with enable_beta_job_managers = True, server runs fine. But after I clicked on Manage local data, it gave the following error: URL: http://pwbc.garvan.unsw.edu.**au/galaxy_dev/data_admin/**manage_datahttp://pwbc.garvan.unsw.edu.au/galaxy_dev/data_admin/manage_data[^ http://pwbc.garvan.unsw.edu.**au/galaxy_dev/data_admin/**manage_datahttp://pwbc.garvan.unsw.edu.au/galaxy_dev/data_admin/manage_data ] File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/eggs/** WebError-0.8a-py2.6.egg/**weberror/evalexception/**middleware.py', line 364 in respond app_iter = self.application(environ, detect_start_response) File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/eggs/** Paste-1.6-py2.6.egg/paste/**debug/prints.py', line 98 in __call__ environ, self.app) File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/eggs/** Paste-1.6-py2.6.egg/paste/**wsgilib.py', line 539 in intercept_output app_iter = application(environ, replacement_start_response) File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/eggs/** Paste-1.6-py2.6.egg/paste/**recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/eggs/** Paste-1.6-py2.6.egg/paste/**httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/lib/** galaxy/web/framework/base.py', line 160 in __call__ body = method( trans, **kwargs ) File '/misc/PWBCAdminStore/app/**galaxy/galaxy_dist/dev/lib/** galaxy/web/framework/__init__.**py', line 184 in decorator return
[galaxy-dev] LDAP integration and migration of users
Hi all, I wonder if somebody with experience could enlighten me on these issues regarding authentication of users. - is it possible to have LDAP integration simultaneously with local Galaxy users. What I want to reach is when LDAP integration is working, that I can still add users through the Galaxy admin interface (users that are not present in the LDAP). - is there a way to transfer all histories, workflows and visualisation to another user? For example, an existing user receives a new email and password, and want to transfer all his/her stuff to this new account. Can this be done? I was thinking about when a user gets transferred to the LDAP, he/she would be able to transfer all data from the old account to the new account. Thanks, Joachim -- Joachim Jacob, PhD Rijvisschestraat 120, 9052 Zwijnaarde Tel: +32 9 244.66.34 Bioinformatics Training and Services (BITS) http://www.bits.vib.be @bitsatvib ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Manage local data admin tool failed with dbkey KeyError
Derrick, I have not actually tested the local data manager on CloudBioLinux, but that sounds like something worth looking into. As for the rsync server mentioned in another email, the data manager checks our rsync server first, then tries generating indexes if the rsync server doesn't have them. However, it will always produce indexes, as I decided to allow for the option to regenerate indexes, if the local ones had somehow become corrupted. --Dave B. On 9/11/12 02:03:45.000, Derrick Lin wrote: Dave, I upgraded the galaxy to Sep 7 release. The KeyError error has gone as expected. But the deferred_job_queue error persists. We are running galaxy in the cluster (SGE) here. Is it possible that Manage local data need to be configured in the cluster env? Regards, Derrick On Mon, Sep 10, 2012 at 10:59 PM, Dave Bouvier d...@bx.psu.edu mailto:d...@bx.psu.edu wrote: Derrick, The deferred_job_queue instance in the job manager object is only instantiated when the enable_beta_job_managers variable is set to True. Is it possible that your previous setting of that variable was somehow reverted? The KeyError error you mentioned in your previous should be fixed in the latest -dist and -central revisions. --Dave B. On 9/9/12 21:20:44.000, Derrick Lin wrote: Hi guys, I found that the problematic loc files were created by CloudBioLinux and point to the reference indexes downloaded by CloudBioLinux script. I suspect that Manage Local Data cannot recognize the file structure created by CloudBioLinux? Anyhow, I removed all the loc files from CloudBioLinux, then Manage Local Data can run now. Soon after I tried to download the bowtie hg18 from ucsc, I was given another error caused by deferred job queue: URL: http://129.94.136.11:8080/__galaxy_dev/data_admin/__download_build http://129.94.136.11:8080/galaxy_dev/data_admin/download_build File '/misc/PWBCAdminStore/app/__galaxy/galaxy_dist/dev/eggs/__WebError-0.8a-py2.6.egg/__weberror/evalexception/__middleware.py', line 364 in respond app_iter = self.application(environ, detect_start_response) File '/misc/PWBCAdminStore/app/__galaxy/galaxy_dist/dev/eggs/__Paste-1.6-py2.6.egg/paste/__debug/prints.py', line 98 in __call__ environ, self.app) File '/misc/PWBCAdminStore/app/__galaxy/galaxy_dist/dev/eggs/__Paste-1.6-py2.6.egg/paste/__wsgilib.py', line 539 in intercept_output app_iter = application(environ, replacement_start_response) File '/misc/PWBCAdminStore/app/__galaxy/galaxy_dist/dev/eggs/__Paste-1.6-py2.6.egg/paste/__recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/misc/PWBCAdminStore/app/__galaxy/galaxy_dist/dev/eggs/__Paste-1.6-py2.6.egg/paste/__httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/misc/PWBCAdminStore/app/__galaxy/galaxy_dist/dev/lib/__galaxy/web/framework/base.py', line 160 in __call__ body = method( trans, **kwargs ) File '/misc/PWBCAdminStore/app/__galaxy/galaxy_dist/dev/lib/__galaxy/web/framework/__init__.__py', line 184 in decorator return func( self, trans, *args, **kwargs ) File '/misc/PWBCAdminStore/app/__galaxy/galaxy_dist/dev/lib/__galaxy/web/controllers/data___admin.py', line 199 in download_build jobid = trans.app.job_manager.__deferred_job_queue.plugins['__GenomeTransferPlugin'].create___job( trans, url, dbkey, longname, indexers ) AttributeError: 'JobManager' object has no attribute 'deferred_job_queue' Anyone can give some hints about what was going on? Regards, Derrick On Fri, Sep 7, 2012 at 4:26 PM, Derrick Lin klin...@gmail.com mailto:klin...@gmail.com mailto:klin...@gmail.com mailto:klin...@gmail.com wrote: Hi guys, I enabled the tool with enable_beta_job_managers = True, server runs fine. But after I clicked on Manage local data, it gave the following error: URL: http://pwbc.garvan.unsw.edu.__au/galaxy_dev/data_admin/__manage_data http://pwbc.garvan.unsw.edu.au/galaxy_dev/data_admin/manage_data [^ http://pwbc.garvan.unsw.edu.__au/galaxy_dev/data_admin/__manage_data http://pwbc.garvan.unsw.edu.au/galaxy_dev/data_admin/manage_data] File '/misc/PWBCAdminStore/app/__galaxy/galaxy_dist/dev/eggs/__WebError-0.8a-py2.6.egg/__weberror/evalexception/__middleware.py', line 364 in respond app_iter = self.application(environ, detect_start_response)
[galaxy-dev] Galaxy error running NCBI BLAST: blastx
Hello, Can any one please tell me what could be causing this error message when I try to submit a blast job to Galaxy? An error occurred running this job:BLAST Database error: No alias or index file found for protein database [/9720/genome_references/ncbi/nr-protein-db/nr-newstyle/nr] in search path [/9720/galaxyprod/galaxy-dist/database/job_working_directory/1454::] Return error code 2 from command:blast:blastx. Thank you. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Galaxy error running NCBI BLAST: blastx
On Tue, Sep 11, 2012 at 3:11 PM, kauerb...@comcast.net wrote: Hello, Can any one please tell me what could be causing this error message when I try to submit a blast job to Galaxy? An error occurred running this job:BLAST Database error: No alias or index file found for protein database [/9720/genome_references/ncbi/nr-protein-db/nr-newstyle/nr] in search path [/9720/galaxyprod/galaxy-dist/database/job_working_directory/1454::] Return error code 2 from command:blast:blastx. That means that BLAST+ didn't find the NR database files. What does this give at the command line? ls /9720/genome_references/ncbi/nr-protein-db/nr-newstyle/nr* You should have a lot of files including nr.pal, nr.00.phd, ..,. nr.07.psq Perhaps you've got the wrong BLAST database path setup in the blastdb_p.loc file? Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] hg19 reference genome
Hi, We have installed our own Galaxy server and started NGS analysis but during the mapping procedure using either BWA for illumina or BFAST, no reference genome index is available. We have uploaded the hg19.fa file and put it in the Galaxy-dist/database folder. We modified the BWA_index.loc file (the same for BFAST file) indicating the path to the file. We restart the Galaxy. However, still no reference is available through the NGS: mapping modules?. Is there any way to solve this probably simple problem. Kind regards / Sandrine Abdelhadi -- Sandrine Imbeaud INSERM, UMR U-674, IUH Université Paris Descartes Génomique Fonctionnelle des tumeurs solides 27 rue Juliette Dodu F75010 Paris, France TEL: +33 (0)1 53 72 51 98 FAX: +33 (0)1 53 72 51 92 MOBILE: +33 (0)6 12 69 80 29 http://www.inserm-u674.net/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Galaxy error running NCBI BLAST: blastx
Please remember to CC the mailing list. On Tue, Sep 11, 2012 at 3:37 PM, kauerb...@comcast.net wrote: Hi Peter, This is what I get: [auerbach@babar ~]$ ls /9720/genome_references/ncbi/nr-protein-db/nr-newstyle/nr* /9720/genome_references/ncbi/nr-protein-db/nr-newstyle/nr /9720/genome_references/ncbi/nr-protein-db/nr-newstyle/nr.gz Normally you would download and decompress all of the NR database from the NCBI FTP site. It is currently in seven parts (seven tar balls), available here: ftp://ftp.ncbi.nlm.nih.gov/blast/db/ Where did your nr.gz file come from? That isn't normal... but perhaps it was copied from another server - what is inside it? Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] markupsafe module conflicts
Galaxy imported a library which already existed in the python system folder. See the error message below. /home/x/work/galaxy-dist/lib/galaxy/__init__.py:79: UserWarning: Module markupsafe was already imported from /usr/lib64/python2.6/site-packages/markupsafe/__init__.pyc, but /home/x/work/galaxy-dist/eggs/MarkupSafe-0.12-py2.6-linux-x86_64-ucs4.egg is being added to sys.path self.check_version_conflict() The system package is python-markupsafe-0.9.2-4.el6.x86_64 Galaxy has a newer version, it is MarkupSafe-0.12-py2.6-linux-x86_64-ucs4.egg Any idea how to resolve this problem? Thanks, Fenglou ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Server Error in history framer when debug = False
On Aug 3, 2012, at 10:42 AM, David Roquis wrote: Dear Galaxy community, I have noticed on forums that some users are experiencing this problem too, but no clues on how to fix it. When I set debug = false in universe.wsgi.ini, I get a server error in the history frame (history works perfectly when debug = true). Trying to create a new history, or to upload a saved one, doesn't work either. I use postgresql and nginx, but I this problem also appears on a freshly cloned instance of galaxy-dist. I tried on two machines with Fedora 13 + python 2.6.4 and Fedora 17 + python 2.7.3. As my goal is to use Galaxy in a production environment (with 2 webservers, 1 manager and 2 handlers), I would really like to be able to turn off debug. Here is what I can find in my paster.log when I turn off debug. Thanks a lot if anyone has a solution! David Starting server in PID 4664. serving on 0.0.0.0:8081 view at http://127.0.0.1:8081 127.0.0.1 - - [03/Aug/2012:16:26:49 +0200] GET / HTTP/1.0 200 - - Mozilla/5.0 (X11; Linux x86_64; rv:14.0) Gecko/20100101 Firefox/14.0.1 127.0.0.1 - - [03/Aug/2012:16:26:49 +0200] GET /root/tool_menu HTTP/1.0 200 - http://127.0.0.1:8080/; Mozilla/5.0 (X11; Linux x86_64; rv:14.0) Gecko/20100101 Firefox/14.0.1 127.0.0.1 - - [03/Aug/2012:16:26:49 +0200] GET /history HTTP/1.0 200 - http://127.0.0.1:8080/; Mozilla/5.0 (X11; Linux x86_64; rv:14.0) Gecko/20100101 Firefox/14.0.1 127.0.0.1 - - [03/Aug/2012:16:26:49 +0200] GET /history HTTP/1.0 500 - http://127.0.0.1:8080/; Mozilla/5.0 (X11; Linux x86_64; rv:14.0) Gecko/20100101 Firefox/14.0.1 Error - type 'exceptions.TypeError': 'NoneType' object is not callable URL: http://127.0.0.1/history File '/usr/lib/python2.7/site-packages/paste/exceptions/errormiddleware.py', line 144 in __call__ app_iter = self.application(environ, sr_checker) File '/usr/lib/python2.7/site-packages/paste/recursive.py', line 84 in __call__ return self.application(environ, start_response) File '/usr/lib/python2.7/site-packages/paste/httpexceptions.py', line 633 in __call__ return self.application(environ, start_response) File '/home/galaxy2ei/galaxy-dist/lib/galaxy/web/framework/base.py', line 169 in __call__ return body( environ, start_response ) File '/home/galaxy2ei/galaxy-dist/lib/galaxy/web/framework/__init__.py', line 772 in render template.render_context( context ) File '/home/galaxy2ei/galaxy-dist/eggs/Mako-0.4.1-py2.7.egg/mako/template.py', line 319 in render_context **kwargs) File '/home/galaxy2ei/galaxy-dist/eggs/Mako-0.4.1-py2.7.egg/mako/runtime.py', line 692 in _render_context _exec_template(inherit, lclcontext, args=args, kwargs=kwargs) File '/home/galaxy2ei/galaxy-dist/eggs/Mako-0.4.1-py2.7.egg/mako/runtime.py', line 718 in _exec_template callable_(context, *args, **kwargs) File '/home/galaxy2ei/galaxy-dist/database/compiled_templates/root/history.mako.py', line 62 in render_body __M_writer(u'\n\n') File '/home/galaxy2ei/galaxy-dist/lib/galaxy/web/framework/__init__.py', line 769 in write response_write( d.encode( 'utf-8' ) ) TypeError: 'NoneType' object is not callable Hi David, Sorry it took so long to get back to you. The problem appears to be that Galaxy is using a version of Paste from your site-packages rather than the egg bundled with Galaxy. A quick solution to this would be to set up a virtualenv as described in the production server documentation and run Galaxy under that interpreter. --nate CGI Variables - CONTENT_LENGTH: '0' HTTP_ACCEPT: 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' HTTP_ACCEPT_ENCODING: 'gzip, deflate' HTTP_ACCEPT_LANGUAGE: 'en-us,en;q=0.5' HTTP_CONNECTION: 'close' HTTP_COOKIE: 'galaxysession=c6ca0ddb55be603a1a924b5a883eb84f93f9e2c6424cabed485f00c49dc51dd6632e1e679e865175' HTTP_HOST: '127.0.0.1' HTTP_REFERER: 'http://127.0.0.1:8080/' HTTP_USER_AGENT: 'Mozilla/5.0 (X11; Linux x86_64; rv:14.0) Gecko/20100101 Firefox/14.0.1' HTTP_X_FORWARDED_FOR: '127.0.0.1' HTTP_X_FORWARDED_HOST: '127.0.0.1' ORGINAL_HTTP_HOST: 'galaxy_app' ORGINAL_REMOTE_ADDR: '127.0.0.1' PATH_INFO: '/history' REMOTE_ADDR: '127.0.0.1' REQUEST_METHOD: 'GET' SERVER_NAME: '0.0.0.0' SERVER_PORT: '8081' SERVER_PROTOCOL: 'HTTP/1.0' Configuration - __file__: '/home/galaxy2ei/galaxy-dist/universe_wsgi.ini' admin_users: 'david.roq...@yahoo.com, galaxyt...@2ei.com' allow_library_path_paste: 'True' allow_user_dataset_purge: 'True' allow_user_deletion: 'True' allow_user_impersonation: 'True' database_connection: 'postgres://galaxy2ei:galaxy2ei@localhost:5432/galaxyprod' database_engine_option_server_side_cursors: 'True' database_engine_option_strategy: 'threadlocal' datatypes_config_file: 'datatypes_conf.xml' debug: 'False' default_job_handlers: 'handler0' enable_quotas: 'True' file_path:
[galaxy-dev] GATK v2.x
Is anyone working on adapting the GATK tools for a more recent version of GATK? The distributed version is 1.4, and GATK is up to 2.1. David Hoover Helix Systems Staff ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] GATK v2.x
David, because of the restrictive licensing terms of GATK2 the main Galaxy Team will not continue to develop any GATK tools except for those which remain under the open source license (basically, GATK 1.6). We will be very happy to have wrappers for these and other non-open-source tools contributed to the toolshed however for those who wish to use them. -- jt On Tue, Sep 11, 2012 at 10:41 AM, David Hoover hoove...@helix.nih.gov wrote: Is anyone working on adapting the GATK tools for a more recent version of GATK? The distributed version is 1.4, and GATK is up to 2.1. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] hg19 reference genome
Hi, We have installed our own Galaxy server and started using the NGS modules through it but during the mapping procedure using either BWA for illumina or BFAST, no reference genome index is available. We have uploaded the hg19.fa file and put it in the Galaxy-dist/database folder. We modified the BWA_index.loc file (the same for BFAST file) indicating the path to the file. We restart the Galaxy. However, still no reference is available through the NGS: mapping modules?. Is there any way to solve this probably simple problem. Kind regards / Sandrine Abdelhadi -- Sandrine Imbeaud INSERM, UMR U-674, IUH Université Paris Descartes Génomique Fonctionnelle des tumeurs solides 27 rue Juliette Dodu F75010 Paris, France TEL: +33 (0)1 53 72 51 98 FAX: +33 (0)1 53 72 51 92 http://www.inserm-u674.net/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Track jobs in database should be True? Re: Shell script to start Galaxy in multi-server environment
On Aug 17, 2012, at 10:54 AM, Karger, Amir wrote: Replying to myself. The reason the runner was in a sleep state is the logic in lib/galaxy/web/config.py says: if ( len( self.job_handlers ) == 1 ) and ( self.job_handlers[0] == self.server_name ) and ( self.job_manager == self.server_name ): self.track_jobs_in_database = False Yeah, this logic doesn't correctly handle the situation (below) where you have a single manager/handler but separate web processes. For our dev instance, we have a single server acting as the job manager and the job handler, and we have two web servers also running on the dev box. So Galaxy apparently decides not to track the jobs in the database. However, this means it never finds any jobs to run. When we explicitly set self.track_jobs_in_database to be true in config.py, Galaxy correctly finds and runs jobs. I guess the webapps think that Galaxy *is* tracking jobs in the database, so they put jobs in there that never get pulled out? Or should it actually work when track_jobs_in_database is false, as long as the job manager and job handler(and webapps?) are on the same server. In that case, do we know why it didn't work? I'm happy to be running track_jobs_in_database=True, because our prod server is going to have separate machines doing web vs. job handling/managing. I've made it possible to manually set track_jobs_in_database in universe_wsgi.ini in 244b4cb100d1. --nate Thanks, -Amir ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Restarting Galaxy
On Aug 25, 2012, at 10:07 PM, Kenneth R. Auerbach wrote: Hello, Can someone tell me if stopping and then restarting the Galaxy process would prevent currently-running jobs from completing after the restart? Will all the jobs remain in the queue after the restart? Also, what is the best way to restart the Galaxy server? Users are complaining that jobs are extremely slow to start running. Is there a way to speed things up? Any info or suggestions would be greatly appreciated. Hi Kenneth, If you haven't taken some of the steps like using PostgreSQL instead of SQLite, and setting 'set_metadata_externally = True', please do so, as these will make a big difference on performance. Full documentation is at: http://usegalaxy.org/production/ You can't restart the server and recover running jobs if they're running through the local runner. Jobs that are waiting to run when the server shuts down will not be lost. To recover running jobs, you'll need to set up one of the cluster job runners (which you can do, even if you're only submitting jobs to the local system). --nate Thank you. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] FreeBayes bug - update to a toolshed tool wrapper
Thanks for information Greg. Actually, it turned out to be two things
causing my issues:
1) My copy of tool-data/sam_fa_indices.loc.sample was not owned by the
galaxy user (though it was writable by that user). This caused the
exception which caused the installation to get stuck at cloning:
File
'/data/local/galaxy/galaxy-prod/lib/galaxy/web/controllers/admin_toolshed.py',
line 709 in handle_repository_contents
copy_sample_files( trans.app, sample_files,
sample_files_copied=sample_files_copied )
File '/data/local/galaxy/galaxy-prod/lib/galaxy/util/shed_util.py', line
311 in copy_sample_files
copy_sample_file( app, filename, dest_path=dest_path )
File '/data/local/galaxy/galaxy-prod/lib/galaxy/util/shed_util.py', line
299 in copy_sample_file
shutil.copy( full_source_path, full_destination_path )
File '/usr/local/python/lib/python2.6/shutil.py', line 85 in copy
copymode(src, dst)
File '/usr/local/python/lib/python2.6/shutil.py', line 59 in copymode
os.chmod(dst, mode)
OSError: [Errno 1] Operation not permitted:
'/data/local/galaxy/galaxy-prod/tool-data/sam_fa_indices.loc.sample'
2) Once that happened, the directory to clone the repository into still
existed (and wasn't empty), which caused the error below, also causing
the process to get stuck in cloning status.
File
'/data/local/galaxy/galaxy-prod/eggs/Paste-1.6-py2.6.egg/paste/exceptions/errormiddleware.py',
line 143 in __call__
app_iter = self.application(environ, start_response)
File
'/data/local/galaxy/galaxy-prod/eggs/Paste-1.6-py2.6.egg/paste/recursive.py',
line 80 in __call__
return self.application(environ, start_response)
File
'/data/local/galaxy/galaxy-prod/lib/galaxy/web/framework/middleware/remoteuser.py',
line 91 in __call__
return self.app( environ, start_response )
File
'/data/local/galaxy/galaxy-prod/eggs/Paste-1.6-py2.6.egg/paste/httpexceptions.py',
line 632 in __call__
return self.application(environ, start_response)
File '/data/local/galaxy/galaxy-prod/lib/galaxy/web/framework/base.py',
line 160 in __call__
body = method( trans, **kwargs )
File
'/data/local/galaxy/galaxy-prod/lib/galaxy/web/framework/__init__.py',
line 184 in decorator
return func( self, trans, *args, **kwargs )
File
'/data/local/galaxy/galaxy-prod/lib/galaxy/web/controllers/admin_toolshed.py',
line 846 in manage_repositories
self.install_tool_shed_repositories( trans,
repositories_for_installation, reinstalling=reinstalling, **decoded_kwd )
File
'/data/local/galaxy/galaxy-prod/lib/galaxy/web/framework/__init__.py',
line 184 in decorator
return func( self, trans, *args, **kwargs )
File
'/data/local/galaxy/galaxy-prod/lib/galaxy/web/controllers/admin_toolshed.py',
line 618 in install_tool_shed_repositories
clone_repository( repository_clone_url, os.path.abspath(
relative_install_dir ), ctx_rev )
File '/data/local/galaxy/galaxy-prod/lib/galaxy/util/shed_util.py', line
287 in clone_repository
rev=util.listify( str( ctx_rev ) ) )
File
'/data/local/galaxy/galaxy-prod/eggs/mercurial-2.1.2-py2.6-linux-x86_64-ucs2.egg/mercurial/commands.py',
line 1157 in clone
branch=opts.get('branch'))
File
'/data/local/galaxy/galaxy-prod/eggs/mercurial-2.1.2-py2.6-linux-x86_64-ucs2.egg/mercurial/hg.py',
line 270 in clone
raise util.Abort(_(destination '%s' is not empty) % dest)
Abort: destination
'/data/local/galaxy/shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/freebayes/d3bf1e86b243/freebayes'
is not empty
Once I manually cleaned those issues up things progressed normally.
Hope this helps to make the process more robust.
Lance
Greg Von Kuster wrote:
Hi Lance,
You'll need to execute the following sql manually in your Galaxy database.
update tool_shed_repository set status = 'New' where status = 'Cloning';
Next, you'll have to update your production Galaxy instance code base
to at least changeset 2531e085f262 (the human-readable changeset
revision is 7503). This revision is required to install the freebayes
tool from the main Galaxy tool shed. You can also choose to wait
until the next Galaxy release to upgrade if you do not need the
freebayes tool in your local instance immediately.
Sorry for the inconvenience on this.
Greg Von Kuster
On Aug 31, 2012, at 3:59 PM, Lance Parsons wrote:
Thanks for the explanation. That makes sense. I went ahead and
installed the new version along with dependencies in my development
environments and all went well. Then in production, the installation
seems to have failed and I'm now stuck with freebayes in the
cloning status. I tried to deactivate and uninstall it so I could
try again to monitor what went wrong and I just get an error:
Error - type 'exceptions.AttributeError': 'NoneType' object has no
attribute 'startswith'
URL:
http://galaxy.princeton.edu/admin_toolshed/browse_repositories?operation=deactivate+or+uninstallid=ac6e1b48798577db
File
[galaxy-dev] FreeBayes Installation Bug
Today I ran into a new issues with installing a specific version of Freebayes. It turns out to be an issue with how git deals with submodules, something I'm only just learning about myself. In order to ensure that FreeBayes is built with the correct versions of all of the various submodules for the desired revision, it seems that two commands are needed: git checkout 9608597d12e127c847ae03aa03440ab63992fedf git submodule update --recursive In the current version of tool_dependencies.xml, there is a git reset --hard 9608597d12e127c847ae03aa03440ab63992fedf which seems to work similarly to the checkout command above, however, it does not put the submodules back at the correct revisions. That is only now an issue since they were updated only recently. I've attached a patch to the tool_dependencies.xml file that should once again allow FreeBayes to be built and installed automatically. It would be great if you could update the Tool Shed repo. Thanks. -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University diff --git a/tool_dependencies.xml b/tool_dependencies.xml --- a/tool_dependencies.xml +++ b/tool_dependencies.xml @@ -4,7 +4,8 @@ install version=1.0 actions action type=shell_commandgit clone --recursive git://github.com/ekg/freebayes.git/action -action type=shell_commandgit reset --hard 9608597d12e127c847ae03aa03440ab63992fedf/action +action type=shell_commandgit checkout 9608597d12e127c847ae03aa03440ab63992fedf/action +action type=shell_commandgit submodule update --recursive/action action type=shell_commandmake/action action type=move_directory_files source_directorybin/source_directory ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Disk Quota Error
I'm using the CloudMap SNP mapping workflow and have come across a disk quota error. Is it possible to increase my allocated disk quota to finish my workflow? Thanks, Cole ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Automatic installation of third party dependancies
Thanks Greg. I used you're updated version and added numpy as a separate dependency. It seems to work in my development system. I haven't updated my qa or production versions yet, so I can't check there. Perhaps you could test it and let me know if there are issues, etc. Glad I can be of help. Lance Greg Von Kuster wrote: Hello Lance, See my inline comments. On Sep 4, 2012, at 3:15 PM, Lance Parsons wrote: I've put together a tool wrapper for the htseq-count script that is part of the HTSeq python package and uploaded that to the tool shed. However, I have discovered that the tool dependancies do not install properly. There are a couple of issues that I've run into. 1) The biggest issue is the inability to set both the PYTHONPATH and PATH environment variables as part of installation. If I put two separateaction type=set_environment tags, then the second overwrites the first in the env.sh file. If instead, I put twoenvironment_variable tags within the action, only one (the second) gets executed during installation. The above issues have been corrected in changeset 7621:108cda898646, which is currently available only in the Galaxy central repository. There is a new Galaxy distribution scheduled to go out today, but unfortunately this changeset will not be included. From here on, Galaxy releases are tentatively scheduled for every 2 weeks, so if you don't want to pull from the Galaxy central repository, the fix should be available in the next release 2 weeks from now. I've slightly altered your tool_dependencies.xml file (mostly because I've implemented support for a new make_directory tag. It is attached below - you should updated your tool_dependencies.xml file in your htseq_count repository with this updated version. 2) Another issue is the inability to install Numpy as a dependancy to HTSeq. I can get Numpy to install, but it won't be in the PYTHONPATH when HTSeq is installed (or executed). Were you attempting to install numpy as another package defined in your tool-dependencies.xml file? If so, the fixes in the above changeset should allow you to do that now since you can install it before htseq and set PYTHONPATH to point to numpy as part of the installation. Let me know if you need additional help or bump into further problems. The tool dependency installation components are fairly immature, so adding enhancements for new tools like yours is very valuable. Thanks for your help on this, Greg Von Kuster Any thoughts on how to address these issues would be appreciated. -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Migrate Galaxy database from MySQL to Postgres
Dear all, After playing with several different tools, I got the conversion working by using this python program: http://pypi.python.org/pypi/py-mysql2pgsql Just follow the program manual. Both the installation and the execution are quite straight forward. Hope this may help someone in future. Cheers, Leon On Tue, Sep 4, 2012 at 10:54 PM, Leon Mei hailiang@nbic.nl wrote: Dear list members, Did anyone migrate the Galaxy database from MySQL to Postgres lately? Any verified script available? Simple solution by adding a postgres flag in mysqldump (http://dev.list.galaxyproject.org/MySQL-2-Postgres-td4139570.html) didn't work for me. It throws lots of syntax errors on int(), double quote, etc. My mysql version: 14.14, distribution 5.5.25a My postgres version: 9.1 I am now on a quest following the suggested list at: http://wiki.postgresql.org/wiki/Converting_from_other_Databases_to_PostgreSQL#Scripts.2C_programs I appreciate if someone can share his or her experience on this. Thanks a lot, Leon -- Hailiang (Leon) Mei Netherlands Bioinformatics Center BioAssist NGS Taskforce - http://ngs.nbic.nl Skype: leon_meiMobile: +31 6 41709231 -- Hailiang (Leon) Mei Netherlands Bioinformatics Center BioAssist NGS Taskforce - http://ngs.nbic.nl Skype: leon_meiMobile: +31 6 41709231 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] select with =20 options renders as autocompleting text box
Hi, I notice that a param with type select renders differently depending on how many options there are. If there are 20 options, it appears as a normal select box. If there are = 20 options, it looks like a text box (but it has autocompletion and does have a dropdown menu), and Inspect Element in Chrome reveals that it in fact is an input type=text with the classes text-and-autocomplete-select ac_input. Is this deliberate? If so, can the behavior be turned off? I suppose that there is no functional impact, but it looks sort of odd and jarring to have several select boxes one of which happens to look different just because of the number of options it has. Example: param name=distance type=select help= label=Distance Method force_select=FALSE option value=None1/option /param ...renders as a normal select box. param name=distance type=select help= label=Distance Method force_select=FALSE option value=None1/option option value=None2/option option value=None3/option option value=None4/option option value=None5/option option value=None6/option option value=None7/option option value=None8/option option value=None9/option option value=None10/option option value=None11/option option value=None12/option option value=None13/option option value=None14/option option value=None15/option option value=None16/option option value=None17/option option value=None18/option option value=None19/option option value=None20/option /param ...renders as a text box with autocomplete. Thanks, Dan ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Galaxy error
Hello, Can anyone tell me why I keep getting this error when I try to do a blast search in Galaxy? I checked that the nr database file was there and had the right permissions. Does this error mean that Galaxy can't find the nr file or does it mean something else? An error occurred running this job:BLAST Database error: No alias or index file found for protein database [/9720/genome_references/ncbi/nr-protein-db/nr-newstyle/nr] in search path [/9720/galaxyprod/galaxy-dist/database/job_working_directory/1485::] Return error code 2 from command: blast - Thank you. Ken. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Manage local data admin tool failed with dbkey KeyError
Hi Dave, Thanks for the clarification. Our testing shows that the local data manager works in the vanilla Galaxy (simply hg clone, then run.sh). It failed on our clustered instance that it has been configured to use multiple web threads, one job manager and two job handlers. Local data manager is a great idea, it would be even better if it can fetch the indexes from the local file system. Cheers, Derrick On Tue, Sep 11, 2012 at 10:31 PM, Dave Bouvier d...@bx.psu.edu wrote: Derrick, I have not actually tested the local data manager on CloudBioLinux, but that sounds like something worth looking into. As for the rsync server mentioned in another email, the data manager checks our rsync server first, then tries generating indexes if the rsync server doesn't have them. However, it will always produce indexes, as I decided to allow for the option to regenerate indexes, if the local ones had somehow become corrupted. --Dave B. On 9/11/12 02:03:45.000, Derrick Lin wrote: Dave, I upgraded the galaxy to Sep 7 release. The KeyError error has gone as expected. But the deferred_job_queue error persists. We are running galaxy in the cluster (SGE) here. Is it possible that Manage local data need to be configured in the cluster env? Regards, Derrick On Mon, Sep 10, 2012 at 10:59 PM, Dave Bouvier d...@bx.psu.edu mailto:d...@bx.psu.edu wrote: Derrick, The deferred_job_queue instance in the job manager object is only instantiated when the enable_beta_job_managers variable is set to True. Is it possible that your previous setting of that variable was somehow reverted? The KeyError error you mentioned in your previous should be fixed in the latest -dist and -central revisions. --Dave B. On 9/9/12 21:20:44.000, Derrick Lin wrote: Hi guys, I found that the problematic loc files were created by CloudBioLinux and point to the reference indexes downloaded by CloudBioLinux script. I suspect that Manage Local Data cannot recognize the file structure created by CloudBioLinux? Anyhow, I removed all the loc files from CloudBioLinux, then Manage Local Data can run now. Soon after I tried to download the bowtie hg18 from ucsc, I was given another error caused by deferred job queue: URL: http://129.94.136.11:8080/__**galaxy_dev/data_admin/__** download_buildhttp://129.94.136.11:8080/__galaxy_dev/data_admin/__download_build http://129.94.136.11:8080/**galaxy_dev/data_admin/** download_buildhttp://129.94.136.11:8080/galaxy_dev/data_admin/download_build File '/misc/PWBCAdminStore/app/__**galaxy/galaxy_dist/dev/eggs/__** WebError-0.8a-py2.6.egg/__**weberror/evalexception/__**middleware.py', line 364 in respond app_iter = self.application(environ, detect_start_response) File '/misc/PWBCAdminStore/app/__**galaxy/galaxy_dist/dev/eggs/__** Paste-1.6-py2.6.egg/paste/__**debug/prints.py', line 98 in __call__ environ, self.app) File '/misc/PWBCAdminStore/app/__**galaxy/galaxy_dist/dev/eggs/__** Paste-1.6-py2.6.egg/paste/__**wsgilib.py', line 539 in intercept_output app_iter = application(environ, replacement_start_response) File '/misc/PWBCAdminStore/app/__**galaxy/galaxy_dist/dev/eggs/__** Paste-1.6-py2.6.egg/paste/__**recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/misc/PWBCAdminStore/app/__**galaxy/galaxy_dist/dev/eggs/__** Paste-1.6-py2.6.egg/paste/__**httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/misc/PWBCAdminStore/app/__**galaxy/galaxy_dist/dev/lib/__** galaxy/web/framework/base.py', line 160 in __call__ body = method( trans, **kwargs ) File '/misc/PWBCAdminStore/app/__**galaxy/galaxy_dist/dev/lib/__** galaxy/web/framework/__init__.**__py', line 184 in decorator return func( self, trans, *args, **kwargs ) File '/misc/PWBCAdminStore/app/__**galaxy/galaxy_dist/dev/lib/__** galaxy/web/controllers/data___**admin.py', line 199 in download_build jobid = trans.app.job_manager.__**deferred_job_queue.plugins['__** GenomeTransferPlugin'].create_**__job( trans, url, dbkey, longname, indexers ) AttributeError: 'JobManager' object has no attribute 'deferred_job_queue' Anyone can give some hints about what was going on? Regards, Derrick On Fri, Sep 7, 2012 at 4:26 PM, Derrick Lin klin...@gmail.com mailto:klin...@gmail.com mailto:klin...@gmail.com mailto:klin...@gmail.com wrote: Hi guys, I
