Re: [galaxy-dev] [galaxy-iuc] Script to help maintain toolshed repos across toolsheds
On Thu, May 2, 2013 at 7:16 AM, Ira Cooke iraco...@gmail.com wrote: Hi all, I've written a script to help deal with the problem of maintaining toolshed tools across multiple toolsheds (eg test and release) The problem I encountered was that switching between test and production versions of a suite of tools can be quite painful because every repository definition like this repository toolshed=http://toolshed.g2.bx.psu.edu; name=proteomics_datatypes owner=iracooke changeset_revision=463328a6967f/ needs to be updated to a different toolshed url and (by extension) a different changeset revision. The idea with this script is that you should be able to point it at a directory containing a toolshed repository and it will create a copy of that repository in which the toolshed urls (and changeset revisions) have been updated to correct values for a different toolshed. I'm not sure how others are dealing with this issue (perhaps there is another easier way) .. but I've found this helped me alot so I thought I'd share https://bitbucket.org/iracooke/galaxy_repo_bundler/ Cheers Ira Thanks Ira, I've not made as heavy use of inter-repository dependencies as you, but thus far I have ignored the problem (only a couple of my repositories are affected), in the hope this limitation will be fixed sooner rather than later. Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] toolshed down?
hi, I'm getting the following error when trying to reach the main/test toolsheds. Are they down? This webpage is not available Chromium's connection attempt to toolshed.g2.bx.psu.edu was rejected. The website may be down, or your network may not be properly configured. also from within galaxy, I can't reach it, nor can the galaxy-process itself: galaxy.model.migrate.check INFO 2013-05-02 12:12:45,219 At database version 114 tool_shed.galaxy_install.migrate.check DEBUG 2013-05-02 12:12:45,390 pysqlite=2 egg successfully loaded for sqlite dialect The URL http://toolshed.g2.bx.psu.edu/repository/get_tool_dependencies?name=bowtie_wrappersowner=devteamchangeset_revision=0c7e4eadfb3cfrom_install_manager=True raised the exception: urlopen error [Errno 111] Connection refused The URL http://toolshed.g2.bx.psu.edu/repository/get_tool_dependencies?name=bowtie_color_wrappersowner=devteamchangeset_revision=fd0914e451c5from_install_manager=True raised the exception: urlopen error [Errno 111] Connection refused The URL http://toolshed.g2.bx.psu.edu/repository/get_tool_dependencies?name=lastzowner=devteamchangeset_revision=0801f8207d30from_install_manager=True raised the exception: urlopen error [Errno 111] Connection refused The URL http://toolshed.g2.bx.psu.edu/repository/get_tool_dependencies?name=lastz_paired_readsowner=devteamchangeset_revision=96825cee5c25from_install_manager=True raised the exception: urlopen error [Errno 111] Connection refused Traceback (most recent call last): File /galaxy/galaxy-beta/galaxy-dist/lib/galaxy/webapps/galaxy/buildapp.py, line 37, in app_factory app = UniverseApplication( global_conf = global_conf, **kwargs ) File /galaxy/galaxy-beta/galaxy-dist/lib/galaxy/app.py, line 54, in __init__ verify_tools( self, db_url, kwargs.get( 'global_conf', {} ).get( '__file__', None ), self.config.database_engine_options ) File /galaxy/galaxy-beta/galaxy-dist/lib/tool_shed/galaxy_install/migrate/check.py, line 56, in verify_tools tool_shed_accessible, missing_tool_configs_dict = common_util.check_for_missing_tools( app, tool_panel_configs, latest_tool_migration_script_number ) File /galaxy/galaxy-beta/galaxy-dist/lib/tool_shed/util/common_util.py, line 66, in check_for_missing_tools return tool_shed_accessible, missing_tool_configs_dict UnboundLocalError: local variable 'missing_tool_configs_dict' referenced before assignment -- Geert Vandeweyer, Ph.D. Department of Medical Genetics University of Antwerp Prins Boudewijnlaan 43 2650 Edegem Belgium Tel: +32 (0)3 275 97 56 E-mail: geert.vandewe...@ua.ac.be http://ua.ac.be/cognitivegenetics http://www.linkedin.com/pub/geert-vandeweyer/26/457/726 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] toolshed down?
Hello Geert, Thanks for reporting this - it looks like our server or network is down. We'll get this resolved as soon as possible and let you know when it is available. Greg Von Kuster On May 2, 2013, at 6:16 AM, Geert Vandeweyer wrote: hi, I'm getting the following error when trying to reach the main/test toolsheds. Are they down? This webpage is not available Chromium's connection attempt to toolshed.g2.bx.psu.edu was rejected. The website may be down, or your network may not be properly configured. also from within galaxy, I can't reach it, nor can the galaxy-process itself: galaxy.model.migrate.check INFO 2013-05-02 12:12:45,219 At database version 114 tool_shed.galaxy_install.migrate.check DEBUG 2013-05-02 12:12:45,390 pysqlite=2 egg successfully loaded for sqlite dialect The URL http://toolshed.g2.bx.psu.edu/repository/get_tool_dependencies?name=bowtie_wrappersowner=devteamchangeset_revision=0c7e4eadfb3cfrom_install_manager=True raised the exception: urlopen error [Errno 111] Connection refused The URL http://toolshed.g2.bx.psu.edu/repository/get_tool_dependencies?name=bowtie_color_wrappersowner=devteamchangeset_revision=fd0914e451c5from_install_manager=True raised the exception: urlopen error [Errno 111] Connection refused The URL http://toolshed.g2.bx.psu.edu/repository/get_tool_dependencies?name=lastzowner=devteamchangeset_revision=0801f8207d30from_install_manager=True raised the exception: urlopen error [Errno 111] Connection refused The URL http://toolshed.g2.bx.psu.edu/repository/get_tool_dependencies?name=lastz_paired_readsowner=devteamchangeset_revision=96825cee5c25from_install_manager=True raised the exception: urlopen error [Errno 111] Connection refused Traceback (most recent call last): File /galaxy/galaxy-beta/galaxy-dist/lib/galaxy/webapps/galaxy/buildapp.py, line 37, in app_factory app = UniverseApplication( global_conf = global_conf, **kwargs ) File /galaxy/galaxy-beta/galaxy-dist/lib/galaxy/app.py, line 54, in __init__ verify_tools( self, db_url, kwargs.get( 'global_conf', {} ).get( '__file__', None ), self.config.database_engine_options ) File /galaxy/galaxy-beta/galaxy-dist/lib/tool_shed/galaxy_install/migrate/check.py, line 56, in verify_tools tool_shed_accessible, missing_tool_configs_dict = common_util.check_for_missing_tools( app, tool_panel_configs, latest_tool_migration_script_number ) File /galaxy/galaxy-beta/galaxy-dist/lib/tool_shed/util/common_util.py, line 66, in check_for_missing_tools return tool_shed_accessible, missing_tool_configs_dict UnboundLocalError: local variable 'missing_tool_configs_dict' referenced before assignment -- Geert Vandeweyer, Ph.D. Department of Medical Genetics University of Antwerp Prins Boudewijnlaan 43 2650 Edegem Belgium Tel: +32 (0)3 275 97 56 E-mail: geert.vandewe...@ua.ac.be http://ua.ac.be/cognitivegenetics http://www.linkedin.com/pub/geert-vandeweyer/26/457/726 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Inconsistent menus in Galaxy Tool Shed
On Thu, May 2, 2013 at 1:35 AM, Greg Von Kuster g...@bx.psu.edu wrote: Hi Peter, Standardized Repository Actions menu in the tool shed is available in changeset revision 9555:fadc4b334f3e which is currently running on the test tool shed. Thanks for your request! Greg Von Kuster Great - it looks like the refactoring reduced the overall code and template size too. https://bitbucket.org/galaxy/galaxy-central/commits/fadc4b334f3e Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Script to help maintain toolshed repos across toolsheds
Hi all, I've written a script to help deal with the problem of maintaining toolshed tools across multiple toolsheds (eg test and release) The problem I encountered was that switching between test and production versions of a suite of tools can be quite painful because every repository definition like this repository toolshed=http://toolshed.g2.bx.psu.edu; name=proteomics_datatypes owner=iracooke changeset_revision=463328a6967f/ needs to be updated to a different toolshed url and (by extension) a different changeset revision. The idea with this script is that you should be able to point it at a directory containing a toolshed repository and it will create a copy of that repository in which the toolshed urls (and changeset revisions) have been updated to correct values for a different toolshed. I'm not sure how others are dealing with this issue (perhaps there is another easier way) .. but I've found this helped me alot so I thought I'd share https://bitbucket.org/iracooke/galaxy_repo_bundler/ Cheers Ira ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] user password different type encoding
Hello dev-team, I would like to add the different type of password encryption to the users in my galaxy instance. I started working with the current password encoding script: /home/apps/galaxy-dist/lib/galaxy/util/hash_util.py I will keep the current sha1 and add another layer of encryption to the sha1 hash, otherwise I need to force all my users to change the password and follow the new hashing method. Can anyone please point me any other place/script which I missed regarding the encryption/decryption of user authentication. thanks in advance, --/Vipin ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] user password different type encoding
That should be the only place, it is called from the some methods of the User model object. So you could modify it to always hash new passwords in a different way, but check old passwords with sha1 first, then something else. Although it might be nice to move the functionality into security.validate_user_input since it is really specific to user passwords, especially with those changes. I'd be happy to see this go into main with sha256 or something similar. Also, we could consider adding a random per-user salt field if you are really concerned about this. -- James Taylor, Assistant Professor, Biology/CS, Emory University On Thu, May 2, 2013 at 10:21 AM, Vipin TS vipin...@gmail.com wrote: Hello dev-team, I would like to add the different type of password encryption to the users in my galaxy instance. I started working with the current password encoding script: /home/apps/galaxy-dist/lib/galaxy/util/hash_util.py I will keep the current sha1 and add another layer of encryption to the sha1 hash, otherwise I need to force all my users to change the password and follow the new hashing method. Can anyone please point me any other place/script which I missed regarding the encryption/decryption of user authentication. thanks in advance, --/Vipin ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] toolshed down?
Both tool shed's should now be accessible. Sorry for the downtime. On May 2, 2013, at 6:26 AM, Greg Von Kuster wrote: Hello Geert, Thanks for reporting this - it looks like our server or network is down. We'll get this resolved as soon as possible and let you know when it is available. Greg Von Kuster On May 2, 2013, at 6:16 AM, Geert Vandeweyer wrote: hi, I'm getting the following error when trying to reach the main/test toolsheds. Are they down? This webpage is not available Chromium's connection attempt to toolshed.g2.bx.psu.edu was rejected. The website may be down, or your network may not be properly configured. also from within galaxy, I can't reach it, nor can the galaxy-process itself: galaxy.model.migrate.check INFO 2013-05-02 12:12:45,219 At database version 114 tool_shed.galaxy_install.migrate.check DEBUG 2013-05-02 12:12:45,390 pysqlite=2 egg successfully loaded for sqlite dialect The URL http://toolshed.g2.bx.psu.edu/repository/get_tool_dependencies?name=bowtie_wrappersowner=devteamchangeset_revision=0c7e4eadfb3cfrom_install_manager=True raised the exception: urlopen error [Errno 111] Connection refused The URL http://toolshed.g2.bx.psu.edu/repository/get_tool_dependencies?name=bowtie_color_wrappersowner=devteamchangeset_revision=fd0914e451c5from_install_manager=True raised the exception: urlopen error [Errno 111] Connection refused The URL http://toolshed.g2.bx.psu.edu/repository/get_tool_dependencies?name=lastzowner=devteamchangeset_revision=0801f8207d30from_install_manager=True raised the exception: urlopen error [Errno 111] Connection refused The URL http://toolshed.g2.bx.psu.edu/repository/get_tool_dependencies?name=lastz_paired_readsowner=devteamchangeset_revision=96825cee5c25from_install_manager=True raised the exception: urlopen error [Errno 111] Connection refused Traceback (most recent call last): File /galaxy/galaxy-beta/galaxy-dist/lib/galaxy/webapps/galaxy/buildapp.py, line 37, in app_factory app = UniverseApplication( global_conf = global_conf, **kwargs ) File /galaxy/galaxy-beta/galaxy-dist/lib/galaxy/app.py, line 54, in __init__ verify_tools( self, db_url, kwargs.get( 'global_conf', {} ).get( '__file__', None ), self.config.database_engine_options ) File /galaxy/galaxy-beta/galaxy-dist/lib/tool_shed/galaxy_install/migrate/check.py, line 56, in verify_tools tool_shed_accessible, missing_tool_configs_dict = common_util.check_for_missing_tools( app, tool_panel_configs, latest_tool_migration_script_number ) File /galaxy/galaxy-beta/galaxy-dist/lib/tool_shed/util/common_util.py, line 66, in check_for_missing_tools return tool_shed_accessible, missing_tool_configs_dict UnboundLocalError: local variable 'missing_tool_configs_dict' referenced before assignment -- Geert Vandeweyer, Ph.D. Department of Medical Genetics University of Antwerp Prins Boudewijnlaan 43 2650 Edegem Belgium Tel: +32 (0)3 275 97 56 E-mail: geert.vandewe...@ua.ac.be http://ua.ac.be/cognitivegenetics http://www.linkedin.com/pub/geert-vandeweyer/26/457/726 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Tool Shed Request - sanity check uploads?
Hi all, For what is I think the second time (fortunately this was only on the Test Tool Shed), I have managed to upload a tar-ball to the wrong repository: 2:fae4084a0bc0 http://testtoolshed.g2.bx.psu.edu/view/peterjc/fastq_paired_unpaired It could have been a glitch, but more likely is was simply inattention on my part (most likely - multiple tab browsing has its downsides). This leads me to offer some suggestions for how to prevent this kind of user error in future: (1) After the upload, show a preview of the changes for confirmation (the drawback is this could take a long time to process for a large upload, and lead to time outs). (2) After upload, before committing the changes, compare the tool contents - adding or removing some tools is normal, replacing ALL the tools with different tools is probably an error and should be subject to user confirmation. (3) Offer a repository action to revert to a prior commit (current workaround is to re-upload the old good file, which I did as 3:81ac324efa49 in this case). Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Tests for Workflows on the tool shed?
Hello all, I'm looking at how best to incorporate Galaxy workflows into a publication - a standalone *.ga file in supplementary materials would probably work but is static and I fear likely to bit rot. I think posting each workflow (or set of workflows) as a repository on the main Galaxy Tool Shed would be better, as I can continue to update it as needed, and it can be linking to it via the citable URL, e.g. Greg's example on the Test Tool Shed which declares some dependencies: http://testtoolshed.g2.bx.psu.edu/view/greg/heteroplasmy_workflow (This assumes the Tool Shed URLs will be stable, but I'm having to assume that anyway for tool wrappers.) However, my natural next question is how do we define unit tests for these workflows? In the case of a methods paper, some sample data for testing would be reasonable. In the case of a more applied paper, the workflow could perhaps include the actual data used in the paper and reproduce the published results (ambitious but quite possible for moderate sized datasets). I'm guessing this would be a major enhancement - something for Greg and Dave to tackle once the current round of work for testing tools on the Tool Shed is done (grin). Shall I open a Trello issue for this? Regards, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Load balancing and job configuration
Dear galaxy-dev-list, I am trying to use new-style job configuration whith a load-balancing on 2 web servers and 2 job hanlders. I have configured load balancing as follows (universe_wsgi.ini): # Configuration of the internal HTTP server. [server:web0] use = egg:Paste#http port = 8083 host = 127.0.0.1 use_threadpool = true threadpool_workers = 7 [server:web1] use = egg:Paste#http port = 8082 host = 127.0.0.1 use_threadpool = true threadpool_workers = 7 [server:manager] use = egg:Paste#http port = 8079 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 [server:handler0] use = egg:Paste#http port = 8090 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 [server:handler1] use = egg:Paste#http port = 8091 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 This configuration works fine (manager.log): galaxy.jobs.manager DEBUG 2013-04-23 12:20:22,901 Starting job handler galaxy.jobs.runners DEBUG 2013-04-23 12:20:22,902 Starting 5 LocalRunner workers galaxy.jobs DEBUG 2013-04-23 12:20:22,904 Loaded job runner 'galaxy.jobs.runners.local:LocalJobRunner' as 'local' galaxy.jobs.runners DEBUG 2013-04-23 12:20:22,909 Starting 3 LWRRunner workers galaxy.jobs DEBUG 2013-04-23 12:20:22,911 Loaded job runner 'galaxy.jobs.runners.lwr:LwrJobRunner' as 'lwr' galaxy.jobs DEBUG 2013-04-23 12:20:22,911 Legacy destination with id 'local:///', url 'local:///' converted, got params: galaxy.jobs.handler DEBUG 2013-04-23 12:20:22,911 Loaded job runners plugins: lwr:local galaxy.jobs.handler INFO 2013-04-23 12:20:22,912 job handler stop queue started galaxy.jobs.handler INFO 2013-04-23 12:20:22,919 job handler queue started To use new-style job configuration I have create the following job_conf.xml: ?xml version=1.0? !-- A sample job config that explicitly configures job running the way it is configured by default (if there is no explicit config). -- job_conf plugins plugin id=local type=runner load=galaxy.jobs.runners.local:LocalJobRunner/ /plugins handlers default=handlers !-- Additional job handlers - the id should match the name of a [server:id] in universe_wsgi.ini. -- handler id=server:web0 tags=handlers/ handler id=server:web1 tags=handlers/ /handlers destinations default=local destination id=local runner=local/ /destinations /job_conf When I restart the instance the manager.log outputs: galaxy.jobs DEBUG 2013-05-02 17:25:28,402 Loading job configuration from ./job_conf.xml galaxy.jobs DEBUG 2013-05-02 17:25:28,402 Read definition for handler 'server:web0' galaxy.jobs DEBUG 2013-05-02 17:25:28,403 Read definition for handler 'server:web1' galaxy.jobs DEBUG 2013-05-02 17:25:28,403 handlers default set to child with id or tag 'handlers' galaxy.jobs DEBUG 2013-05-02 17:25:28,403 destinations default set to child with id or tag 'local' galaxy.jobs DEBUG 2013-05-02 17:25:28,403 Done loading job configuration So everything seems fine but when I try to run a tool, the corresponding job is in permanent waiting status (gray color in history). I think I have missed something in the configuration process, Thanks for your help, Olivier ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed Request - sanity check uploads?
On Thu, May 2, 2013 at 4:29 PM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi all, For what is I think the second time (fortunately this was only on the Test Tool Shed), I have managed to upload a tar-ball to the wrong repository: 2:fae4084a0bc0 http://testtoolshed.g2.bx.psu.edu/view/peterjc/fastq_paired_unpaired It could have been a glitch, but more likely is was simply inattention on my part (most likely - multiple tab browsing has its downsides). This leads me to offer some suggestions for how to prevent this kind of user error in future: (1) After the upload, show a preview of the changes for confirmation (the drawback is this could take a long time to process for a large upload, and lead to time outs). (2) After upload, before committing the changes, compare the tool contents - adding or removing some tools is normal, replacing ALL the tools with different tools is probably an error and should be subject to user confirmation. (3) Offer a repository action to revert to a prior commit (current workaround is to re-upload the old good file, which I did as 3:81ac324efa49 in this case). Peter Another even simpler change (which would also make a big difference) is to actually have the name of the current repository on the Upload a single file or a tarball page. Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Load balancing and job configuration
Oliver, The new job running file is also providing me a lot of headaches. The documentation on site is correct, but some of the items are not yet implemented, for example, DRMAA external scripts still need to be in the universe_wsgi.ini, yet the site say to put them in job_conf.xml file! Lot of hair pulling and waiting response from IRC to figure this out...wasn't fun. As for your issue, you want to specify your handlers, not your webserver for your handler types. handler id=handler0 tags=handlers/ handler id=handler1 tags=handlers/ Let me know if this helps, -Adam -- Adam Brenner Computer Science, Undergraduate Student Donald Bren School of Information and Computer Sciences Research Computing Support Office of Information Technology http://www.oit.uci.edu/rcs/ University of California, Irvine www.ics.uci.edu/~aebrenne/ aebre...@uci.edu On Thu, May 2, 2013 at 9:00 AM, Olivier Inizan olivier.ini...@versailles.inra.fr wrote: Dear galaxy-dev-list, I am trying to use new-style job configuration whith a load-balancing on 2 web servers and 2 job hanlders. I have configured load balancing as follows (universe_wsgi.ini): # Configuration of the internal HTTP server. [server:web0] use = egg:Paste#http port = 8083 host = 127.0.0.1 use_threadpool = true threadpool_workers = 7 [server:web1] use = egg:Paste#http port = 8082 host = 127.0.0.1 use_threadpool = true threadpool_workers = 7 [server:manager] use = egg:Paste#http port = 8079 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 [server:handler0] use = egg:Paste#http port = 8090 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 [server:handler1] use = egg:Paste#http port = 8091 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 This configuration works fine (manager.log): galaxy.jobs.manager DEBUG 2013-04-23 12:20:22,901 Starting job handler galaxy.jobs.runners DEBUG 2013-04-23 12:20:22,902 Starting 5 LocalRunner workers galaxy.jobs DEBUG 2013-04-23 12:20:22,904 Loaded job runner 'galaxy.jobs.runners.local:LocalJobRunner' as 'local' galaxy.jobs.runners DEBUG 2013-04-23 12:20:22,909 Starting 3 LWRRunner workers galaxy.jobs DEBUG 2013-04-23 12:20:22,911 Loaded job runner 'galaxy.jobs.runners.lwr:LwrJobRunner' as 'lwr' galaxy.jobs DEBUG 2013-04-23 12:20:22,911 Legacy destination with id 'local:///', url 'local:///' converted, got params: galaxy.jobs.handler DEBUG 2013-04-23 12:20:22,911 Loaded job runners plugins: lwr:local galaxy.jobs.handler INFO 2013-04-23 12:20:22,912 job handler stop queue started galaxy.jobs.handler INFO 2013-04-23 12:20:22,919 job handler queue started To use new-style job configuration I have create the following job_conf.xml: ?xml version=1.0? !-- A sample job config that explicitly configures job running the way it is configured by default (if there is no explicit config). -- job_conf plugins plugin id=local type=runner load=galaxy.jobs.runners.local:LocalJobRunner/ /plugins handlers default=handlers !-- Additional job handlers - the id should match the name of a [server:id] in universe_wsgi.ini. -- handler id=server:web0 tags=handlers/ handler id=server:web1 tags=handlers/ /handlers destinations default=local destination id=local runner=local/ /destinations /job_conf When I restart the instance the manager.log outputs: galaxy.jobs DEBUG 2013-05-02 17:25:28,402 Loading job configuration from ./job_conf.xml galaxy.jobs DEBUG 2013-05-02 17:25:28,402 Read definition for handler 'server:web0' galaxy.jobs DEBUG 2013-05-02 17:25:28,403 Read definition for handler 'server:web1' galaxy.jobs DEBUG 2013-05-02 17:25:28,403 handlers default set to child with id or tag 'handlers' galaxy.jobs DEBUG 2013-05-02 17:25:28,403 destinations default set to child with id or tag 'local' galaxy.jobs DEBUG 2013-05-02 17:25:28,403 Done loading job configuration So everything seems fine but when I try to run a tool, the corresponding job is in permanent waiting status (gray color in history). I think I have missed something in the configuration process, Thanks for your help, Olivier ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] MPileup for cloudman instance
Hi All, I have noticed that there is no Mpileup available in either of the tool sheds. Is there a simple way to install it in my instance? Thanks, Iry The information in this email, including attachments, may be confidential and is intended solely for the addressee(s). If you believe you received this email by mistake, please notify the sender by return email as soon as possible. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] user password different type encoding
Thanks James, I have updated the password of one user in galaxy_user table with the new algorithm, I also adjusted the function new_secure_hash in /lib/galaxy/util/hash_util.py in such a way that it returns the new hash instead of sha1. Now I tried to login, it fails to get the account, I think there is something going wrong in the password hash comparison. Can you please assit here. +++ b/lib/galaxy/util/hash_util.py Thu May 02 14:33:07 2013 -0400 @@ -25,13 +25,60 @@ Returns either a sha1 hash object (if called with no arguments), or a hexdigest of the sha1 hash of the argument `text_type`. +import hashlib +from os import urandom +from base64 import b64encode, b64decode +from itertools import izip +from pbkdf2 import pbkdf2_bin + +SALT_LENGTH = 12 +KEY_LENGTH = 24 +HASH_FUNCTION = 'sha256' +COST_FACTOR = 1 + if text_type: +#return sha1( text_type ).hexdigest() + +sec_hash_1 = sha1( text_type ).hexdigest() + +if isinstance(sec_hash_1, unicode): +sec_hash_1 = sec_hash_1.encode('utf-8') +salt = b64encode(urandom(SALT_LENGTH)) + +return 'PBKDF2${0}${1}${2}${3}'.format( +HASH_FUNCTION, +COST_FACTOR, +salt, +b64encode(pbkdf2_bin(sec_hash_1, salt, COST_FACTOR, KEY_LENGTH, getattr(hashlib, HASH_FUNCTION thanks, Vipin That should be the only place, it is called from the some methods of the User model object. So you could modify it to always hash new passwords in a different way, but check old passwords with sha1 first, then something else. Although it might be nice to move the functionality into security.validate_user_input since it is really specific to user passwords, especially with those changes. I'd be happy to see this go into main with sha256 or something similar. Also, we could consider adding a random per-user salt field if you are really concerned about this. -- James Taylor, Assistant Professor, Biology/CS, Emory University On Thu, May 2, 2013 at 10:21 AM, Vipin TS vipin...@gmail.com wrote: Hello dev-team, I would like to add the different type of password encryption to the users in my galaxy instance. I started working with the current password encoding script: /home/apps/galaxy-dist/lib/galaxy/util/hash_util.py I will keep the current sha1 and add another layer of encryption to the sha1 hash, otherwise I need to force all my users to change the password and follow the new hashing method. Can anyone please point me any other place/script which I missed regarding the encryption/decryption of user authentication. thanks in advance, --/Vipin ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] galaxysession cookie secure flag
Hi dev-team, We have placed our galaxy instance ssl and I need to make sure that the secure flag is set on the cookie (commonly represented by the word “secure” under the Security column) but I am not able to do the same. something like below: [image: Inline image 2] when I checked on my instance I saw as below: [image: Inline image 3] I have made necessary changes to my ssl.conf to put the flag as secure, but it seems not appearing here. Header edit Set-Cookie ^(.*)$ $1;Secure;HttpOnly does anybody have an experience in setting up the same. thanks in advance, --/Vipin image.pngimage.png___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed Request - sanity check uploads?
Hi Peter, This fix is now running on the test tool shed. Thanks for pointing out this flaw! Greg Von Kuster On May 2, 2013, at 12:54 PM, Peter Cock wrote: On Thu, May 2, 2013 at 4:29 PM, Peter Cock p.j.a.c...@googlemail.com wrote: Another even simpler change (which would also make a big difference) is to actually have the name of the current repository on the Upload a single file or a tarball page. Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Problem accessing test tool shed repositories
Ilya, Thank you for reporting this issue, a fix has been committed in 9627:8d8368ab03ff, and the test tool shed has been updated. --Dave B. On 5/2/13 13:01:54.000, Sytchev, Ilya wrote: Hi, I can't install any tools from the test tool shed into the local Galaxy instances. I can see the list of the repositories but nothing happens when I click on any of them. I can still access repositories in the main tool shed. The same thing happens in fresh installations of both galaxy-central and galaxy-dist. Attached screenshot shows a list of Sequence Analysis repositories in the test tool shed (note the lack of buttons). I'd appreciate any help. Thanks, Ilya ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] user password different type encoding
I have started testing with creating a new user and the password hash created using new algorithm, galaxy=# select username,email,password from galaxy_user where email = ' fml...@gmail.com'; username | email | password --+--+-- | fml...@gmail.com | PBKDF2$sha256$1$e0DVCuGEua3ebxxU$Bh6 I have updated the length of password column to 80 characters in my table and still the stored password seems to be in 40 char long, I print the hash after creating the second hash (password - sha1 hash 40 char long- pbdkf2 hash 69 char long) before storing into the database table, I believing the hash has been truncated, any idea what is happening here. I am not seeing any clue in the code. thanks, Vipin Thanks James, I have updated the password of one user in galaxy_user table with the new algorithm, I also adjusted the function new_secure_hash in /lib/galaxy/util/hash_util.py in such a way that it returns the new hash instead of sha1. Now I tried to login, it fails to get the account, I think there is something going wrong in the password hash comparison. Can you please assit here. +++ b/lib/galaxy/util/hash_util.py Thu May 02 14:33:07 2013 -0400 @@ -25,13 +25,60 @@ Returns either a sha1 hash object (if called with no arguments), or a hexdigest of the sha1 hash of the argument `text_type`. +import hashlib +from os import urandom +from base64 import b64encode, b64decode +from itertools import izip +from pbkdf2 import pbkdf2_bin + +SALT_LENGTH = 12 +KEY_LENGTH = 24 +HASH_FUNCTION = 'sha256' +COST_FACTOR = 1 + if text_type: +#return sha1( text_type ).hexdigest() + +sec_hash_1 = sha1( text_type ).hexdigest() + +if isinstance(sec_hash_1, unicode): +sec_hash_1 = sec_hash_1.encode('utf-8') +salt = b64encode(urandom(SALT_LENGTH)) + +return 'PBKDF2${0}${1}${2}${3}'.format( +HASH_FUNCTION, +COST_FACTOR, +salt, +b64encode(pbkdf2_bin(sec_hash_1, salt, COST_FACTOR, KEY_LENGTH, getattr(hashlib, HASH_FUNCTION thanks, Vipin That should be the only place, it is called from the some methods of the User model object. So you could modify it to always hash new passwords in a different way, but check old passwords with sha1 first, then something else. Although it might be nice to move the functionality into security.validate_user_input since it is really specific to user passwords, especially with those changes. I'd be happy to see this go into main with sha256 or something similar. Also, we could consider adding a random per-user salt field if you are really concerned about this. -- James Taylor, Assistant Professor, Biology/CS, Emory University On Thu, May 2, 2013 at 10:21 AM, Vipin TS vipin...@gmail.com wrote: Hello dev-team, I would like to add the different type of password encryption to the users in my galaxy instance. I started working with the current password encoding script: /home/apps/galaxy-dist/lib/galaxy/util/hash_util.py I will keep the current sha1 and add another layer of encryption to the sha1 hash, otherwise I need to force all my users to change the password and follow the new hashing method. Can anyone please point me any other place/script which I missed regarding the encryption/decryption of user authentication. thanks in advance, --/Vipin ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] [galaxy-user] Amazon EC2: An error occurred running this job: Job output not returned from cluster
I am getting similar errors as Brian reported back in March. (Note, we appear to have the same last name, but no relation) An error occurred with this dataset: *Job output not returned from cluster* * * - Running on Cloudman with 5-6 nodes. (xlarge) - The error seems to occur consistently when I launch multiple workflows in batch (using bioblend) - Probably not relevant, but is failing on a BWA step. - I am able to run successfully the same workflow against one of the datasets that failed in batch. - Change-set is from Feb 8, 2013. 8794:1c7174911392. Stable branch. Prior to that, I was running different galaxy instances using changesets from last year and never ran into this problem. - I'm seeing errors like: galaxy.jobs.runners.drmaa WARNING 2013-05-02 17:07:51,991 Job output not returned from cluster: [Errno 2] No such file or directory\ : '/mnt/galaxyData/tmp/job_working_directory/002/2066/2066.drmec' - In this example, the /mnt/galaxyData/tmp/job_working_directory/002/2066 folder and /mnt/galaxyData/tmp/job_working_directory/002/2066/2066.drmec files do not exist. Any suggestions? Seems like this might be some type of resource contention issue, but I'm not sure where to investigate next. Thanks in advance, Dave On Mon, Mar 11, 2013 at 9:04 AM, Brian Lin brian@tufts.edu wrote: Hi guys, I'm running a galaxy cloudman instance and running the usual tophat-cufflinks-cuffdiff workflow from RNAseq data. I am using a m2.4xlarge as a master node, and autoscaling from 0-4 workers of the m2.xlarge type. I have gotten the error: An error occurred running this job: *Job output not returned from cluster* when running fasta groomer, tophat, and now cufflinks. Following up troubleshooting from other people in the mailing list, I have set a new line in universe_wsgi.ini of retry_job_output_collection=30 Unfortunately, this does not seem to have fixed the problem. The stdout is blank, and stderr gives Job output not returned from cluster Under manage jobs in the admin panel, it lists 4 out of the 6 jobs as currently running. What is confusing is that of the 4 running, one has already returned the error in the user dataset panel and yet is still listed as running. From the SGE log, I see these errors: 03/11/2013 14:32:52|worker|ip-10-159-47-223|W|job 42.1 failed on host ip-10-30-130-84.ec2.internal before writing exit_status because: shepherd exited with exit status 19: before writing exit_status 03/11/2013 14:32:52|worker|ip-10-159-47-223|W|job 43.1 failed on host ip-10-30-130-84.ec2.internal before writing exit_status because: shepherd exited with exit status 19: before writing exit_status 03/11/2013 14:39:41|worker|ip-10-159-47-223|E|adminhost ip-10-30-130-84.ec2.internal already exists 03/11/2013 14:40:07|worker|ip-10-159-47-223|E|exechost ip-10-30-130-84.ec2.internal already exists 03/11/2013 14:50:54|worker|ip-10-159-47-223|E|adminhost ip-10-30-130-84.ec2.internal already exists 03/11/2013 14:50:55|worker|ip-10-159-47-223|E|exechost ip-10-30-130-84.ec2.internal already exists Does anyone have any idea how to solve this error? It has removed my ability to use workflows completely and I still have not been able to run a single analysis to completion due to it. Thanks for any insight anyone provide! Brian -- Brian Lin cont...@brian-lin.com brian@tufts.edu ___ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using reply all in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Problem accessing test tool shed repositories
Thanks for getting this fixed so quickly! Ilya On 5/2/13 4:32 PM, Dave Bouvier d...@bx.psu.edu wrote: Ilya, Thank you for reporting this issue, a fix has been committed in 9627:8d8368ab03ff, and the test tool shed has been updated. --Dave B. On 5/2/13 13:01:54.000, Sytchev, Ilya wrote: Hi, I can't install any tools from the test tool shed into the local Galaxy instances. I can see the list of the repositories but nothing happens when I click on any of them. I can still access repositories in the main tool shed. The same thing happens in fresh installations of both galaxy-central and galaxy-dist. Attached screenshot shows a list of Sequence Analysis repositories in the test tool shed (note the lack of buttons). I'd appreciate any help. Thanks, Ilya ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] user password different type encoding
I have updated the table schema from the script to adjust the column length from the following script: lib/galaxy/model/mapping.py Now my new registration passwords are encrypted with second layer of authentication using PBKDF2 new entry from the database table: galaxy=# select username,email,password from galaxy_user where email = ' vi...@mail.com'; username | email | password --++--- | vi...@mail.com | PBKDF2$sha256$1$lv0RfxbU3SymKvEA$l4RH9f9xHrH4pcf9n6ELP1MWjG+hooEW BUT now I am experiencing the problem with authenticating the newly registered user name. once I log out, I can’t log back in again – password invalid. This tells me there is something going on with the password hash/compare function. Can you please guide through the right module to look for this, Will be quite helpful, --/Vipin I have started testing with creating a new user and the password hash created using new algorithm, galaxy=# select username,email,password from galaxy_user where email = ' fml...@gmail.com'; username | email | password --+--+-- | fml...@gmail.com | PBKDF2$sha256$1$e0DVCuGEua3ebxxU$Bh6 I have updated the length of password column to 80 characters in my table and still the stored password seems to be in 40 char long, I print the hash after creating the second hash (password - sha1 hash 40 char long- pbdkf2 hash 69 char long) before storing into the database table, I believing the hash has been truncated, any idea what is happening here. I am not seeing any clue in the code. thanks, Vipin Thanks James, I have updated the password of one user in galaxy_user table with the new algorithm, I also adjusted the function new_secure_hash in /lib/galaxy/util/hash_util.py in such a way that it returns the new hash instead of sha1. Now I tried to login, it fails to get the account, I think there is something going wrong in the password hash comparison. Can you please assit here. +++ b/lib/galaxy/util/hash_util.py Thu May 02 14:33:07 2013 -0400 @@ -25,13 +25,60 @@ Returns either a sha1 hash object (if called with no arguments), or a hexdigest of the sha1 hash of the argument `text_type`. +import hashlib +from os import urandom +from base64 import b64encode, b64decode +from itertools import izip +from pbkdf2 import pbkdf2_bin + +SALT_LENGTH = 12 +KEY_LENGTH = 24 +HASH_FUNCTION = 'sha256' +COST_FACTOR = 1 + if text_type: +#return sha1( text_type ).hexdigest() + +sec_hash_1 = sha1( text_type ).hexdigest() + +if isinstance(sec_hash_1, unicode): +sec_hash_1 = sec_hash_1.encode('utf-8') +salt = b64encode(urandom(SALT_LENGTH)) + +return 'PBKDF2${0}${1}${2}${3}'.format( +HASH_FUNCTION, +COST_FACTOR, +salt, +b64encode(pbkdf2_bin(sec_hash_1, salt, COST_FACTOR, KEY_LENGTH, getattr(hashlib, HASH_FUNCTION thanks, Vipin That should be the only place, it is called from the some methods of the User model object. So you could modify it to always hash new passwords in a different way, but check old passwords with sha1 first, then something else. Although it might be nice to move the functionality into security.validate_user_input since it is really specific to user passwords, especially with those changes. I'd be happy to see this go into main with sha256 or something similar. Also, we could consider adding a random per-user salt field if you are really concerned about this. -- James Taylor, Assistant Professor, Biology/CS, Emory University On Thu, May 2, 2013 at 10:21 AM, Vipin TS vipin...@gmail.com wrote: Hello dev-team, I would like to add the different type of password encryption to the users in my galaxy instance. I started working with the current password encoding script: /home/apps/galaxy-dist/lib/galaxy/util/hash_util.py I will keep the current sha1 and add another layer of encryption to the sha1 hash, otherwise I need to force all my users to change the password and follow the new hashing method. Can anyone please point me any other place/script which I missed regarding the encryption/decryption of user authentication. thanks in advance, --/Vipin ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all
Re: [galaxy-dev] [Biopython] GFF parsing with biopython
Thank you. On Wed, May 1, 2013 at 8:04 PM, Brad Chapman chapm...@50mail.com wrote: Mic; (moving to galaxy-dev list so folks there can follow, but future questions are more appropriate for the Biopython list only since this isn't a Galaxy question) I have the following GFF file from a SNAP X1 SNAPEinit 25792712-3.221 + . X1-snap.1 [...] With the code below I have tried to parse the above GFF file The attributes you're missing are parts of the feature, not the SeqRecord itself, which is why you're seeing attribute error. Here's a full example that pulls all of the information from an example line: from BCBio import GFF in_file = snap.gff with open(in_file) as in_handle: for rec in GFF.parse(in_handle): feature = rec.features[0] print rec.id print feature.qualifiers[source][0] print feature.type print feature.location.start print feature.location.end print feature.qualifiers[score][0] print feature.location.strand print feature.qualifiers.get(X1-snap.1, [None])[0] which outputs: X1 SNAP Einit 2578 2712 -3.221 1 true Hope this helps, Brad ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/