[galaxy-dev] Galaxy Tool Shed packages for Biopython
Hi all, I've send this to both the Galaxy and Biopython developers lists, and hope this will make sense to both groups. If you've not heard of Galaxy, start here: http://galaxyproject.org - while the easy to guess Biopython website is at http://biopython.org Brad Chapman and I are both Biopython core developers, and are also both on the IUC Galaxy Tool Shed committee because we've been quite involved in wrapping and writing tools for use on Galaxy. Fellow committee member Björn Grüning has done a lot of the hands on work defining package definitions for dependencies within the Galaxy Tool Shed ecosystem - including defining them for Biopython, NumPy, SciPy, MatPlotLib, etc. We're very grateful for his hard work - most of which is now available under the IUC group account: http://toolshed.g2.bx.psu.edu/view/iuc/ http://testtoolshed.g2.bx.psu.edu/view/iuc/ The Biopython packages, however, are under a dedicated biopython account on the Galaxy Tool Shed to which currently Bjoern, Brad and I have access to: http://toolshed.g2.bx.psu.edu/view/biopython/ http://testtoolshed.g2.bx.psu.edu/view/biopython/ This packaging work was initially tracked in Bjoern's own GitHub repository, https://github.com/bgruening/galaxytools/ We (me, Brad and Bjoern) agreed that a Biopython owned repository would be more sensible in the long term, so I have created this and ported Bjoern's commits to it: https://github.com/biopython/galaxy_packages Currently the Galaxy packagers team on GitHub which has read and write access to this new repository is just me, Brad and Bjoern. Regards, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Viewing old nightly test results on the (Test) Tool Shed
On Thu, Sep 12, 2013 at 2:21 PM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi all, I was wondering if the web interface for the Tool Shed could allow us to look back at old test results? This would be useful in connection with explicit revisions to the tool concerned, but also when something changed which was external to the tool (e.g. a change in Galaxy itself, or the system running the tests, e.g. a library version change). Right now I'm wondering what happened to this tool, which from memory was working, but which failed last night following me updating it yesterday: http://testtoolshed.g2.bx.psu.edu/view/peterjc/clinod Previously this was installing correctly (and I remember debugging the tests themselves due to the number of threads available), but now I seem to have a failed installation: Tests that failed Tool id: clinod Tool version: clinod Test: test_tool_00 (functional.test_toolbox.TestForTool_testtoolshed.g2.bx.psu.edu/repos/peterjc/clinod/clinod/0.0.5) Stderr: Fatal error: Exit code 1 () Error: Unable to access jarfile /ToolDependencies/clinod/1.3/peterjc/clinod/1d09534d6dcd/clinod-1.3.jar Traceback: Traceback (most recent call last): File /var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py, line 171, in test_tool self.do_it( td, shed_tool_id=shed_tool_id ) File /var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py, line 102, in do_it self.verify_dataset_correctness( outfile, hid=elem_hid, maxseconds=testdef.maxseconds, attributes=attributes, shed_tool_id=shed_tool_id ) File /var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/base/twilltestcase.py, line 782, in verify_dataset_correctness self._assert_dataset_state( elem, 'ok' ) File /var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/base/twilltestcase.py, line 606, in _assert_dataset_state raise AssertionError( errmsg ) AssertionError: Expecting dataset state 'ok', but state is 'error'. Dataset blurb: error Whatever broke the clinod install seems to have been transient - it worked last night (identical bar a commit with a trivial update to the help text in the tool XML). http://testtoolshed.g2.bx.psu.edu/view/peterjc/clinod/b81bf955f659 However, I am still in interested in the general issue of it sometimes being desirable to see old test results. Regards, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Galaxy Tool Shed packages for Biopython
On Fri, Sep 13, 2013 at 9:54 AM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi all, ... The Biopython packages, however, are under a dedicated biopython account on the Galaxy Tool Shed to which currently Bjoern, Brad and I have access to: http://toolshed.g2.bx.psu.edu/view/biopython/ http://testtoolshed.g2.bx.psu.edu/view/biopython/ This packaging work was initially tracked in Bjoern's own GitHub repository, https://github.com/bgruening/galaxytools/ We (me, Brad and Bjoern) agreed that a Biopython owned repository would be more sensible in the long term, so I have created this and ported Bjoern's commits to it: https://github.com/biopython/galaxy_packages Reviewing Bjoern's glimmer3 package I released I was missing the requirement tag information in that README, hopefully this is correct now? https://github.com/biopython/galaxy_packages/commit/1d118ab296e9e4d432a913a2ad8d13c680c32a90 https://github.com/biopython/galaxy_packages/blob/1d118ab296e9e4d432a913a2ad8d13c680c32a90/README.rst What I am unclear on is how version information needed in the requirement is determined - must it be a perfect match to the tool_dependencies.xml it points at? i.e. matplotlib 1.2 versus 1.2.1, and scipy 0.12 versus 0.12.0 What happens when that gets updated (since the example repository_dependencies.xml implicitly points at the latest version of the tool at upload time)? Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] toolshed tool versions and tool panel
Hi Greg, When updating the version number of a tool I noticed that, as expected, both the old version and the new version of the tool become available in the Galaxy environment. What I did not expect is that the new version resulted in a new entry in the menu bar (see screenshot): [cid:image001.png@01CEB07A.3ECB90B0] If I remember correctly, you showed in Oslo that the tool version becomes a choice in the tool itself, something like the screenshot below: [cid:image002.png@01CEB07A.9929B870] Is this a setting somewhere or are we facing a bug? Best regards, Pieter. Pieter Lukasse Wageningen UR, Plant Research International Departments of Bioscience and Bioinformatics Wageningen Campus, Building 107, Droevendaalsesteeg 1, 6708 PB, Wageningen, the Netherlands +31-317480891; skype: pieter.lukasse.wur http://www.pri.wur.nlhttp://www.pri.wur.nl/ inline: image001.pnginline: image002.png___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] A proposed modules extension for toolshed wrappers
Another vote for the excellent modules system from us at AgResearch Galaxy's dependency injection works in a similar way, without requiring all the external dependencies. We don't need unload since we compose a unique environment for every tool execution. This is much cleaner that loading everything into the environment before running Galaxy, we want tools to be executed with as clean an environment as possible. Setting the environment for each tool just prior to running the tool is clearly better. What I'm doing currently is loading an environment module just before running a tool in Galaxy. So far, I have done this in the tool XML file by prefixing the command with a module load command, e.g. command. /etc/profile; module load EMBOSS/5.0.0; infoseq -sequence [snipped] /command The ugly . /etc/profile is needed for technical reasons. I don't like this, as it's a bit hacky, but what it achieves is useful and flexible. I envisage an extension to the tool XML syntax which would let me write it like this, say: requirementsrequirement type=module version=5.0.0EMBOSS/requirement/requirements commandinfoseq -sequence [snipped] /command This will enable my native EMBOSS wrapper to run the defined version of the (system-installed) EMBOSS suite. However, it would only be worth me writing this extension if there's a likelihood of getting it into Galaxy. Is there? Here's where I'm coming from. I already have most of the applications I want installed on my system. I am working on updating the RPMs to support side-by-side installs of multiple versions, with version selection by the environment modules facility. This is a small change (and improvement) on what I have today. It's a work in progress, but it's a manageable amount of work. It's looking likely that this multi-version packaging approach may be adopted by an official CentOS Scientific Repo, and this could change the landscape in terms of packaging of scientific applications for major Linux distributions. Who knows, that's all in the future. In the meantime, there's work going on in Galaxy land to package every application wanted in Galaxy, by writing install scripts, encapsulating environment variables in tool XML files, etc. (There is difficulty here in writing install scripts which work on every platform, which I have commented on previously.) The big question is, will the Galaxy team accept contributions to Galaxy to support different ways of solving this packaging problem? In particular, allowing for environment modules to be loaded as part of running a tool, and having this functionality in the toolshed? cheers, Simon === Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. === ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] problems with toolshed package update process
Hello Pieter, The version of mercurial you are using is likely older than version 2.2.3. See: http://wiki.galaxyproject.org/ToolShedRepositoryFeatures#Pushing_changes_to_a_repository_using_hg_from_the_command_line Can you check this? Greg Von Kuster On Sep 13, 2013, at 5:54 AM, Lukasse, Pieter pieter.luka...@wur.nl wrote: Hi Greg, We have been trying out the update process as follows: 1-Make changes to one of the files in the repository 2-Commit and push changes to hg After these two steps we expected to be able to update the tool in Galaxy. However this was not possible. We noticed that every time we pushed a new revision to the toolshed hg we had to use the option “Reset all repository metadata”, i.e. the new revision would be seen as an update only after doing this reset. Is this the normal behaviour? I can imagine that in some cases the admin would like to review a new committed changeset before enabling it as an update...could it be this is a configuration we forgot to change somewhere? Best regards, Pieter. image001.png Pieter Lukasse Wageningen UR, Plant Research International Departments of Bioscience and Bioinformatics Wageningen Campus, Building 107, Droevendaalsesteeg 1, 6708 PB, Wageningen, the Netherlands +31-317480891; skype: pieter.lukasse.wur http://www.pri.wur.nl ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] toolshed tool versions and tool panel
Pieter, Thank you for this report, I have reproduced this behavior in my local Galaxy instance. I did however notice that the tool version drop-down is still available on the tool's page, but multiple entries in the left-hand tool panel is not intended behavior. I've created a Trello card to track our progress on resolving that issue: https://trello.com/c/SlNI79mR --Dave B. On 09/13/2013 06:27 AM, Lukasse, Pieter wrote: Hi Greg, When updating the version number of a tool I noticed that, as expected, both the old version and the new version of the tool become available in the Galaxy environment. What I did not expect is that the new version resulted in a new entry in the menu bar (see screenshot): If I remember correctly, you showed in Oslo that the tool version becomes a choice in the tool itself, something like the screenshot below: Is this a setting somewhere or are we facing a bug? Best regards, Pieter. Pieter Lukasse Wageningen UR, Plant Research International Departments of Bioscience and Bioinformatics Wageningen Campus, Building 107, Droevendaalsesteeg 1, 6708 PB, Wageningen, the Netherlands +31-317480891; skype: pieter.lukasse.wur http://www.pri.wur.nl http://www.pri.wur.nl/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Circumvent Show all button
Hello Jeremy, Thanks a lot. That was the right parameter. :-) Best regards, Gromobir There is currently no simple way to circumvent this behavior. If you want to modify the code, take a look at display_data() in data.py -- you'll want to bump up the max_peek_size (line 351 in -central) above 10 MB for HTML datasets. Good luck, J. On Sep 9, 2013, at 6:16 AM, Gromobir wrote: Hey guys, Just one short question: Is there any way to circumvent the Show all dialog within galaxy, if the data I would like to display is too large? I would like to display some xhtml files, which can really have a lot of lines and I don't want the users of my tool to click on this button every time before accessing the page. Any hints are highly appreciated. Best regards, Gromobir ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Bug: Galaxy Tool Shed dependency communication via HTTP GET method
When installing Galaxy Tool Shed repositories into a local Galaxy instance, tool shed dependecies are exchanged as an encoded python dictionary via the HTTP GET method. If a tool shed repository contains enough dependencies, the length of this encoded dictionary can become extremely long. In my case, it became long enough for my Apache server to choke on it: [Wed Sep 11 15:34:53 2013] [error] [client XXX.XXX.XXX.XXX] request failed: URI too long (longer than 8190) The HTTP standard does not place a limit on the length of a URI, but the de facto limit is 2000 characters. See the following stackoverflow answer: http://stackoverflow.com/a/417184 I would recommend either: 1) communicating toolshed dependencies via an HTTP POST 2) redesigning tool shed - instance communications As a stopgap measure, I've modified lib/tool_shed/util/encoding_util.py to use a more compact encoding scheme. I'm attaching a copy in case someone else runs into the same problem. -rico import base64 import zlib from galaxy import eggs from galaxy.util.hash_util import hmac_new from galaxy.util.json import json_fix import pkg_resources pkg_resources.require( simplejson ) import simplejson encoding_sep = '__esep__' encoding_sep2 = '__esepii__' def tool_shed_decode( val ): mac, encoded_val = val.split( ':', 1 ) assert mac == hmac_new( 'ToolShedAndGalaxyMustHaveThisSameKey', encoded_val ) compressed_val = base64.urlsafe_b64decode( encoded_val.encode( 'utf8' ) ) json_val = zlib.decompress( compressed_val ) value = simplejson.loads( json_val ) return json_fix( value ) def tool_shed_encode( val ): json_val = simplejson.dumps( val, separators=( ',', ':' ) ) compressed_val = zlib.compress( json_val, 9 ) encoded_val = base64.urlsafe_b64encode( compressed_val ) mac = hmac_new( 'ToolShedAndGalaxyMustHaveThisSameKey', encoded_val ) return %s:%s % ( mac, encoded_val ) ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] A proposed modules extension for toolshed wrappers
Just been reading a bit more about the Galaxy packaging system. Here's a slight modification to what I was suggesting that might fit in a bit better. Apologies for not being more familiar with the existing system before proposing extensions. Recall that my goal is to support using a system-installed (native) package, at a defined version, which I aim to achieve by loading the appropriate environment module before running a tool. We still have tool_dependencies.xml defining a package at a particular version, but rather than download and build the source code, there's just a directive that says how to pick up the correct program version at runtime, e.g. which environment module to load. So instead of the tool_dependencies.xml fragment: tool_dependency package name=bwa version=0.6.2 install version=1.0 actions action type=download_by_urlhttp://downloads.sourceforge.net/project/bio-bwa/bwa-0.6.2.tar.bz2/action action type=shell_commandmake/action action type=move_file sourcebwa/source destination$INSTALL_DIR/bin/destination /action action type=set_environment environment_variable name=PATH action=prepend_to$INSTALL_DIR/bin/environment_variable /action /actions /install /package /tool_dependency We have something like this (NB: element and attribute names are for illustrative purposes only): tool_dependency package name=bwa version=0.6.2 use_native actions action type=module_loadbwa/0.6.2/action /actions /use_native /package /tool_dependency This causes the right thing (module load bwa/0.6.2) to be stuck into the dependencies env.sh file when this package is installed from the toolshed. We could call this toolshed tool native_package_bwa_0_6_2, to avoid confusion with the existing download-and-make one. We might want a bit of flexibility on what actions are supported (in case we want to support Software Collections, for example). What do you think? cheers, Simon PS: In case it wasn't already clear, solving this problem well is quite important to us here at AgResearch. ;-) === Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. === ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Geek question re HTTP POST and Galaxy
I've been knocking my head over this all afternoon and I was hoping to buy a vowel, use a lifeline, whatever. I'm trying to use the Ember JS framework (which is very nice) for some extensions. It relies heavily on a REST model using the same method with all of the HTTP forms including Action HTTP Verb URL FindGET /people/123 Find AllGET /people Update PUT /people/123 (with the HTTP payload packed with the JSON content) Create POST/people Delete DELETE /people/123 How can I make an API function that responds to all five forms and what should be in buildapp.py? All of the code in Galaxy currently uses the kwd (keyword) parameter. But there are no keywords here. Confused, Ted ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] A proposed modules extension for toolshed wrappers
I'd like to propose a slightly different approach. I don't like the idea of introducing a different type of requirement for this because it really isn't a different type of requirement. You are still requiring a package, you just want to use a different mechanism to provide it. So, instead, what about creating a plugin for tool requirement handling? You could replace the default dependency injection approach with one based on modules. So, the tool configurations would not need to change, you would just change the configuration of your instance to inject module loads instead of source env.shs. And yes, we would definitely accept something like this as long as it is a plugin that is not enabled by default. -- James Taylor, Associate Professor, Biology/CS, Emory University On Thu, Sep 12, 2013 at 7:03 PM, Guest, Simon simon.gu...@agresearch.co.nz wrote: Another vote for the excellent modules system from us at AgResearch Galaxy's dependency injection works in a similar way, without requiring all the external dependencies. We don't need unload since we compose a unique environment for every tool execution. This is much cleaner that loading everything into the environment before running Galaxy, we want tools to be executed with as clean an environment as possible. Setting the environment for each tool just prior to running the tool is clearly better. What I'm doing currently is loading an environment module just before running a tool in Galaxy. So far, I have done this in the tool XML file by prefixing the command with a module load command, e.g. command. /etc/profile; module load EMBOSS/5.0.0; infoseq -sequence [snipped] /command The ugly . /etc/profile is needed for technical reasons. I don't like this, as it's a bit hacky, but what it achieves is useful and flexible. I envisage an extension to the tool XML syntax which would let me write it like this, say: requirementsrequirement type=module version=5.0.0EMBOSS/requirement/requirements commandinfoseq -sequence [snipped] /command This will enable my native EMBOSS wrapper to run the defined version of the (system-installed) EMBOSS suite. However, it would only be worth me writing this extension if there's a likelihood of getting it into Galaxy. Is there? Here's where I'm coming from. I already have most of the applications I want installed on my system. I am working on updating the RPMs to support side-by-side installs of multiple versions, with version selection by the environment modules facility. This is a small change (and improvement) on what I have today. It's a work in progress, but it's a manageable amount of work. It's looking likely that this multi-version packaging approach may be adopted by an official CentOS Scientific Repo, and this could change the landscape in terms of packaging of scientific applications for major Linux distributions. Who knows, that's all in the future. In the meantime, there's work going on in Galaxy land to package every application wanted in Galaxy, by writing install scripts, encapsulating environment variables in tool XML files, etc. (There is difficulty here in writing install scripts which work on every platform, which I have commented on previously.) The big question is, will the Galaxy team accept contributions to Galaxy to support different ways of solving this packaging problem? In particular, allowing for environment modules to be loaded as part of running a tool, and having this functionality in the toolshed? cheers, Simon === Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. === ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
