Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
On Thu, Nov 14, 2013 at 4:35 PM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi Dave, Good news, the BLAST+ tests appear to have all passed on the Test Tool Shed, http://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus/f2478dc77ccb ... Over on the main Tool Shed, the binary installation seems to be failing (still using the bash script magic - is the test system still missing bash, or is there a different problem here?). Here too, there doesn't seem to be any mention of the tools missing tests. At this point (given it is working on the Test Tool Shed), I think it should be safe to update the BLAST+ packages to use the new architecture/os specific action tags (a recent feature which is now supported in the stable Galaxy releases): http://toolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_26 http://toolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_27 http://toolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_28 I've just done that on the main Tool Shed, fingers crossed that will give a green light on the overnight test results for the BLAST+ wrappers. Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
On Tue, Nov 12, 2013 at 3:33 AM, Dave Bouvier d...@bx.psu.edu wrote: Peter, Thanks for bringing this to our attention, we're working on fixing a number of issues with the test framework, and hope to have more information for you tomorrow. --Dave B. Hi Dave, Good news, the BLAST+ tests appear to have all passed on the Test Tool Shed, http://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus/f2478dc77ccb Tool test results Automated test environment Time tested: ~ 5 hours ago System: Linux 3.8.0-30-generic Architecture: x86_64 Python version: 2.7.4 Galaxy revision: 11318:7553213e0646 Galaxy database version: 117 Tool shed revision: Tool shed database version: Tool shed mercurial version: Tests that passed successfully Tool id: blastxml_to_tabular Tool version: blastxml_to_tabular Test: test_tool_00 (functional.test_toolbox.TestForTool_testtoolshed.g2.bx.psu.edu/repos/peterjc/ncbi_blast_plus/blastxml_to_tabular/0.0.11) ... Tool id: ncbi_tblastx_wrapper Tool version: ncbi_tblastx_wrapper Test: test_tool_00 (functional.test_toolbox.TestForTool_testtoolshed.g2.bx.psu.edu/repos/peterjc/ncbi_blast_plus/ncbi_tblastx_wrapper/0.0.21) Curiously however, this no longer seems to be complaining about the BLAST+ tools without any tests - a new bug? Over on the main Tool Shed, the binary installation seems to be failing (still using the bash script magic - is the test system still missing bash, or is there a different problem here?). Here too, there doesn't seem to be any mention of the tools missing tests. At this point (given it is working on the Test Tool Shed), I think it should be safe to update the BLAST+ packages to use the new architecture/os specific action tags (a recent feature which is now supported in the stable Galaxy releases): http://toolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_26 http://toolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_27 http://toolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_28 Any objections? Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
Bash is easily obtained on these systems and I think the extra functionality available in post-Bourne shells ought to be allowed. I also question how many people are going to run tools on *BSD since most underlying analysis tools tend to only target Linux. That said, shell code should be restricted to Bourne-compatible syntax whenever there’s no good reason to use non-Bourne features, e.g. if all you’re doing is `export FOO=foo`, you should not be forcing the use of bash. In cases where bash really is required (say, you’re using arrays), the script should explicitly specify '#!/bin/bash' (or '#!/usr/bin/env bash'?) rather than '#!/bin/sh'. —nate On Nov 11, 2013, at 11:04 AM, James Taylor ja...@jamestaylor.org wrote: This is not an objection, but do we need bash? Can we live with posix sh? We should ask this question about every requirement we introduce. (bash is not part of the default installation of FreeBSD or OpenBSD for example. bash is unfortunately licensed under GPLV3, so if you are trying to create an OS not polluted by viral licensing you don't get bash). On Mon, Oct 7, 2013 at 11:36 AM, John Chilton chil...@msi.umn.edu wrote: My own preference is that we specify at least /bin/sh and /bin/bash are available before utilizing the tool shed. Is there an objection to this from any corner? Is there realistically a system that Galaxy should support that will not have /bin/bash available? -- James Taylor, Associate Professor, Biology/CS, Emory University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
On Thu, Nov 14, 2013 at 5:13 PM, Nate Coraor n...@bx.psu.edu wrote:
Bash is easily obtained on these systems and I think the extra functionality
available in post-Bourne shells ought to be allowed. I also question how
many people are going to run tools on *BSD since most underlying
analysis tools tend to only target Linux.
So could bash be declared an expected Galaxy dependency? i.e. The
core system libraries and tools which Tool Authors may assume, and
which Galaxy Administrators should install?
That said, shell code should be restricted to Bourne-compatible syntax
whenever there’s no good reason to use non-Bourne features, e.g. if
all you’re doing is `export FOO=foo`, you should not be forcing the use
of bash. In cases where bash really is required (say, you’re using
arrays), the script should explicitly specify '#!/bin/bash' (or
'#!/usr/bin/env
bash'?) rather than '#!/bin/sh'.
I agree that any shell script (e.g. a tool wrapper) which is bash specific
should say that in the hash-bang line, rather than '#!/bin/sh'.
What about command line magic like -num_threads \${GALAXY_SLOTS:-8}
in a command tag using bash specific environment variable default values?
What about bash specific if statements in action type=shell_command
sections of a tool_dependencies.xml file (which is what the BLAST+
packages currently use on the main Tool Shed, pending an update
to use arch/os specific tags as tested on the Test Tool Shed)?
Peter
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
On Nov 14, 2013, at 12:21 PM, Peter Cock p.j.a.c...@googlemail.com wrote:
On Thu, Nov 14, 2013 at 5:13 PM, Nate Coraor n...@bx.psu.edu wrote:
Bash is easily obtained on these systems and I think the extra functionality
available in post-Bourne shells ought to be allowed. I also question how
many people are going to run tools on *BSD since most underlying
analysis tools tend to only target Linux.
So could bash be declared an expected Galaxy dependency? i.e. The
core system libraries and tools which Tool Authors may assume, and
which Galaxy Administrators should install?
It’s my opinion that it could. I’ll start a discussion about this shortly so
we can hammer out the rest.
That said, shell code should be restricted to Bourne-compatible syntax
whenever there’s no good reason to use non-Bourne features, e.g. if
all you’re doing is `export FOO=foo`, you should not be forcing the use
of bash. In cases where bash really is required (say, you’re using
arrays), the script should explicitly specify '#!/bin/bash' (or
'#!/usr/bin/env
bash'?) rather than '#!/bin/sh'.
I agree that any shell script (e.g. a tool wrapper) which is bash specific
should say that in the hash-bang line, rather than '#!/bin/sh'.
What about command line magic like -num_threads \${GALAXY_SLOTS:-8}
in a command tag using bash specific environment variable default values?
${FOO:-default} is, surprisingly, Bourne-compatible.
What about bash specific if statements in action type=shell_command
sections of a tool_dependencies.xml file (which is what the BLAST+
packages currently use on the main Tool Shed, pending an update
to use arch/os specific tags as tested on the Test Tool Shed)?
This is a bit tricker, since there’s currently no way to specify installation
actions to run on a system other than the Galaxy application server. However,
if we are saying that bash should be available to tools, I’d think there is no
reason to say that it’s not expected to be available to tool installation.
Unfortunately, I believe the current installation methods use subprocesses’
default, which is sh, which is not going to be bash on some systems (on Debian,
it’s dash).
—nate
Peter
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
If, by current installation methods you are not referring to Tool Shed repository installation methods for installing tool dependencies, then please read no further. For installing tool dependencies along with repositories from the Tool Shed, fabric is currently being used. However, it's been on my list for a while to eliminate fabric if possible so that I have more management of the threads and process ids during installation. If fabric / subprocess is a problem here, I can attempt to raise the prority of looking at this. Greg Von Kuster On Nov 14, 2013, at 12:38 PM, Nate Coraor n...@bx.psu.edu wrote: This is a bit tricker, since there’s currently no way to specify installation actions to run on a system other than the Galaxy application server. However, if we are saying that bash should be available to tools, I’d think there is no reason to say that it’s not expected to be available to tool installation. Unfortunately, I believe the current installation methods use subprocesses’ default, which is sh, which is not going to be bash on some systems (on Debian, it’s dash). —nate ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
Hi Dave, I think this is a new regression from the Test Tool Shed installation framework, http://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus/f2478dc77ccb Tool test results Automated test environment Tools missing tests or test data Installation errors - no functional tests were run for any tools in this changeset revision Tool dependencies TypeNameVersion blast+ package 2.2.27 Error (Invalid file %s specified, ignoring set_environment_for_install action.,/ToolDepsTest/blast+/2.2.27/iuc/package_blast_plus_2_2_27/eab09bc4d63e/env.sh) Is that really a comma in the path? Is this a simple typo in a config file? Note this tool is using: http://testtoolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_27/eab09bc4d63e (using platform specific actions) -- Meanwhile, over on the main Tool Shed, which I would like to update to use arch-specific actions for the packages (since that is now in the stable Galaxy releases), and switch from BLAST+ 2.2.26 to 2.2.27, we have a different install failure: http://toolshed.g2.bx.psu.edu/view/devteam/ncbi_blast_plus/70e7dcbf6573 still using 2.2.26 via shell magic, http://toolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_26/40c69b76b46e /bin/sh: 1: blastn: not found Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
This is not an objection, but do we need bash? Can we live with posix sh? We should ask this question about every requirement we introduce. (bash is not part of the default installation of FreeBSD or OpenBSD for example. bash is unfortunately licensed under GPLV3, so if you are trying to create an OS not polluted by viral licensing you don't get bash). On Mon, Oct 7, 2013 at 11:36 AM, John Chilton chil...@msi.umn.edu wrote: My own preference is that we specify at least /bin/sh and /bin/bash are available before utilizing the tool shed. Is there an objection to this from any corner? Is there realistically a system that Galaxy should support that will not have /bin/bash available? -- James Taylor, Associate Professor, Biology/CS, Emory University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
Peter, Thanks for bringing this to our attention, we're working on fixing a number of issues with the test framework, and hope to have more information for you tomorrow. --Dave B. On 2013-11-11 06:11, Peter Cock wrote: Hi Dave, I think this is a new regression from the Test Tool Shed installation framework, http://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus/f2478dc77ccb Tool test results Automated test environment Tools missing tests or test data Installation errors - no functional tests were run for any tools in this changeset revision Tool dependencies TypeNameVersion blast+ package 2.2.27 Error (Invalid file %s specified, ignoring set_environment_for_install action.,/ToolDepsTest/blast+/2.2.27/iuc/package_blast_plus_2_2_27/eab09bc4d63e/env.sh) Is that really a comma in the path? Is this a simple typo in a config file? Note this tool is using: http://testtoolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_27/eab09bc4d63e (using platform specific actions) -- Meanwhile, over on the main Tool Shed, which I would like to update to use arch-specific actions for the packages (since that is now in the stable Galaxy releases), and switch from BLAST+ 2.2.26 to 2.2.27, we have a different install failure: http://toolshed.g2.bx.psu.edu/view/devteam/ncbi_blast_plus/70e7dcbf6573 still using 2.2.26 via shell magic, http://toolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_26/40c69b76b46e /bin/sh: 1: blastn: not found Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
Peter, I did some investigating, and it turns out that the if [[ condition ]] syntax used in the ncbi tool dependency is only compatible with bash, not sh. For sh, the corresponding syntax would look like: if [ $string = value ] then echo string is equal to value fi However, it strikes me that the platform detection and download url determination could also be done using the recently introduced actions_group feature, which would bypass shell-dependent conditional syntax entirely. --Dave B. On 10/07/2013 09:45 AM, Peter Cock wrote: On Tue, Sep 24, 2013 at 10:58 AM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi all, Some good news, the strange missing /bin/sh error has gone away on the Test Tool Shed: http://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus/aecdffbc08fc The BLAST+ wrapper tests also look good on the main Tool Shed: http://toolshed.g2.bx.psu.edu/view/devteam/ncbi_blast_plus /70e7dcbf6573 Hooray! Peter P.S. Another plug for explicitly testing Tool Dependency Definition package installation on the Tool Shed: https://trello.com/c/HVGrShnC/1042-tool-shed-should-test-installation-of-packages Bad news, the strange missing /bin/sh error has come back again on both the main and Test Tool Shed :( These are the same revisions of the BLAST+ wrapper, so the only changes are in Galaxy itself, the Tool Shed itself, or the Tool Shed test environment: http://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus/aecdffbc08fc http://toolshed.g2.bx.psu.edu/view/devteam/ncbi_blast_plus /70e7dcbf6573 Installation errors - no functional tests were run for any tools in this changeset revision Tool dependencies TypeNameVersion blast+ package 2.2.26+ Error /bin/sh: 2: [[: not found /bin/sh: 3: [[: not found /bin/sh: 4: [[: not found /bin/sh: 6: [[: not found /bin/sh: 7: [[: not found tar: option requires an argument -- 'f' Try `tar --help' or `tar --usage' for more information. Regards, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
On Mon, Oct 7, 2013 at 3:44 PM, Peter Cock p.j.a.c...@googlemail.com wrote: On Mon, Oct 7, 2013 at 3:37 PM, Dave Bouvier d...@bx.psu.edu wrote: Peter, I did some investigating, and it turns out that the if [[ condition ]] syntax used in the ncbi tool dependency is only compatible with bash, not sh. Does that mean the test system has switched between sh and bash and back to sh again? Is there anything written down about the expected shell(s) available for a standard Galaxy instance? For sh, the corresponding syntax would look like: if [ $string = value ] then echo string is equal to value fi However, it strikes me that the platform detection and download url determination could also be done using the recently introduced actions_group feature, which would bypass shell-dependent conditional syntax entirely. Yes, but is that in stable releases for galaxy-dist yet? In the meantime I will try that on the Test Tool Shed, based on the tool_dependencies.xml example you shared before: http://testtoolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_26/c83440a9bdfb Note there seems to me to be a lot of duplication - I would like a way to unambiguously combine multiple CPU architectures into a single action. For example, combine these: actions os=darwin architecture=x86_64 actions os=darwin architecture=i386 into: actions os=darwin architecture=x86_64,i386 or just: actions os=darwin and similarly, combine these: actions os=linux architecture=i386 actions os=linux architecture=i686 into something like: actions os=linux architecture=i386,i686 Assuming that works on the Test Tool Shed, and the code for this is already in the stable releases, I will update the main Tool Shed definition as well. Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
Peter, The platform detection code is not yet in the stable branch, but is intended to be included in the upcoming release. The automated testing framework does run the default branch, though, so the nightly tests will make use of the new definitions. --Dave B. On 10/07/2013 11:22 AM, Peter Cock wrote: On Mon, Oct 7, 2013 at 3:44 PM, Peter Cock p.j.a.c...@googlemail.com wrote: On Mon, Oct 7, 2013 at 3:37 PM, Dave Bouvier d...@bx.psu.edu wrote: Peter, I did some investigating, and it turns out that the if [[ condition ]] syntax used in the ncbi tool dependency is only compatible with bash, not sh. Does that mean the test system has switched between sh and bash and back to sh again? Is there anything written down about the expected shell(s) available for a standard Galaxy instance? For sh, the corresponding syntax would look like: if [ $string = value ] then echo string is equal to value fi However, it strikes me that the platform detection and download url determination could also be done using the recently introduced actions_group feature, which would bypass shell-dependent conditional syntax entirely. Yes, but is that in stable releases for galaxy-dist yet? In the meantime I will try that on the Test Tool Shed, based on the tool_dependencies.xml example you shared before: http://testtoolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_26/c83440a9bdfb Note there seems to me to be a lot of duplication - I would like a way to unambiguously combine multiple CPU architectures into a single action. For example, combine these: actions os=darwin architecture=x86_64 actions os=darwin architecture=i386 into: actions os=darwin architecture=x86_64,i386 or just: actions os=darwin and similarly, combine these: actions os=linux architecture=i386 actions os=linux architecture=i686 into something like: actions os=linux architecture=i386,i686 Assuming that works on the Test Tool Shed, and the code for this is already in the stable releases, I will update the main Tool Shed definition as well. Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
actions_group is neither in stable nor in default branch of galaxy-dist repository, it's present only in galaxy-central. Nicola Il giorno lun, 07/10/2013 alle 11.24 -0400, Dave Bouvier ha scritto: Peter, The platform detection code is not yet in the stable branch, but is intended to be included in the upcoming release. The automated testing framework does run the default branch, though, so the nightly tests will make use of the new definitions. --Dave B. On 10/07/2013 11:22 AM, Peter Cock wrote: On Mon, Oct 7, 2013 at 3:44 PM, Peter Cock p.j.a.c...@googlemail.com wrote: On Mon, Oct 7, 2013 at 3:37 PM, Dave Bouvier d...@bx.psu.edu wrote: Peter, I did some investigating, and it turns out that the if [[ condition ]] syntax used in the ncbi tool dependency is only compatible with bash, not sh. Does that mean the test system has switched between sh and bash and back to sh again? Is there anything written down about the expected shell(s) available for a standard Galaxy instance? For sh, the corresponding syntax would look like: if [ $string = value ] then echo string is equal to value fi However, it strikes me that the platform detection and download url determination could also be done using the recently introduced actions_group feature, which would bypass shell-dependent conditional syntax entirely. Yes, but is that in stable releases for galaxy-dist yet? In the meantime I will try that on the Test Tool Shed, based on the tool_dependencies.xml example you shared before: http://testtoolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_26/c83440a9bdfb Note there seems to me to be a lot of duplication - I would like a way to unambiguously combine multiple CPU architectures into a single action. For example, combine these: actions os=darwin architecture=x86_64 actions os=darwin architecture=i386 into: actions os=darwin architecture=x86_64,i386 or just: actions os=darwin and similarly, combine these: actions os=linux architecture=i386 actions os=linux architecture=i686 into something like: actions os=linux architecture=i386,i686 Assuming that works on the Test Tool Shed, and the code for this is already in the stable releases, I will update the main Tool Shed definition as well. Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
On Mon, Oct 7, 2013 at 10:22 AM, Peter Cock p.j.a.c...@googlemail.com wrote: On Mon, Oct 7, 2013 at 3:44 PM, Peter Cock p.j.a.c...@googlemail.com wrote: On Mon, Oct 7, 2013 at 3:37 PM, Dave Bouvier d...@bx.psu.edu wrote: Peter, I did some investigating, and it turns out that the if [[ condition ]] syntax used in the ncbi tool dependency is only compatible with bash, not sh. Does that mean the test system has switched between sh and bash and back to sh again? Is there anything written down about the expected shell(s) available for a standard Galaxy instance? I don't think it is documented anywhere and there has been some disagreement on this in the past. I think we should do our best to reach some sort of conclusion on this and document it in the following card: https://trello.com/c/7VTlX9rD My own preference is that we specify at least /bin/sh and /bin/bash are available before utilizing the tool shed. Is there an objection to this from any corner? Is there realistically a system that Galaxy should support that will not have /bin/bash available? I know there is some difference in bash behavior between platforms, but that is hopefully minimal and we can address inconsistencies one at a time. Thanks, -John For sh, the corresponding syntax would look like: if [ $string = value ] then echo string is equal to value fi However, it strikes me that the platform detection and download url determination could also be done using the recently introduced actions_group feature, which would bypass shell-dependent conditional syntax entirely. Yes, but is that in stable releases for galaxy-dist yet? In the meantime I will try that on the Test Tool Shed, based on the tool_dependencies.xml example you shared before: http://testtoolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_26/c83440a9bdfb Note there seems to me to be a lot of duplication - I would like a way to unambiguously combine multiple CPU architectures into a single action. For example, combine these: actions os=darwin architecture=x86_64 actions os=darwin architecture=i386 into: actions os=darwin architecture=x86_64,i386 or just: actions os=darwin and similarly, combine these: actions os=linux architecture=i386 actions os=linux architecture=i686 into something like: actions os=linux architecture=i386,i686 Assuming that works on the Test Tool Shed, and the code for this is already in the stable releases, I will update the main Tool Shed definition as well. Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
On Mon, Oct 7, 2013 at 4:45 PM, Nicola Soranzo sora...@crs4.it wrote: actions_group is neither in stable nor in default branch of galaxy-dist repository, it's present only in galaxy-central. Nicola i.e. for now only they can only be used and tested on the Test Tool Shed? It would be helpful if the bash vs sh problem on the (Test) Tool Shed cluster could be addressed in the short term (and as John points out, documented). Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
Am Montag, den 07.10.2013, 10:36 -0500 schrieb John Chilton: On Mon, Oct 7, 2013 at 10:22 AM, Peter Cock p.j.a.c...@googlemail.com wrote: On Mon, Oct 7, 2013 at 3:44 PM, Peter Cock p.j.a.c...@googlemail.com wrote: On Mon, Oct 7, 2013 at 3:37 PM, Dave Bouvier d...@bx.psu.edu wrote: Peter, I did some investigating, and it turns out that the if [[ condition ]] syntax used in the ncbi tool dependency is only compatible with bash, not sh. Does that mean the test system has switched between sh and bash and back to sh again? Is there anything written down about the expected shell(s) available for a standard Galaxy instance? I don't think it is documented anywhere and there has been some disagreement on this in the past. I think we should do our best to reach some sort of conclusion on this and document it in the following card: https://trello.com/c/7VTlX9rD My own preference is that we specify at least /bin/sh and /bin/bash are available before utilizing the tool shed. Is there an objection to this from any corner? Is there realistically a system that Galaxy should support that will not have /bin/bash available? I know there is some difference in bash behavior between platforms, but that is hopefully minimal and we can address inconsistencies one at a time. I think we can and should assume /bin/bash on every system. We also should finally write down the dependency list. We talked a lot about it and we collected a lot of system libraries and dependencies the TS and all tools can depend on. We just need to write it down. I will comment on that trello card! Bjoern Thanks, -John For sh, the corresponding syntax would look like: if [ $string = value ] then echo string is equal to value fi However, it strikes me that the platform detection and download url determination could also be done using the recently introduced actions_group feature, which would bypass shell-dependent conditional syntax entirely. Yes, but is that in stable releases for galaxy-dist yet? In the meantime I will try that on the Test Tool Shed, based on the tool_dependencies.xml example you shared before: http://testtoolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_26/c83440a9bdfb Note there seems to me to be a lot of duplication - I would like a way to unambiguously combine multiple CPU architectures into a single action. For example, combine these: actions os=darwin architecture=x86_64 actions os=darwin architecture=i386 into: actions os=darwin architecture=x86_64,i386 or just: actions os=darwin and similarly, combine these: actions os=linux architecture=i386 actions os=linux architecture=i686 into something like: actions os=linux architecture=i386,i686 Assuming that works on the Test Tool Shed, and the code for this is already in the stable releases, I will update the main Tool Shed definition as well. Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
Hi all, Some good news, the strange missing /bin/sh error has gone away on the Test Tool Shed: http://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus/aecdffbc08fc The BLAST+ wrapper tests also look good on the main Tool Shed: http://toolshed.g2.bx.psu.edu/view/devteam/ncbi_blast_plus /70e7dcbf6573 Hooray! Peter P.S. Another plug for explicitly testing Tool Dependency Definition package installation on the Tool Shed: https://trello.com/c/HVGrShnC/1042-tool-shed-should-test-installation-of-packages ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
On Fri, Sep 20, 2013 at 11:10 AM, Peter Cock p.j.a.c...@googlemail.com wrote: On Wed, Sep 18, 2013 at 11:47 AM, Peter Cock p.j.a.c...@googlemail.com wrote: On Tue, Aug 27, 2013 at 2:18 PM, Dave Bouvier d...@bx.psu.edu wrote: Peter, I also tried running the command that returns error code 64 on the same system that runs the automated tests, and it downloaded the correct file for that operating system and architecture. So I'm not sure why it's failing when run through buildbot, but I'll look into it and get back to you. I've added a trello card to track progress on this issue: https://trello.com/c/9ERDMc7j/1079-toolshed-investigate-cause-of-shell-commands-failing-through-buildbot-when-they-are-valid-commands --Dave B. Hi Dave, Something seems to have changed again on the Test Tool Shed, http://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus/4ae1bac976f3 Installation errors - no functional tests were run for any tools in this changeset revision Tool dependencies TypeNameVersion blast+ package 2.2.26+ Error /bin/sh: 2: [[: not found /bin/sh: 3: [[: not found /bin/sh: 4: [[: not found /bin/sh: 6: [[: not found /bin/sh: 7: [[: not found tar: option requires an argument -- 'f' Try `tar --help' or `tar --usage' for more information. Is the test cluster really missing /bin/sh? Can you check that and post the full installation log please? I really would like to push an update to ncbi_blast_plus to the main Tool Shed with an updated README file (using RST) and the new citation information - plus of course handling the binary dependency via the new package: http://toolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_26 http://testtoolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_26 Once this is done there is a backlog of updates I want to tackle following our productive BoF meeting at GCC2013: http://wiki.galaxyproject.org/Events/GCC2013/BoF/GalaxyBlast Thanks, Peter A related bug for Greg (I think), despite this strange no /bin/sh error, this repository is not listed on the Test Tool Shed under Latest revision: installation errors. Regards, Peter Aside from the oddities reported via the Test Tool Shed testing, feedback for the package_blast_plus_2_2_26 dependency has been positive, so I have just updated the BLAST+ wrappers on the main Tool Shed to use it (v0.0.20): http://toolshed.g2.bx.psu.edu/view/devteam/ncbi_blast_plus/70e7dcbf6573 Fingers crossed this works 'in the wild', and that the curious missing /bin/sh issue on the test machines can be solved. Regards, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
On Wed, Sep 18, 2013 at 11:47 AM, Peter Cock p.j.a.c...@googlemail.com wrote: On Tue, Aug 27, 2013 at 2:18 PM, Dave Bouvier d...@bx.psu.edu wrote: Peter, I also tried running the command that returns error code 64 on the same system that runs the automated tests, and it downloaded the correct file for that operating system and architecture. So I'm not sure why it's failing when run through buildbot, but I'll look into it and get back to you. I've added a trello card to track progress on this issue: https://trello.com/c/9ERDMc7j/1079-toolshed-investigate-cause-of-shell-commands-failing-through-buildbot-when-they-are-valid-commands --Dave B. Hi Dave, Something seems to have changed again on the Test Tool Shed, http://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus/4ae1bac976f3 Installation errors - no functional tests were run for any tools in this changeset revision Tool dependencies TypeNameVersion blast+ package 2.2.26+ Error /bin/sh: 2: [[: not found /bin/sh: 3: [[: not found /bin/sh: 4: [[: not found /bin/sh: 6: [[: not found /bin/sh: 7: [[: not found tar: option requires an argument -- 'f' Try `tar --help' or `tar --usage' for more information. Is the test cluster really missing /bin/sh? Can you check that and post the full installation log please? I really would like to push an update to ncbi_blast_plus to the main Tool Shed with an updated README file (using RST) and the new citation information - plus of course handling the binary dependency via the new package: http://toolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_26 http://testtoolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_26 Once this is done there is a backlog of updates I want to tackle following our productive BoF meeting at GCC2013: http://wiki.galaxyproject.org/Events/GCC2013/BoF/GalaxyBlast Thanks, Peter A related bug for Greg (I think), despite this strange no /bin/sh error, this repository is not listed on the Test Tool Shed under Latest revision: installation errors. Regards, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
On Tue, Aug 27, 2013 at 2:18 PM, Dave Bouvier d...@bx.psu.edu wrote: Peter, I also tried running the command that returns error code 64 on the same system that runs the automated tests, and it downloaded the correct file for that operating system and architecture. So I'm not sure why it's failing when run through buildbot, but I'll look into it and get back to you. I've added a trello card to track progress on this issue: https://trello.com/c/9ERDMc7j/1079-toolshed-investigate-cause-of-shell-commands-failing-through-buildbot-when-they-are-valid-commands --Dave B. Hi Dave, Something seems to have changed again on the Test Tool Shed, http://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus/4ae1bac976f3 Installation errors - no functional tests were run for any tools in this changeset revision Tool dependencies TypeNameVersion blast+ package 2.2.26+ Error /bin/sh: 2: [[: not found /bin/sh: 3: [[: not found /bin/sh: 4: [[: not found /bin/sh: 6: [[: not found /bin/sh: 7: [[: not found tar: option requires an argument -- 'f' Try `tar --help' or `tar --usage' for more information. Is the test cluster really missing /bin/sh? Can you check that and post the full installation log please? I really would like to push an update to ncbi_blast_plus to the main Tool Shed with an updated README file (using RST) and the new citation information - plus of course handling the binary dependency via the new package: http://toolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_26 http://testtoolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_26 Once this is done there is a backlog of updates I want to tackle following our productive BoF meeting at GCC2013: http://wiki.galaxyproject.org/Events/GCC2013/BoF/GalaxyBlast Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
Simon wrote: An example is the NCBI BLAST+ suite, which failed to build on my (EL6) system, because it couldn't run /usr/bin/touch. That's pretty dumb, and pretty simple to solve in isolation - it needs to be running /bin/touch instead. This was an oddity in the NCBI compile scripts for BLAST 2.2.26+ which can be avoided via a symlink, editing their code, or more simply just downloading their provided binaries. I prefer the later because it is simpler, faster, and should better ensure reproducibility by eliminating things like different compilers and their settings. Unfortunately the new install process to do this from within the Tool Shed is glitching on the Tool Shed's nightly tests: Nevertheless, the simple binary install is working for me and Bjoern. Could you try the Test Tool Shed package and let us know if that works for you? Hi Peter, It seems to me that there's quite a bit still to do on the Tool Shed dependency packaging machinery. I don't think I'm inclined to be part of that testing process, sorry. (Rather short of time, having to choose carefully where I get most bang for buck.) As an alternative, I note your requirement for having multiple versions of a package installed side-by-side, which isn't traditionally offered by package repositories, although there is nothing to preclude that. In fact, I am currently in discussion with the CentOS project leader about collaborating on a scientific repo for Enterprise Linux (RHEL, CentOS, Scientific Linux, Oracle Linux). I will be feeding in my ideas for multiple version installs, and will see where that takes us. Who knows, you may have better choices soon on how to install packages for the Tool Shed. cheers, Simon === Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. === ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
Hi Simon, Further to our discussion here: http://lists.bx.psu.edu/pipermail/galaxy-dev/2013-August/016310.html http://lists.bx.psu.edu/pipermail/galaxy-dev/2013-August/016316.html Simon wrote: An example is the NCBI BLAST+ suite, which failed to build on my (EL6) system, because it couldn't run /usr/bin/touch. That's pretty dumb, and pretty simple to solve in isolation - it needs to be running /bin/touch instead. This was an oddity in the NCBI compile scripts for BLAST 2.2.26+ which can be avoided via a symlink, editing their code, or more simply just downloading their provided binaries. I prefer the later because it is simpler, faster, and should better ensure reproducibility by eliminating things like different compilers and their settings. Unfortunately the new install process to do this from within the Tool Shed is glitching on the Tool Shed's nightly tests: On Tue, Aug 27, 2013 at 2:18 PM, Dave Bouvier d...@bx.psu.edu wrote: Peter, I also tried running the command that returns error code 64 on the same system that runs the automated tests, and it downloaded the correct file for that operating system and architecture. So I'm not sure why it's failing when run through buildbot, but I'll look into it and get back to you. I've added a trello card to track progress on this issue: https://trello.com/c/9ERDMc7j/1079-toolshed-investigate-cause-of-shell-commands-failing-through-buildbot-when-they-are-valid-commands --Dave B. Nevertheless, the simple binary install is working for me and Bjoern. Could you try the Test Tool Shed package and let us know if that works for you? Test Tool Shed, http://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus depends on this, http://testtoolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_26 Or, try installing this directly (although it won't get used yet): http://toolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_26 Given some confirmation then I think I will update the Main Tool Shed http://toolshed.g2.bx.psu.edu/view/devteam/ncbi_blast_plus (which currently tries to compile BLAST) to instead depend on the new package which fetches the binaries. Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
Peter, I also tried running the command that returns error code 64 on the same system that runs the automated tests, and it downloaded the correct file for that operating system and architecture. So I'm not sure why it's failing when run through buildbot, but I'll look into it and get back to you. I've added a trello card to track progress on this issue: https://trello.com/c/9ERDMc7j/1079-toolshed-investigate-cause-of-shell-commands-failing-through-buildbot-when-they-are-valid-commands --Dave B. On 08/26/2013 10:32 AM, Peter Cock wrote: On Mon, Aug 26, 2013 at 3:10 PM, Dave Bouvier d...@bx.psu.edu wrote: Peter, I apologize for the delay, here is the information you requested: ubuntu@ip-10-0-0-243:~$ uname -a Linux ip-10-0-0-243 3.8.0-25-generic #37-Ubuntu SMP Thu Jun 6 20:47:07 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux ubuntu@ip-10-0-0-243:~$ arch x86_64 ubuntu@ip-10-0-0-243:~$ echo $0 -bash --Dave B. Thanks Dave, Sadly that looks just like my server where it works, $ uname Linux $ arch x86_64 $ echo $0 -bash I was considering trying to have the tool_dependencies.xml install recipe call a shell or Python script instead - but the new Tool Shed Tool dependency definition repository type prevents that (unless using a workaround like downloading the script first). Hmm. Does anyone have any ideas on this error return code 64? Regards, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
Peter, I apologize for the delay, here is the information you requested: ubuntu@ip-10-0-0-243:~$ uname -a Linux ip-10-0-0-243 3.8.0-25-generic #37-Ubuntu SMP Thu Jun 6 20:47:07 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux ubuntu@ip-10-0-0-243:~$ arch x86_64 ubuntu@ip-10-0-0-243:~$ echo $0 -bash --Dave B. On 08/26/2013 09:44 AM, Peter Cock wrote: (Off list) Hi Dave, Greg, Just a reminder - without a bit more information about the Tool Shed cluster I am stuck about how to debug this install issue. Thanks, Peter On Wed, Aug 21, 2013 at 2:05 PM, Peter Cock p.j.a.c...@googlemail.com wrote: On Wed, Aug 21, 2013 at 1:45 PM, Dave Bouvier d...@bx.psu.edu wrote: Peter, Here is the relevant log output for the installation of ncbi_blast_plus. A quick google informs me that exit code 64 means command line usage error, for what it's worth. ... tool_shed.util.shed_util_common DEBUG 2013-08-20 23:34:47,886 Adding new row (or updating an existing row) for repository 'package_blast_plus_2_2_26' in the tool_shed_repository table, status set to 'New'. Warning: local() encountered an error (return code 64) while executing ' if [[ $(uname) == Linux ]]; then export FILENAME=ncbi-blast-2.2.26+-ia32-linux.tar.gz; fi if [[ $(arch) == x86_64 ]]; then export FILENAME=ncbi-blast-2.2.26+-x64-linux.tar.gz; fi if [[ $(uname) == Darwin ]]; then export FILENAME=ncbi-blast-2.2.26+-universal-macosx.tar.gz; fi echo Fetching $FILENAME if [[ $(uname) == Linux ]]; then wget ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/2.2.26/$FILENAME; fi if [[ $(uname) == Darwin ]]; then curl -O ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/2.2.26/$FILENAME; fi tar -zxvf $FILENAME echo Downloaded and decompressed ' Thanks Dave, I've made a note on this Trello card about exposing the log information of unsuccessful builds on the Tool Shed, https://trello.com/c/HVGrShnC/1042-tool-shed-should-test-installation-of-packages Can you tell me what version of Linux this is, and which shell? $ uname ... $ arch ... $ echo $0 ... That might give me a clue about the nature of the failure. My hunch is that I am using something bash specific. The simplest solution would be if you could run each of these command by hand to see if it triggers the same error: if [[ $(uname) == Linux ]]; then export FILENAME=ncbi-blast-2.2.26+-ia32-linux.tar.gz; fi if [[ $(arch) == x86_64 ]]; then export FILENAME=ncbi-blast-2.2.26+-x64-linux.tar.gz; fi if [[ $(uname) == Darwin ]]; then export FILENAME=ncbi-blast-2.2.26+-universal-macosx.tar.gz; fi echo Fetching $FILENAME if [[ $(uname) == Linux ]]; then wget ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/2.2.26/$FILENAME; fi if [[ $(uname) == Darwin ]]; then curl -O ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/2.2.26/$FILENAME; fi tar -zxvf $FILENAME echo Downloaded and decompressed Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
On Mon, Aug 26, 2013 at 3:10 PM, Dave Bouvier d...@bx.psu.edu wrote: Peter, I apologize for the delay, here is the information you requested: ubuntu@ip-10-0-0-243:~$ uname -a Linux ip-10-0-0-243 3.8.0-25-generic #37-Ubuntu SMP Thu Jun 6 20:47:07 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux ubuntu@ip-10-0-0-243:~$ arch x86_64 ubuntu@ip-10-0-0-243:~$ echo $0 -bash --Dave B. Thanks Dave, Sadly that looks just like my server where it works, $ uname Linux $ arch x86_64 $ echo $0 -bash I was considering trying to have the tool_dependencies.xml install recipe call a shell or Python script instead - but the new Tool Shed Tool dependency definition repository type prevents that (unless using a workaround like downloading the script first). Hmm. Does anyone have any ideas on this error return code 64? Regards, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
Hi Dave, I can confirm that it is also working for me. uname -a Linux threonin 3.5.0-36-generic #57-Ubuntu SMP Wed Jun 19 15:10:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux bag@threonin:~/projects/github/galaxytools$ arch x86_64 bag@threonin:~/projects/github/galaxytools$ echo $0 bash Is the toolshed somehow running in chroot or so? Dave can you grep through the log file and can have a look why the dependency shell is not build correctly? Thanks, Bjoern Peter, I apologize for the delay, here is the information you requested: ubuntu@ip-10-0-0-243:~$ uname -a Linux ip-10-0-0-243 3.8.0-25-generic #37-Ubuntu SMP Thu Jun 6 20:47:07 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux ubuntu@ip-10-0-0-243:~$ arch x86_64 ubuntu@ip-10-0-0-243:~$ echo $0 -bash --Dave B. On 08/26/2013 09:44 AM, Peter Cock wrote: (Off list) Hi Dave, Greg, Just a reminder - without a bit more information about the Tool Shed cluster I am stuck about how to debug this install issue. Thanks, Peter On Wed, Aug 21, 2013 at 2:05 PM, Peter Cock p.j.a.c...@googlemail.com wrote: On Wed, Aug 21, 2013 at 1:45 PM, Dave Bouvier d...@bx.psu.edu wrote: Peter, Here is the relevant log output for the installation of ncbi_blast_plus. A quick google informs me that exit code 64 means command line usage error, for what it's worth. ... tool_shed.util.shed_util_common DEBUG 2013-08-20 23:34:47,886 Adding new row (or updating an existing row) for repository 'package_blast_plus_2_2_26' in the tool_shed_repository table, status set to 'New'. Warning: local() encountered an error (return code 64) while executing ' if [[ $(uname) == Linux ]]; then export FILENAME=ncbi-blast-2.2.26+-ia32-linux.tar.gz; fi if [[ $(arch) == x86_64 ]]; then export FILENAME=ncbi-blast-2.2.26+-x64-linux.tar.gz; fi if [[ $(uname) == Darwin ]]; then export FILENAME=ncbi-blast-2.2.26+-universal-macosx.tar.gz; fi echo Fetching $FILENAME if [[ $(uname) == Linux ]]; then wget ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/2.2.26/$FILENAME; fi if [[ $(uname) == Darwin ]]; then curl -O ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/2.2.26/$FILENAME; fi tar -zxvf $FILENAME echo Downloaded and decompressed ' Thanks Dave, I've made a note on this Trello card about exposing the log information of unsuccessful builds on the Tool Shed, https://trello.com/c/HVGrShnC/1042-tool-shed-should-test-installation-of-packages Can you tell me what version of Linux this is, and which shell? $ uname ... $ arch ... $ echo $0 ... That might give me a clue about the nature of the failure. My hunch is that I am using something bash specific. The simplest solution would be if you could run each of these command by hand to see if it triggers the same error: if [[ $(uname) == Linux ]]; then export FILENAME=ncbi-blast-2.2.26+-ia32-linux.tar.gz; fi if [[ $(arch) == x86_64 ]]; then export FILENAME=ncbi-blast-2.2.26+-x64-linux.tar.gz; fi if [[ $(uname) == Darwin ]]; then export FILENAME=ncbi-blast-2.2.26+-universal-macosx.tar.gz; fi echo Fetching $FILENAME if [[ $(uname) == Linux ]]; then wget ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/2.2.26/$FILENAME; fi if [[ $(uname) == Darwin ]]; then curl -O ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/2.2.26/$FILENAME; fi tar -zxvf $FILENAME echo Downloaded and decompressed Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
On Tue, Aug 20, 2013 at 9:15 PM, Dave Bouvier d...@bx.psu.edu wrote: Peter, I believe this was again due to the installation of an earlier repository hanging until the testing framework canceled the build. I've added the repository in question to the exclude list, and hopefully your blast repositories will be properly tested within the next few hours. --Dave B. Thanks Dave, Re: http://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus/946749c8605f The output has changed, but we're back to missing any test results at all now, all I get is: ... Automated tool test results Tool test results Automated test environment Time tested: ~ 10 hours ago System: Linux 3.8.0-25-generic Architecture: x86_64 Python version: 2.7.4 Galaxy revision: 10419:4f4e01316260 Galaxy database version: 115 Tool shed revision: 10413:6e148e87d819 Tool shed database version: 21 Tool shed mercurial version: 2.2.3 Automated tool tests ... Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
On Wed, Aug 21, 2013 at 10:54 AM, Peter Cock p.j.a.c...@googlemail.com wrote: On Tue, Aug 20, 2013 at 9:15 PM, Dave Bouvier d...@bx.psu.edu wrote: Peter, I believe this was again due to the installation of an earlier repository hanging until the testing framework canceled the build. I've added the repository in question to the exclude list, and hopefully your blast repositories will be properly tested within the next few hours. --Dave B. Thanks Dave, Re: http://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus/946749c8605f The output has changed, but we're back to missing any test results at all now, all I get is: ... Automated tool test results Tool test results Automated test environment Time tested: ~ 10 hours ago System: Linux 3.8.0-25-generic Architecture: x86_64 Python version: 2.7.4 Galaxy revision: 10419:4f4e01316260 Galaxy database version: 115 Tool shed revision: 10413:6e148e87d819 Tool shed database version: 21 Tool shed mercurial version: 2.2.3 Automated tool tests ... Test results now visible now (odd?), back to the install problem: Fatal error: Exit code 127 () /bin/sh: 1: blastn: not found /bin/sh: 1: blastn: not found Any clues from the Test Tool Shed log file would be helpful, Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
Peter, Here is the relevant log output for the installation of ncbi_blast_plus. A quick google informs me that exit code 64 means command line usage error, for what it's worth. install_and_test_repositories INFO 2013-08-20 23:34:43,270 Installing and testing revision 946749c8605f of repository ncbi_blast_plus owned by peterjc... install_repository_ncbi_blast_plus (install_and_test_tool_shed_repositories.functional.test_install_repositories.TestInstallRepository_ncbi_blast_plus) Install the repository ncbi_blast_plus from http://testtoolshed.g2.bx.psu.edu. ... tool_shed.util.shed_util_common DEBUG 2013-08-20 23:34:47,484 Adding new row (or updating an existing row) for repository 'ncbi_blast_plus' in the tool_shed_repository table, status set to 'New'. tool_shed.util.shed_util_common DEBUG 2013-08-20 23:34:47,674 Adding new row (or updating an existing row) for repository 'blast_datatypes' in the tool_shed_repository table, status set to 'New'. tool_shed.util.shed_util_common DEBUG 2013-08-20 23:34:47,886 Adding new row (or updating an existing row) for repository 'package_blast_plus_2_2_26' in the tool_shed_repository table, status set to 'New'. Warning: local() encountered an error (return code 64) while executing ' if [[ $(uname) == Linux ]]; then export FILENAME=ncbi-blast-2.2.26+-ia32-linux.tar.gz; fi if [[ $(arch) == x86_64 ]]; then export FILENAME=ncbi-blast-2.2.26+-x64-linux.tar.gz; fi if [[ $(uname) == Darwin ]]; then export FILENAME=ncbi-blast-2.2.26+-universal-macosx.tar.gz; fi echo Fetching $FILENAME if [[ $(uname) == Linux ]]; then wget ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/2.2.26/$FILENAME; fi if [[ $(uname) == Darwin ]]; then curl -O ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/2.2.26/$FILENAME; fi tar -zxvf $FILENAME echo Downloaded and decompressed ' galaxy.datatypes.registry DEBUG 2013-08-20 23:34:52,010 Loading datatypes from /var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/tmp/tmpskjEfq/tmp-toolshed-acalpdoM4u1f galaxy.datatypes.registry DEBUG 2013-08-20 23:34:52,022 Loaded sniffer for datatype 'galaxy.datatypes.blast:BlastXml' tool_shed.util.tool_util DEBUG 2013-08-20 23:34:54,285 The '/var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/tool-data/testtoolshed.g2.bx.psu.edu/repos/peterjc/ncbi_blast_plus/946749c8605f/tool_data_table_conf.xml' data table file was not found, but was expected to be copied from 'tool_data_table_conf.xml.sample' during repository installation. galaxy.tools DEBUG 2013-08-20 23:34:55,359 Loaded tool id: testtoolshed.g2.bx.psu.edu/repos/peterjc/ncbi_blast_plus/ncbi_blastx_wrapper/0.0.19, version: 0.0.19 into tool panel. galaxy.tools DEBUG 2013-08-20 23:34:55,435 Loaded tool id: testtoolshed.g2.bx.psu.edu/repos/peterjc/ncbi_blast_plus/ncbi_tblastx_wrapper/0.0.20, version: 0.0.20 into tool panel. galaxy.tools DEBUG 2013-08-20 23:34:55,510 Loaded tool id: testtoolshed.g2.bx.psu.edu/repos/peterjc/ncbi_blast_plus/ncbi_tblastn_wrapper/0.0.20, version: 0.0.20 into tool panel. galaxy.tools DEBUG 2013-08-20 23:34:55,541 Loaded tool id: testtoolshed.g2.bx.psu.edu/repos/peterjc/ncbi_blast_plus/ncbi_blastdbcmd_info/0.0.6, version: 0.0.6 into tool panel. galaxy.tools DEBUG 2013-08-20 23:34:55,618 Loaded tool id: testtoolshed.g2.bx.psu.edu/repos/peterjc/ncbi_blast_plus/ncbi_blastp_wrapper/0.0.20, version: 0.0.20 into tool panel. galaxy.tools DEBUG 2013-08-20 23:34:55,693 Loaded tool id: testtoolshed.g2.bx.psu.edu/repos/peterjc/ncbi_blast_plus/ncbi_rpstblastn_wrapper/0.0.4, version: 0.0.4 into tool panel. galaxy.tools DEBUG 2013-08-20 23:34:55,722 Loaded tool id: testtoolshed.g2.bx.psu.edu/repos/peterjc/ncbi_blast_plus/ncbi_makeblastdb/0.0.5, version: 0.0.5 into tool panel. galaxy.tools DEBUG 2013-08-20 23:34:55,797 Loaded tool id: testtoolshed.g2.bx.psu.edu/repos/peterjc/ncbi_blast_plus/ncbi_blastn_wrapper/0.0.20, version: 0.0.20 into tool panel. galaxy.tools DEBUG 2013-08-20 23:34:55,861 Loaded tool id: testtoolshed.g2.bx.psu.edu/repos/peterjc/ncbi_blast_plus/blastxml_to_tabular/0.0.11, version: 0.0.11 into tool panel. galaxy.tools DEBUG 2013-08-20 23:34:55,938 Loaded tool id: testtoolshed.g2.bx.psu.edu/repos/peterjc/ncbi_blast_plus/ncbi_rpsblast_wrapper/0.0.4, version: 0.0.4 into tool panel. galaxy.tools DEBUG 2013-08-20 23:34:55,971 Loaded tool id: testtoolshed.g2.bx.psu.edu/repos/peterjc/ncbi_blast_plus/ncbi_blastdbcmd_wrapper/0.0.6, version: 0.0.6 into tool panel. --Dave B. On 8/21/13 08:08:49.000, Peter Cock wrote: On Wed, Aug 21, 2013 at 10:54 AM, Peter Cock p.j.a.c...@googlemail.com wrote: On Tue, Aug 20, 2013 at 9:15 PM, Dave Bouvier d...@bx.psu.edu wrote: Peter, I believe this was again
Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
On Wed, Aug 21, 2013 at 1:45 PM, Dave Bouvier d...@bx.psu.edu wrote: Peter, Here is the relevant log output for the installation of ncbi_blast_plus. A quick google informs me that exit code 64 means command line usage error, for what it's worth. install_and_test_repositories INFO 2013-08-20 23:34:43,270 Installing and testing revision 946749c8605f of repository ncbi_blast_plus owned by peterjc... install_repository_ncbi_blast_plus (install_and_test_tool_shed_repositories.functional.test_install_repositories.TestInstallRepository_ncbi_blast_plus) Install the repository ncbi_blast_plus from http://testtoolshed.g2.bx.psu.edu. ... tool_shed.util.shed_util_common DEBUG 2013-08-20 23:34:47,484 Adding new row (or updating an existing row) for repository 'ncbi_blast_plus' in the tool_shed_repository table, status set to 'New'. tool_shed.util.shed_util_common DEBUG 2013-08-20 23:34:47,674 Adding new row (or updating an existing row) for repository 'blast_datatypes' in the tool_shed_repository table, status set to 'New'. tool_shed.util.shed_util_common DEBUG 2013-08-20 23:34:47,886 Adding new row (or updating an existing row) for repository 'package_blast_plus_2_2_26' in the tool_shed_repository table, status set to 'New'. Warning: local() encountered an error (return code 64) while executing ' if [[ $(uname) == Linux ]]; then export FILENAME=ncbi-blast-2.2.26+-ia32-linux.tar.gz; fi if [[ $(arch) == x86_64 ]]; then export FILENAME=ncbi-blast-2.2.26+-x64-linux.tar.gz; fi if [[ $(uname) == Darwin ]]; then export FILENAME=ncbi-blast-2.2.26+-universal-macosx.tar.gz; fi echo Fetching $FILENAME if [[ $(uname) == Linux ]]; then wget ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/2.2.26/$FILENAME; fi if [[ $(uname) == Darwin ]]; then curl -O ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/2.2.26/$FILENAME; fi tar -zxvf $FILENAME echo Downloaded and decompressed ' Thanks Dave, I've made a note on this Trello card about exposing the log information of unsuccessful builds on the Tool Shed, https://trello.com/c/HVGrShnC/1042-tool-shed-should-test-installation-of-packages Can you tell me what version of Linux this is, and which shell? $ uname ... $ arch ... $ echo $0 ... That might give me a clue about the nature of the failure. My hunch is that I am using something bash specific. The simplest solution would be if you could run each of these command by hand to see if it triggers the same error: if [[ $(uname) == Linux ]]; then export FILENAME=ncbi-blast-2.2.26+-ia32-linux.tar.gz; fi if [[ $(arch) == x86_64 ]]; then export FILENAME=ncbi-blast-2.2.26+-x64-linux.tar.gz; fi if [[ $(uname) == Darwin ]]; then export FILENAME=ncbi-blast-2.2.26+-universal-macosx.tar.gz; fi echo Fetching $FILENAME if [[ $(uname) == Linux ]]; then wget ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/2.2.26/$FILENAME; fi if [[ $(uname) == Darwin ]]; then curl -O ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/2.2.26/$FILENAME; fi tar -zxvf $FILENAME echo Downloaded and decompressed Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
On Wed, Aug 14, 2013 at 11:56 AM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi Greg, I'm hoping you (or Dave) can throw a little light on why the NCBI BLAST+ nightly tests are not working again yet: http://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus/949fa0294c0d e.g. Fatal error: Exit code 127 () /bin/sh: 1: blastn: not found /bin/sh: 1: blastn: not found As of that commit, the ncbi_blast_plus repository depends on the new IUC repository package_blast_plus_2_2_26 which now depends on the IUC repository package_boost_1_53, which depends on Bjoern's package_bzlib_1_0, i.e. this chain: http://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus - http://testtoolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_26 - http://testtoolshed.g2.bx.psu.edu/view/iuc/package_boost_1_53 - http://testtoolshed.g2.bx.psu.edu/view/bgruening/package_bzlib_1_0 Sadly the (Test) Tool Shed isn't showing me any install logs, so I am hoping *you* have access to them. I've also requested individual package repositories (i.e. type Tool dependency definition) be tested nightly for installation problems: https://trello.com/c/HVGrShnC/1042-tool-shed-should-test-installation-of-packages Meanwhile, I have tried this locally - the dependencies seem to install, but the BLAST+ package says Installed, missing tool dependencies. It looks like the BLAST+ make fails, but I don't know how/if I can see this from the Galaxy Admin web-interface. I can see the logs at the command line, $ tail -n 30 blast+/2.2.26+/iuc/package_blast_plus_2_2_26/9ff9bfdcd04f/INSTALLATION.log make[1]: Leaving directory `/opt/galaxy-dist/database/tmp/tmp-toolshed-mtdutx4hN/ncbi-blast-2.2.26+-src/c++/GCC412-Debug64/build' # # cd c++ ./configure --with-boost=$BOOST_ROOT_DIR --prefix=/opt/galaxy-dist-shed-tools/blast+/2.2.26+/iuc/package_blast_plus_2_2_26/9ff9bfdcd04f make make install STDERR basename: missing operand Try `basename --help' for more information. configure: WARNING: Untested Boost version; may prove incompatible. configure: WARNING: If so, please re-run this script with the flag --without-boost. make[1]: [/opt/galaxy-dist/database/tmp/tmp-toolshed-mtdutx4hN/ncbi-blast-2.2.26+-src/c++/src/db/Makefile.in] Error 1 (ignored) make[1]: [/opt/galaxy-dist/database/tmp/tmp-toolshed-mtdutx4hN/ncbi-blast-2.2.26+-src/c++/src/db/Makefile.in] Error 1 (ignored) make[1]: [/opt/galaxy-dist/database/tmp/tmp-toolshed-mtdutx4hN/ncbi-blast-2.2.26+-src/c++/src/ctools/Makefile.in] Error 1 (ignored) make[1]: [/opt/galaxy-dist/database/tmp/tmp-toolshed-mtdutx4hN/ncbi-blast-2.2.26+-src/c++/src/ctools/Makefile.in] Error 1 (ignored) make[1]: [/opt/galaxy-dist/database/tmp/tmp-toolshed-mtdutx4hN/ncbi-blast-2.2.26+-src/c++/src/misc/Makefile.in] Error 1 (ignored) make[1]: [/opt/galaxy-dist/database/tmp/tmp-toolshed-mtdutx4hN/ncbi-blast-2.2.26+-src/c++/src/misc/Makefile.in] Error 1 (ignored) make[1]: [/opt/galaxy-dist/database/tmp/tmp-toolshed-mtdutx4hN/ncbi-blast-2.2.26+-src/c++/src/gui/Makefile.in] Error 1 (ignored) make[1]: [/opt/galaxy-dist/database/tmp/tmp-toolshed-mtdutx4hN/ncbi-blast-2.2.26+-src/c++/src/gui/Makefile.in] Error 1 (ignored) make[1]: [/opt/galaxy-dist/database/tmp/tmp-toolshed-mtdutx4hN/ncbi-blast-2.2.26+-src/c++/src/sample/Makefile.in] Error 1 (ignored) make[1]: [/opt/galaxy-dist/database/tmp/tmp-toolshed-mtdutx4hN/ncbi-blast-2.2.26+-src/c++/src/sample/Makefile.in] Error 1 (ignored) make[1]: [/opt/galaxy-dist/database/tmp/tmp-toolshed-mtdutx4hN/ncbi-blast-2.2.26+-src/c++/src/internal/Makefile.in] Error 1 (ignored) make[1]: [/opt/galaxy-dist/database/tmp/tmp-toolshed-mtdutx4hN/ncbi-blast-2.2.26+-src/c++/src/internal/Makefile.in] Error 1 (ignored) make[2]: [/opt/galaxy-dist/database/tmp/tmp-toolshed-mtdutx4hN/ncbi-blast-2.2.26+-src/c++/src/corelib/test/Makefile.in] Error 1 (ignored) make[2]: [/opt/galaxy-dist/database/tmp/tmp-toolshed-mtdutx4hN/ncbi-blast-2.2.26+-src/c++/src/corelib/test/Makefile.in] Error 1 (ignored) /bin/sh: /usr/bin/touch: No such file or directory make[3]: *** [sources.usr] Error 127 make[2]: *** [sources.usr.locked] Error 2 make[1]: *** [all_r.real] Error 5 make: *** [all] Error 2 # I just checked the BOOST bit worked by manually compiling BLAST+, $ echo 'BOOST_ROOT_DIR=/opt/galaxy-dist-shed-tools/boost/1.53.0/iuc/package_boost_1_53/a72f8efe9201/boost; export BOOST_ROOT_DIR' $ ./configure --with-boost=$BOOST_ROOT_DIR --prefix=/opt/galaxy-dist-shed-tools/blast+/2.2.26+/iuc/package_blast_plus_2_2_26/9ff9bfdcd04f ... *** CONFIGURATION SUCCESSFUL *** $ make ... /bin/sh: /usr/bin/touch: No such file or directory ... make: *** [all] Error 2 So sign of the basename error, but the touch problem persists. Fixed with: $ sudo ln -s /bin/touch /usr/bin/touch Having
Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
On Wed, Aug 7, 2013 at 3:54 PM, Peter Cock p.j.a.c...@googlemail.com wrote: On Wed, Aug 7, 2013 at 3:36 PM, Greg Von Kuster g...@bx.psu.edu wrote: Hi Peter, On Aug 7, 2013, at 10:26 AM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi Greg et al, For the NCBI BLAST+ wrappers, Nicola has suggested we split out the binaries themselves into a separate Tool Shed package, e.g. package_blast_plus_2_2_26 and similar for the later releases. https://github.com/peterjc/galaxy_blast/issues/7 I think this is a good idea. I agree. Excellent, we'll start work on that. ... OK then, I'll setup these using the current tool_dependencies.xml from the ncbi_blast_plus suite (which right now is breaking on the Tool Shed due to its copy of BOOST being updated), first: - http://testtoolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_26 Done, using just the tool_dependencies.xml file which was previously bundled within the ncbi_blast_plus Tool Shed repo: ?xml version=1.0? tool_dependency package name=blast+ version=2.2.26+ install version=1.0 actions action type=download_by_urlftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/2.2.26/ncbi-blast-2.2.26+-src.tar.gz/action action type=shell_commandcd c++ amp;amp; ./configure --prefix=$INSTALL_DIR amp;amp; make amp;amp; make install/action action type=set_environment environment_variable name=PATH action=prepend_to$INSTALL_DIR/bin/environment_variable /action /actions /install readme /readme /package /tool_dependency Now, back in ncbi_blast_plus, I have revised the repository_dependencies.xml to declare this new repository dependency (the revision tag etc is missing to mean the latest version at upload is used): ?xml version=1.0? repositories description=This requires the BLAST datatype definitions (e.g. the BLAST XML format) and the BLAST+ binaries repository name=blast_datatypes owner=devteam / repository name=package_blast_plus_2_2_26 owner=iuc / /repositories Now do I need to change the individual tools at all? e.g. tool id=ncbi_blastn_wrapper name=NCBI BLAST+ blastn version=0.0.20 descriptionSearch nucleotide database with nucleotide query sequence(s)/description !-- If job splitting is enabled, break up the query file into parts -- parallelism method=multi split_inputs=query split_mode=to_size split_size=1000 shared_inputs=subject,histdb merge_outputs=output1/parallelism requirements requirement type=binaryblastn/requirement requirement type=package version=2.2.26+blast+/requirement /requirements The package name and version still match the (relocated) tool_dependencies.xml file, so this should be OK, right? Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
Hi Peter, On Wed, Aug 7, 2013 at 3:54 PM, Peter Cock p.j.a.c...@googlemail.com wrote: On Wed, Aug 7, 2013 at 3:36 PM, Greg Von Kuster g...@bx.psu.edu wrote: Hi Peter, On Aug 7, 2013, at 10:26 AM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi Greg et al, For the NCBI BLAST+ wrappers, Nicola has suggested we split out the binaries themselves into a separate Tool Shed package, e.g. package_blast_plus_2_2_26 and similar for the later releases. https://github.com/peterjc/galaxy_blast/issues/7 I think this is a good idea. I agree. Excellent, we'll start work on that. ... OK then, I'll setup these using the current tool_dependencies.xml from the ncbi_blast_plus suite (which right now is breaking on the Tool Shed due to its copy of BOOST being updated), first: - http://testtoolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_26 Done, using just the tool_dependencies.xml file which was previously bundled within the ncbi_blast_plus Tool Shed repo: ?xml version=1.0? tool_dependency package name=blast+ version=2.2.26+ install version=1.0 actions action type=download_by_urlftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/2.2.26/ncbi-blast-2.2.26+-src.tar.gz/action action type=shell_commandcd c++ amp;amp; ./configure --prefix=$INSTALL_DIR amp;amp; make amp;amp; make install/action action type=set_environment environment_variable name=PATH action=prepend_to$INSTALL_DIR/bin/environment_variable /action /actions /install readme /readme /package /tool_dependency Now, back in ncbi_blast_plus, I have revised the repository_dependencies.xml to declare this new repository dependency (the revision tag etc is missing to mean the latest version at upload is used): Mh, I never tried that but I think that will not work. The blast datatypes are ok to be included in repository_dependency.xml but the package_blast_plus needs to be in tool_dependency.xml, imho. ?xml version=1.0? tool_dependency package name=blast+ version=2.2.26+ repository name=package_blast_plus_2_2_26 owner=iuc / /package /tool_dependency If you have that you do not need to change your tools. Hopefully I'm correct. Bjoern ?xml version=1.0? repositories description=This requires the BLAST datatype definitions (e.g. the BLAST XML format) and the BLAST+ binaries repository name=blast_datatypes owner=devteam / repository name=package_blast_plus_2_2_26 owner=iuc / /repositories Now do I need to change the individual tools at all? e.g. tool id=ncbi_blastn_wrapper name=NCBI BLAST+ blastn version=0.0.20 descriptionSearch nucleotide database with nucleotide query sequence(s)/description !-- If job splitting is enabled, break up the query file into parts -- parallelism method=multi split_inputs=query split_mode=to_size split_size=1000 shared_inputs=subject,histdb merge_outputs=output1/parallelism requirements requirement type=binaryblastn/requirement requirement type=package version=2.2.26+blast+/requirement /requirements The package name and version still match the (relocated) tool_dependencies.xml file, so this should be OK, right? Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
On Wed, Aug 7, 2013 at 5:11 PM, Bjoern Gruening bjoern.gruen...@gmail.com wrote: Hi Peter, Now, back in ncbi_blast_plus, I have revised the repository_dependencies.xml to declare this new repository dependency (the revision tag etc is missing to mean the latest version at upload is used): Mh, I never tried that but I think that will not work. The blast datatypes are ok to be included in repository_dependency.xml but the package_blast_plus needs to be in tool_dependency.xml, imho. ?xml version=1.0? tool_dependency package name=blast+ version=2.2.26+ repository name=package_blast_plus_2_2_26 owner=iuc / /package /tool_dependency If you have that you do not need to change your tools. Hopefully I'm correct. Bjoern I think you're right, certainly that is how this example works: http://toolshed.g2.bx.psu.edu/view/jjohnson/samtools_filter So I don't need this in my repository_dependencies.xml after all... Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
On Wed, Aug 7, 2013 at 5:35 PM, Peter Cock p.j.a.c...@googlemail.com wrote: On Wed, Aug 7, 2013 at 5:11 PM, Bjoern Gruening bjoern.gruen...@gmail.com wrote: Hi Peter, Now, back in ncbi_blast_plus, I have revised the repository_dependencies.xml to declare this new repository dependency (the revision tag etc is missing to mean the latest version at upload is used): Mh, I never tried that but I think that will not work. The blast datatypes are ok to be included in repository_dependency.xml but the package_blast_plus needs to be in tool_dependency.xml, imho. ?xml version=1.0? tool_dependency package name=blast+ version=2.2.26+ repository name=package_blast_plus_2_2_26 owner=iuc / /package /tool_dependency If you have that you do not need to change your tools. Hopefully I'm correct. Bjoern I think you're right, certainly that is how this example works: http://toolshed.g2.bx.psu.edu/view/jjohnson/samtools_filter So I don't need this in my repository_dependencies.xml after all... Trying this out, http://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus/949fa0294c0d Thanks Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
