Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
Peter, I did some investigating, and it turns out that the if [[ condition ]] syntax used in the ncbi tool dependency is only compatible with bash, not sh. For sh, the corresponding syntax would look like: if [ $string = value ] then echo string is equal to value fi However, it strikes me that the platform detection and download url determination could also be done using the recently introduced actions_group feature, which would bypass shell-dependent conditional syntax entirely. --Dave B. On 10/07/2013 09:45 AM, Peter Cock wrote: On Tue, Sep 24, 2013 at 10:58 AM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi all, Some good news, the strange missing /bin/sh error has gone away on the Test Tool Shed: http://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus/aecdffbc08fc The BLAST+ wrapper tests also look good on the main Tool Shed: http://toolshed.g2.bx.psu.edu/view/devteam/ncbi_blast_plus /70e7dcbf6573 Hooray! Peter P.S. Another plug for explicitly testing Tool Dependency Definition package installation on the Tool Shed: https://trello.com/c/HVGrShnC/1042-tool-shed-should-test-installation-of-packages Bad news, the strange missing /bin/sh error has come back again on both the main and Test Tool Shed :( These are the same revisions of the BLAST+ wrapper, so the only changes are in Galaxy itself, the Tool Shed itself, or the Tool Shed test environment: http://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus/aecdffbc08fc http://toolshed.g2.bx.psu.edu/view/devteam/ncbi_blast_plus /70e7dcbf6573 Installation errors - no functional tests were run for any tools in this changeset revision Tool dependencies TypeNameVersion blast+ package 2.2.26+ Error /bin/sh: 2: [[: not found /bin/sh: 3: [[: not found /bin/sh: 4: [[: not found /bin/sh: 6: [[: not found /bin/sh: 7: [[: not found tar: option requires an argument -- 'f' Try `tar --help' or `tar --usage' for more information. Regards, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] fastx_toolkit not recognizing fastq files
Hello, I've installed Galaxy on two separate machines. None of the fastx tools recognize my fastq files. They are not appearing in the drop down list on Galaxy for input options. I'm getting the same behavior on both machines. I've used the tool_shed to install fastx on one machine and I've done it manually on the other. fastx_tools work perfectly from the command line. Cheers Ian ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] fastx_toolkit not recognizing fastq files
Thanks Dannon! That was the fix. It's odd that it isn't part of the auto detect feature. -Ian On Oct 7, 2013, at 10:46 AM, Dannon Baker dannon.ba...@gmail.com wrote: If you look at the attribute details for one of your fastq files, what does Galaxy say the filetype is? You most likely need to flip it from 'fastq' to 'fastqsanger' so Galaxy will know how to handle the file, it doesn't automatically detect this. -Dannon On Mon, Oct 7, 2013 at 10:38 AM, UMD Bioinformatics bioinformatics@gmail.com wrote: Hello, I've installed Galaxy on two separate machines. None of the fastx tools recognize my fastq files. They are not appearing in the drop down list on Galaxy for input options. I'm getting the same behavior on both machines. I've used the tool_shed to install fastx on one machine and I've done it manually on the other. fastx_tools work perfectly from the command line. Cheers Ian ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] GA file format documentation
Is there any documentation on how exported workflow with '.ga' extension are structured? We are looking for the ability to create our workflows without using the workflow editor thru the web portal. Thanks Takadonet ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
On Mon, Oct 7, 2013 at 3:44 PM, Peter Cock p.j.a.c...@googlemail.com wrote: On Mon, Oct 7, 2013 at 3:37 PM, Dave Bouvier d...@bx.psu.edu wrote: Peter, I did some investigating, and it turns out that the if [[ condition ]] syntax used in the ncbi tool dependency is only compatible with bash, not sh. Does that mean the test system has switched between sh and bash and back to sh again? Is there anything written down about the expected shell(s) available for a standard Galaxy instance? For sh, the corresponding syntax would look like: if [ $string = value ] then echo string is equal to value fi However, it strikes me that the platform detection and download url determination could also be done using the recently introduced actions_group feature, which would bypass shell-dependent conditional syntax entirely. Yes, but is that in stable releases for galaxy-dist yet? In the meantime I will try that on the Test Tool Shed, based on the tool_dependencies.xml example you shared before: http://testtoolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_26/c83440a9bdfb Note there seems to me to be a lot of duplication - I would like a way to unambiguously combine multiple CPU architectures into a single action. For example, combine these: actions os=darwin architecture=x86_64 actions os=darwin architecture=i386 into: actions os=darwin architecture=x86_64,i386 or just: actions os=darwin and similarly, combine these: actions os=linux architecture=i386 actions os=linux architecture=i686 into something like: actions os=linux architecture=i386,i686 Assuming that works on the Test Tool Shed, and the code for this is already in the stable releases, I will update the main Tool Shed definition as well. Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
Peter, The platform detection code is not yet in the stable branch, but is intended to be included in the upcoming release. The automated testing framework does run the default branch, though, so the nightly tests will make use of the new definitions. --Dave B. On 10/07/2013 11:22 AM, Peter Cock wrote: On Mon, Oct 7, 2013 at 3:44 PM, Peter Cock p.j.a.c...@googlemail.com wrote: On Mon, Oct 7, 2013 at 3:37 PM, Dave Bouvier d...@bx.psu.edu wrote: Peter, I did some investigating, and it turns out that the if [[ condition ]] syntax used in the ncbi tool dependency is only compatible with bash, not sh. Does that mean the test system has switched between sh and bash and back to sh again? Is there anything written down about the expected shell(s) available for a standard Galaxy instance? For sh, the corresponding syntax would look like: if [ $string = value ] then echo string is equal to value fi However, it strikes me that the platform detection and download url determination could also be done using the recently introduced actions_group feature, which would bypass shell-dependent conditional syntax entirely. Yes, but is that in stable releases for galaxy-dist yet? In the meantime I will try that on the Test Tool Shed, based on the tool_dependencies.xml example you shared before: http://testtoolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_26/c83440a9bdfb Note there seems to me to be a lot of duplication - I would like a way to unambiguously combine multiple CPU architectures into a single action. For example, combine these: actions os=darwin architecture=x86_64 actions os=darwin architecture=i386 into: actions os=darwin architecture=x86_64,i386 or just: actions os=darwin and similarly, combine these: actions os=linux architecture=i386 actions os=linux architecture=i686 into something like: actions os=linux architecture=i386,i686 Assuming that works on the Test Tool Shed, and the code for this is already in the stable releases, I will update the main Tool Shed definition as well. Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
actions_group is neither in stable nor in default branch of galaxy-dist repository, it's present only in galaxy-central. Nicola Il giorno lun, 07/10/2013 alle 11.24 -0400, Dave Bouvier ha scritto: Peter, The platform detection code is not yet in the stable branch, but is intended to be included in the upcoming release. The automated testing framework does run the default branch, though, so the nightly tests will make use of the new definitions. --Dave B. On 10/07/2013 11:22 AM, Peter Cock wrote: On Mon, Oct 7, 2013 at 3:44 PM, Peter Cock p.j.a.c...@googlemail.com wrote: On Mon, Oct 7, 2013 at 3:37 PM, Dave Bouvier d...@bx.psu.edu wrote: Peter, I did some investigating, and it turns out that the if [[ condition ]] syntax used in the ncbi tool dependency is only compatible with bash, not sh. Does that mean the test system has switched between sh and bash and back to sh again? Is there anything written down about the expected shell(s) available for a standard Galaxy instance? For sh, the corresponding syntax would look like: if [ $string = value ] then echo string is equal to value fi However, it strikes me that the platform detection and download url determination could also be done using the recently introduced actions_group feature, which would bypass shell-dependent conditional syntax entirely. Yes, but is that in stable releases for galaxy-dist yet? In the meantime I will try that on the Test Tool Shed, based on the tool_dependencies.xml example you shared before: http://testtoolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_26/c83440a9bdfb Note there seems to me to be a lot of duplication - I would like a way to unambiguously combine multiple CPU architectures into a single action. For example, combine these: actions os=darwin architecture=x86_64 actions os=darwin architecture=i386 into: actions os=darwin architecture=x86_64,i386 or just: actions os=darwin and similarly, combine these: actions os=linux architecture=i386 actions os=linux architecture=i686 into something like: actions os=linux architecture=i386,i686 Assuming that works on the Test Tool Shed, and the code for this is already in the stable releases, I will update the main Tool Shed definition as well. Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
On Mon, Oct 7, 2013 at 10:22 AM, Peter Cock p.j.a.c...@googlemail.com wrote: On Mon, Oct 7, 2013 at 3:44 PM, Peter Cock p.j.a.c...@googlemail.com wrote: On Mon, Oct 7, 2013 at 3:37 PM, Dave Bouvier d...@bx.psu.edu wrote: Peter, I did some investigating, and it turns out that the if [[ condition ]] syntax used in the ncbi tool dependency is only compatible with bash, not sh. Does that mean the test system has switched between sh and bash and back to sh again? Is there anything written down about the expected shell(s) available for a standard Galaxy instance? I don't think it is documented anywhere and there has been some disagreement on this in the past. I think we should do our best to reach some sort of conclusion on this and document it in the following card: https://trello.com/c/7VTlX9rD My own preference is that we specify at least /bin/sh and /bin/bash are available before utilizing the tool shed. Is there an objection to this from any corner? Is there realistically a system that Galaxy should support that will not have /bin/bash available? I know there is some difference in bash behavior between platforms, but that is hopefully minimal and we can address inconsistencies one at a time. Thanks, -John For sh, the corresponding syntax would look like: if [ $string = value ] then echo string is equal to value fi However, it strikes me that the platform detection and download url determination could also be done using the recently introduced actions_group feature, which would bypass shell-dependent conditional syntax entirely. Yes, but is that in stable releases for galaxy-dist yet? In the meantime I will try that on the Test Tool Shed, based on the tool_dependencies.xml example you shared before: http://testtoolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_26/c83440a9bdfb Note there seems to me to be a lot of duplication - I would like a way to unambiguously combine multiple CPU architectures into a single action. For example, combine these: actions os=darwin architecture=x86_64 actions os=darwin architecture=i386 into: actions os=darwin architecture=x86_64,i386 or just: actions os=darwin and similarly, combine these: actions os=linux architecture=i386 actions os=linux architecture=i686 into something like: actions os=linux architecture=i386,i686 Assuming that works on the Test Tool Shed, and the code for this is already in the stable releases, I will update the main Tool Shed definition as well. Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
On Mon, Oct 7, 2013 at 4:45 PM, Nicola Soranzo sora...@crs4.it wrote: actions_group is neither in stable nor in default branch of galaxy-dist repository, it's present only in galaxy-central. Nicola i.e. for now only they can only be used and tested on the Test Tool Shed? It would be helpful if the bash vs sh problem on the (Test) Tool Shed cluster could be addressed in the short term (and as John points out, documented). Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] GA file format documentation
Nothing formal. The .ga extension is simply the workflow dictionary exported as .json, if that helps -- there's no extra magic to the format. You may want to look at the workflow API, some users have had success integrating workflows created on the fly using it. -Dannon On Mon, Oct 7, 2013 at 11:19 AM, Philip Mabon philipma...@gmail.com wrote: Is there any documentation on how exported workflow with '.ga' extension are structured? We are looking for the ability to create our workflows without using the workflow editor thru the web portal. Thanks Takadonet ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] GA file format documentation
API allows for importing and exporting of workflows but nothing for creating a new workflow. I will dig and see if anyone has published or presented on their internal libraries that create workflows on the fly. Thanks Takadonet On Mon, Oct 7, 2013 at 11:18 AM, Dannon Baker dannon.ba...@gmail.comwrote: Nothing formal. The .ga extension is simply the workflow dictionary exported as .json, if that helps -- there's no extra magic to the format. You may want to look at the workflow API, some users have had success integrating workflows created on the fly using it. -Dannon On Mon, Oct 7, 2013 at 11:19 AM, Philip Mabon philipma...@gmail.comwrote: Is there any documentation on how exported workflow with '.ga' extension are structured? We are looking for the ability to create our workflows without using the workflow editor thru the web portal. Thanks Takadonet ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Adding a tool to Galaxy Main Server
Hi everybody, We are going to add our gene prediction tool to Galaxy Main server, anybody know whom we should contact? Thanks -- Hamid Reza Hassanzadeh, PhD Student Graduate Research Assistant, Bioinformatics Lab, Center for Bioinformatics and Computational Genomics Joint Georgia Tech and Emory Wallace H Coulter Department of Biomedical Engineering, Department of Computer Science at Georgia Institute of Technology office: KACB 1343 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] GA file format documentation
Right, so you'd assemble the json clientside and use /api/workflows/upload to create it. See the import_new_workflow method for more details On Mon, Oct 7, 2013 at 12:31 PM, Philip Mabon philipma...@gmail.com wrote: API allows for importing and exporting of workflows but nothing for creating a new workflow. I will dig and see if anyone has published or presented on their internal libraries that create workflows on the fly. Thanks Takadonet On Mon, Oct 7, 2013 at 11:18 AM, Dannon Baker dannon.ba...@gmail.comwrote: Nothing formal. The .ga extension is simply the workflow dictionary exported as .json, if that helps -- there's no extra magic to the format. You may want to look at the workflow API, some users have had success integrating workflows created on the fly using it. -Dannon On Mon, Oct 7, 2013 at 11:19 AM, Philip Mabon philipma...@gmail.comwrote: Is there any documentation on how exported workflow with '.ga' extension are structured? We are looking for the ability to create our workflows without using the workflow editor thru the web portal. Thanks Takadonet ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Dynamic job runner not being dispatched for a tool
Hi List,
I'm a bit baffled by a new tool I've tried to add to Galaxy. The job runner
does not dispatch it correctly.
I added the tool via the toolshed, set up the executable on the cluster side,
and attempted to hook it up to the dynamic job runner. The destination always
ended up set as local.
Works fine if the default runner is pbs, but if it's local by default, it will
only dispatch locally.
Attempting to work backwards, I copied the wrapper to a local tool directory,
and pared everything down to a minimum to run. There must be something missing.
My trinity_all dynamic definition works fine, Newbler doesn't. Is there
anything obvious here?
~tool_conf.xml
?xml version=1.0?
toolbox
section name=Assembly id=assembly
tool file=sr_assembly/runAssembly.xml /
tool file=ngs_rna/trinity_all.xml /
/section
!-- I have to have two sections in order for any sections to show up at all --
section name=Send Data id=send
tool file=data_destination/epigraph.xml /
tool file=data_destination/epigraph_test.xml /
tool file=genomespace/genomespace_exporter.xml /
/section
/toolbox
~tools/sr_assembly/runAssembly.xml
tool id=runAssembly name=runAssembly version=1.0.1
descriptionDe novo assembly of Roche/454 reads using Newbler/description
command
runAssembly $sanger_input -o $newbler_metrics
/command
inputs
!-- READSEQ INFILES --
param name=sanger_input type=data format=fasta,fastqsanger label=SE
Fasta/Fastq file/
!-- for walltime control --
param name=walltime type=select label=How long will your job need
option selected=true value=18 hrs/option
option value=4848 hrs/option
/param
/inputs
outputs
data name=newbler_metrics format=txt /
/outputs
/tool
~job_conf.xml
?xml version=1.0?
job_conf
plugins workers=4
plugin id=local type=runner
load=galaxy.jobs.runners.local:LocalJobRunner /
plugin id=pbs type=runner load=galaxy.jobs.runners.pbs:PBSJobRunner
/
/plugins
handlers default=handlers
handler id=handler0 tags=handlers/
/handlers
destinations default=local
destination id=local runner=local/
destination id=pbs_normal runner=pbs tags=mycluster
param
id=Resource_Listnodes=1:ppn=4,vmem=32gb,walltime=24:00:00/param
/destination
destination id=newbler_d runner=dynamic
param id=typepython/param
param id=functionnewbler/param
/destination
destination id=trinity_d runner=dynamic
param id=typepython/param
param id=functiontrinity_all/param
/destination
/destinations
tools
tool id=runAssembly destination=newbler_d/
tool id=trinity_all destination=trinity_d/
/tools
limits
/limits
/job_conf
~lib/galaxy/jobs/rules/dynamicjobs.py
import os
from galaxy.jobs import JobDestination
def newbler(job):
incoming = dict( [ ( p.name, p.value ) for p in job.parameters ] )
walltime_para = incoming[walltime]
walltime = int(walltime_para.replace(\,))
params = {}
string = vmem=500gb,nodes=1:ppn=32,walltime=%d:00:00%walltime
params[Resource_List] = string
return JobDestination(runner=pbs, params=params)
def trinity_all(job):
incoming = dict( [ ( p.name, p.value ) for p in job.parameters ] )
walltime_para = incoming[walltime]
walltime = int(walltime_para.replace(\,))
params = {}
string = vmem=500gb,nodes=1:ppn=32,walltime=%d:00:00%walltime
params[Resource_List] = string
return JobDestination(runner=pbs, params=params)
I re-installed the Roche 454 toolsuite from the toolshed once more to see if it
just needed a kick. I copied the tool id into my job_conf file to make sure I
made no mistakes on the id. I tried to send it to dynamic job runner and to pbs
runner. I pulled and merged this galaxy about a month ago, so it should be
mostly up to date - but I've had so many issues that I may have to just
re-install from scratch.
Thanks a ton for any advice,
Carrie Ganote
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Dynamic job runner not being dispatched for a tool
I can look into this some more, but I have a couple quick questions.
Can you try without the tags attributes, I am not sure what is going
to happen since they do not match each other.
Also, have you tried this fix, it looks like it could potentially be
the same problem.
https://bitbucket.org/galaxy/galaxy-central/commits/0b955e54451cf22a1a7c8fa0f92beb9e375f1f1e
-John
On Mon, Oct 7, 2013 at 12:46 PM, Ganote, Carrie L cgan...@iu.edu wrote:
Hi List,
I'm a bit baffled by a new tool I've tried to add to Galaxy. The job runner
does not dispatch it correctly.
I added the tool via the toolshed, set up the executable on the cluster
side, and attempted to hook it up to the dynamic job runner. The destination
always ended up set as local.
Works fine if the default runner is pbs, but if it's local by default, it
will only dispatch locally.
Attempting to work backwards, I copied the wrapper to a local tool
directory, and pared everything down to a minimum to run. There must be
something missing. My trinity_all dynamic definition works fine, Newbler
doesn't. Is there anything obvious here?
~tool_conf.xml
?xml version=1.0?
toolbox
section name=Assembly id=assembly
tool file=sr_assembly/runAssembly.xml /
tool file=ngs_rna/trinity_all.xml /
/section
!-- I have to have two sections in order for any sections to show up at all
--
section name=Send Data id=send
tool file=data_destination/epigraph.xml /
tool file=data_destination/epigraph_test.xml /
tool file=genomespace/genomespace_exporter.xml /
/section
/toolbox
~tools/sr_assembly/runAssembly.xml
tool id=runAssembly name=runAssembly version=1.0.1
descriptionDe novo assembly of Roche/454 reads using Newbler/description
command
runAssembly $sanger_input -o $newbler_metrics
/command
inputs
!-- READSEQ INFILES --
param name=sanger_input type=data format=fasta,fastqsanger
label=SE Fasta/Fastq file/
!-- for walltime control --
param name=walltime type=select label=How long will your job
need
option selected=true value=18 hrs/option
option value=4848 hrs/option
/param
/inputs
outputs
data name=newbler_metrics format=txt /
/outputs
/tool
~job_conf.xml
?xml version=1.0?
job_conf
plugins workers=4
plugin id=local type=runner
load=galaxy.jobs.runners.local:LocalJobRunner /
plugin id=pbs type=runner
load=galaxy.jobs.runners.pbs:PBSJobRunner /
/plugins
handlers default=handlers
handler id=handler0 tags=handlers/
/handlers
destinations default=local
destination id=local runner=local/
destination id=pbs_normal runner=pbs tags=mycluster
param
id=Resource_Listnodes=1:ppn=4,vmem=32gb,walltime=24:00:00/param
/destination
destination id=newbler_d runner=dynamic
param id=typepython/param
param id=functionnewbler/param
/destination
destination id=trinity_d runner=dynamic
param id=typepython/param
param id=functiontrinity_all/param
/destination
/destinations
tools
tool id=runAssembly destination=newbler_d/
tool id=trinity_all destination=trinity_d/
/tools
limits
/limits
/job_conf
~lib/galaxy/jobs/rules/dynamicjobs.py
import os
from galaxy.jobs import JobDestination
def newbler(job):
incoming = dict( [ ( p.name, p.value ) for p in job.parameters ] )
walltime_para = incoming[walltime]
walltime = int(walltime_para.replace(\,))
params = {}
string = vmem=500gb,nodes=1:ppn=32,walltime=%d:00:00%walltime
params[Resource_List] = string
return JobDestination(runner=pbs, params=params)
def trinity_all(job):
incoming = dict( [ ( p.name, p.value ) for p in job.parameters ] )
walltime_para = incoming[walltime]
walltime = int(walltime_para.replace(\,))
params = {}
string = vmem=500gb,nodes=1:ppn=32,walltime=%d:00:00%walltime
params[Resource_List] = string
return JobDestination(runner=pbs, params=params)
I re-installed the Roche 454 toolsuite from the toolshed once more to see if
it just needed a kick. I copied the tool id into my job_conf file to make
sure I made no mistakes on the id. I tried to send it to dynamic job runner
and to pbs runner. I pulled and merged this galaxy about a month ago, so it
should be mostly up to date - but I've had so many issues that I may have to
just re-install from scratch.
Thanks a ton for any advice,
Carrie Ganote
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Dynamic job runner not being dispatched for a tool
It might be due to this bug:http://dev.list.galaxyproject.org/Limit-number-of-Jobs-to-Trinity-tt4661790.html#a4661874
Ilya
From: Ganote, Carrie L cgan...@iu.edu
Date: Monday, October 7, 2013 1:46 PM
To: galaxy-dev@lists.bx.psu.edu galaxy-dev@lists.bx.psu.edu
Subject: [galaxy-dev] Dynamic job runner not being dispatched for a tool
Hi List,
I'm a bit baffled by a new tool I've tried to add to Galaxy. The job runner does not dispatch it correctly.
I added the tool via the toolshed, set up the executable on the cluster side, and attempted to hook it up to the dynamic job runner. The destination always ended up set as local.
Works fine if the default runner is pbs, but if it's local by default, it will only dispatch locally.
Attempting to work backwards, I copied the wrapper to a local tool directory, and pared everything down to a minimum to run. There must be something missing. My trinity_all dynamic definition works fine, Newbler doesn't. Is there anything obvious here?
~tool_conf.xml
?xml version=1.0?
toolbox
section name=Assembly id=assembly
tool file=sr_assembly/runAssembly.xml /
tool file=ngs_rna/trinity_all.xml /
/section
!-- I have to have two sections in order for any sections to show up at all --
section name=Send Data id=send
tool file=data_destination/epigraph.xml /
tool file=data_destination/epigraph_test.xml /
tool file=genomespace/genomespace_exporter.xml /
/section
/toolbox
~tools/sr_assembly/runAssembly.xml
tool id=runAssembly name=runAssembly version=1.0.1
descriptionDe novo assembly of Roche/454 reads using Newbler/description
command
runAssembly $sanger_input -o $newbler_metrics
/command
inputs
!-- READSEQ INFILES --
param name=sanger_input type=data format=fasta,fastqsanger label=SE Fasta/Fastq file/
!-- for walltime control --
param name=walltime type=select label=How long will your job need
option selected=true value=18 hrs/option
option value=4848 hrs/option
/param
/inputs
outputs
data name=newbler_metrics format=txt /
/outputs
/tool
~job_conf.xml
?xml version=1.0?
job_conf
plugins workers=4
plugin id=local type=runner load=galaxy.jobs.runners.local:LocalJobRunner /
plugin id=pbs type=runner load=galaxy.jobs.runners.pbs:PBSJobRunner /
/plugins
handlers default=handlers
handler id=handler0 tags=handlers/
/handlers
destinations default=local
destination id=local runner=local/
destination id=pbs_normal runner=pbs tags=mycluster
param id=Resource_Listnodes=1:ppn=4,vmem=32gb,walltime=24:00:00/param
/destination
destination id=newbler_d runner=dynamic
param id=typepython/param
param id=functionnewbler/param
/destination
destination id=trinity_d runner=dynamic
param id=typepython/param
param id=functiontrinity_all/param
/destination
/destinations
tools
tool id=runAssembly destination=newbler_d/
tool id=trinity_all destination=trinity_d/
/tools
limits
/limits
/job_conf
~lib/galaxy/jobs/rules/dynamicjobs.py
import os
from galaxy.jobs import JobDestination
def newbler(job):
incoming = dict( [ ( p.name, p.value ) for p in job.parameters ] )
walltime_para = incoming[walltime]
walltime = int(walltime_para.replace(\,))
params = {}
string = vmem=500gb,nodes=1:ppn=32,walltime=%d:00:00%walltime
params[Resource_List] = string
return JobDestination(runner=pbs, params=params)
def trinity_all(job):
incoming = dict( [ ( p.name, p.value ) for p in job.parameters ] )
walltime_para = incoming[walltime]
walltime = int(walltime_para.replace(\,))
params = {}
string = vmem=500gb,nodes=1:ppn=32,walltime=%d:00:00%walltime
params[Resource_List] = string
return JobDestination(runner=pbs, params=params)
I re-installed the Roche 454 toolsuite from the toolshed once more to see if it just needed a kick. I copied the tool id into my job_conf file to make sure I made no mistakes on the id. I tried to send it to dynamic job runner and to pbs runner. I pulled and
merged this galaxy about a month ago, so it should be mostly up to date - but I've had so many issues that I may have to just re-install from scratch.
Thanks a ton for any advice,
Carrie Ganote
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Dynamic job runner not being dispatched for a tool
Hi John,
Thank you and Ilya for pointing this out. I was hitting my head on this for 2
days, convinced that I had misconfigured something. The tags didn't seem to
have any effect, so I tried adding the .lower to the tool.get call. That did
the trick.
Thanks again, so much, for the help! You saved me a re-install and quite a bit
of sanity.
Sincerely,
Carrie Ganote
From: jmchil...@gmail.com [jmchil...@gmail.com] on behalf of John Chilton
[chil...@msi.umn.edu]
Sent: Monday, October 07, 2013 2:02 PM
To: Ganote, Carrie L
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Dynamic job runner not being dispatched for a tool
I can look into this some more, but I have a couple quick questions.
Can you try without the tags attributes, I am not sure what is going
to happen since they do not match each other.
Also, have you tried this fix, it looks like it could potentially be
the same problem.
https://bitbucket.org/galaxy/galaxy-central/commits/0b955e54451cf22a1a7c8fa0f92beb9e375f1f1e
-John
On Mon, Oct 7, 2013 at 12:46 PM, Ganote, Carrie L cgan...@iu.edu wrote:
Hi List,
I'm a bit baffled by a new tool I've tried to add to Galaxy. The job runner
does not dispatch it correctly.
I added the tool via the toolshed, set up the executable on the cluster
side, and attempted to hook it up to the dynamic job runner. The destination
always ended up set as local.
Works fine if the default runner is pbs, but if it's local by default, it
will only dispatch locally.
Attempting to work backwards, I copied the wrapper to a local tool
directory, and pared everything down to a minimum to run. There must be
something missing. My trinity_all dynamic definition works fine, Newbler
doesn't. Is there anything obvious here?
~tool_conf.xml
?xml version=1.0?
toolbox
section name=Assembly id=assembly
tool file=sr_assembly/runAssembly.xml /
tool file=ngs_rna/trinity_all.xml /
/section
!-- I have to have two sections in order for any sections to show up at all
--
section name=Send Data id=send
tool file=data_destination/epigraph.xml /
tool file=data_destination/epigraph_test.xml /
tool file=genomespace/genomespace_exporter.xml /
/section
/toolbox
~tools/sr_assembly/runAssembly.xml
tool id=runAssembly name=runAssembly version=1.0.1
descriptionDe novo assembly of Roche/454 reads using Newbler/description
command
runAssembly $sanger_input -o $newbler_metrics
/command
inputs
!-- READSEQ INFILES --
param name=sanger_input type=data format=fasta,fastqsanger
label=SE Fasta/Fastq file/
!-- for walltime control --
param name=walltime type=select label=How long will your job
need
option selected=true value=18 hrs/option
option value=4848 hrs/option
/param
/inputs
outputs
data name=newbler_metrics format=txt /
/outputs
/tool
~job_conf.xml
?xml version=1.0?
job_conf
plugins workers=4
plugin id=local type=runner
load=galaxy.jobs.runners.local:LocalJobRunner /
plugin id=pbs type=runner
load=galaxy.jobs.runners.pbs:PBSJobRunner /
/plugins
handlers default=handlers
handler id=handler0 tags=handlers/
/handlers
destinations default=local
destination id=local runner=local/
destination id=pbs_normal runner=pbs tags=mycluster
param
id=Resource_Listnodes=1:ppn=4,vmem=32gb,walltime=24:00:00/param
/destination
destination id=newbler_d runner=dynamic
param id=typepython/param
param id=functionnewbler/param
/destination
destination id=trinity_d runner=dynamic
param id=typepython/param
param id=functiontrinity_all/param
/destination
/destinations
tools
tool id=runAssembly destination=newbler_d/
tool id=trinity_all destination=trinity_d/
/tools
limits
/limits
/job_conf
~lib/galaxy/jobs/rules/dynamicjobs.py
import os
from galaxy.jobs import JobDestination
def newbler(job):
incoming = dict( [ ( p.name, p.value ) for p in job.parameters ] )
walltime_para = incoming[walltime]
walltime = int(walltime_para.replace(\,))
params = {}
string = vmem=500gb,nodes=1:ppn=32,walltime=%d:00:00%walltime
params[Resource_List] = string
return JobDestination(runner=pbs, params=params)
def trinity_all(job):
incoming = dict( [ ( p.name, p.value ) for p in job.parameters ] )
walltime_para = incoming[walltime]
walltime = int(walltime_para.replace(\,))
params = {}
string = vmem=500gb,nodes=1:ppn=32,walltime=%d:00:00%walltime
params[Resource_List] = string
return JobDestination(runner=pbs, params=params)
I re-installed the Roche 454 toolsuite from the toolshed once more to see if
it just needed a kick. I copied the tool id into my job_conf file to make
sure I made no mistakes on the id. I tried to send it
Re: [galaxy-dev] Missing test results on (Test) Tool Shed
Peter, Thanks for the information, it uncovered a problem with the testing framework that has been resolved in 10817:6b0c3592dde1. The listed repositories should start showing useful results next time the nightly tests run. --Dave B. On 10/04/2013 06:19 AM, Peter Cock wrote: On Fri, Oct 4, 2013 at 11:11 AM, Peter Cock p.j.a.c...@googlemail.com wrote: On Thu, Sep 26, 2013 at 3:29 PM, Dave Bouvier d...@bx.psu.edu wrote: Ross, Peter, Unfortunately, there are some issues with the automated testing framework that I am working on resolving. I've created a Trello card to track my progress: https://trello.com/c/x8mx7HRB --Dave B. Any joy with that from fabric.api import env issue? Meanwhile the set of tools with missing test failures seems to change from day to day, right now the following are listed under Latest revision: failing tool tests yet have not test results: http://testtoolshed.g2.bx.psu.edu/view/peterjc/clinod/0d39248a7a9f (expected to pass) http://testtoolshed.g2.bx.psu.edu/view/peterjc/effectivet3/1e681e19e35c (expected to pass) http://testtoolshed.g2.bx.psu.edu/view/peterjc/nlstradamus/81cac85fc286 (expected to pass) Peter There's a similar set on the main Tool Shed, where I should have the same version of those three tools, which may be interesting for tracing the problem(s): http://toolshed.g2.bx.psu.edu/view/peterjc/clinod (expected to pass) http://toolshed.g2.bx.psu.edu/view/peterjc/effectivet3 (expected to pass) http://toolshed.g2.bx.psu.edu/view/peterjc/nlstradamus (expected to pass) Also missing test results: http://toolshed.g2.bx.psu.edu/view/peterjc/seq_primer_clip/8c02a91a8680 (this revision is expected to fail if Biopython not present, see other thread about packaging NumPy and its dependencies) Regards, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
Am Montag, den 07.10.2013, 10:36 -0500 schrieb John Chilton: On Mon, Oct 7, 2013 at 10:22 AM, Peter Cock p.j.a.c...@googlemail.com wrote: On Mon, Oct 7, 2013 at 3:44 PM, Peter Cock p.j.a.c...@googlemail.com wrote: On Mon, Oct 7, 2013 at 3:37 PM, Dave Bouvier d...@bx.psu.edu wrote: Peter, I did some investigating, and it turns out that the if [[ condition ]] syntax used in the ncbi tool dependency is only compatible with bash, not sh. Does that mean the test system has switched between sh and bash and back to sh again? Is there anything written down about the expected shell(s) available for a standard Galaxy instance? I don't think it is documented anywhere and there has been some disagreement on this in the past. I think we should do our best to reach some sort of conclusion on this and document it in the following card: https://trello.com/c/7VTlX9rD My own preference is that we specify at least /bin/sh and /bin/bash are available before utilizing the tool shed. Is there an objection to this from any corner? Is there realistically a system that Galaxy should support that will not have /bin/bash available? I know there is some difference in bash behavior between platforms, but that is hopefully minimal and we can address inconsistencies one at a time. I think we can and should assume /bin/bash on every system. We also should finally write down the dependency list. We talked a lot about it and we collected a lot of system libraries and dependencies the TS and all tools can depend on. We just need to write it down. I will comment on that trello card! Bjoern Thanks, -John For sh, the corresponding syntax would look like: if [ $string = value ] then echo string is equal to value fi However, it strikes me that the platform detection and download url determination could also be done using the recently introduced actions_group feature, which would bypass shell-dependent conditional syntax entirely. Yes, but is that in stable releases for galaxy-dist yet? In the meantime I will try that on the Test Tool Shed, based on the tool_dependencies.xml example you shared before: http://testtoolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_26/c83440a9bdfb Note there seems to me to be a lot of duplication - I would like a way to unambiguously combine multiple CPU architectures into a single action. For example, combine these: actions os=darwin architecture=x86_64 actions os=darwin architecture=i386 into: actions os=darwin architecture=x86_64,i386 or just: actions os=darwin and similarly, combine these: actions os=linux architecture=i386 actions os=linux architecture=i686 into something like: actions os=linux architecture=i386,i686 Assuming that works on the Test Tool Shed, and the code for this is already in the stable releases, I will update the main Tool Shed definition as well. Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Test Toolshed Biopython package dependency NumPy
Am Montag, den 07.10.2013, 14:07 +0100 schrieb Peter Cock: On Tue, Oct 1, 2013 at 9:18 PM, Dave Bouvier d...@bx.psu.edu wrote: I've done some investigating, and it looks like there is an irregularity with the installation of package_numpy_1_7, and somehow package_biopython_1_62 is not finding the (successfully) installed numpy dependency. I duplicated the dependency chain for seq_select_by_id on my local tool shed, and got the following error for biopython's tool dependency record when installing seq_select_by_id into my local Galaxy instance: export PYTHONPATH=$PYTHONPATH:/var/galaxy/tool_dependencies/biopython/1.62/devteam/package_biopython_1_62/c3c8665cb0af/lib/python export PATH=$PATH:$PATH_NUMPY export PYTHONPATH=$PYTHONPATH:$PYTHONPATH_NUMPY python setup.py install --install-lib /var/galaxy/tool_dependencies/biopython/1.62/devteam/package_biopython_1_62/c3c8665cb0af/lib/python STDOUT running install Numerical Python (NumPy) is not installed. This package is required for many Biopython features. Please install it before you install Biopython. You can install Biopython anyway, but anything dependent on NumPy will not work. If you do this, and later install NumPy, you should then re-install Biopython. You can find NumPy at http://numpy.scipy.org Note that when I installed seq_select_by_id, numpy installed correctly, and PYTHONPATH was populated with the right values. --Dave B. Hi Dave, Bjeorn, Right now NumPy isn't installing at all on the Test Tool Shed. Dave - is is possible that the system was updated and the Fortran compiler removed? e.g. http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_primer_clip/834cf6618a14 reports: Installation errors - no functional tests were run for any tools in this changeset revision Tool dependencies TypeNameVersion lapack package 3.4.2 Error CMake Error: your Fortran compiler: CMAKE_Fortran_COMPILER-NOTFOUND was not found. Please set CMAKE_Fortran_COMPILER to a valid compiler path or name. CMake Error at /usr/share/cmake-2.8/Modules/CMakeFortranInformation.cmake:27 (get_filename_component): get_filename_component called with incorrect number of arguments Call Stack (most recent call first): CMakeLists.txt:2 (project) CMake Error: CMAKE_Fortran_COMPILER not set, after EnableLanguage CMake Error: Internal CMake error, TryCompile configure of cmake failed CMake Error at /usr/share/cmake-2.8/Modules/CMakeFortranInformation.cmake:27 (get_filename_component): get_filename_component called with incorrect number of arguments Call Stack (most recent call first): CMakeLists.txt:2 (project) CMake Error: CMAKE_Fortran_COMPILER not set, after EnableLanguage CMake Error: Internal CMake error, TryCompile configure of cmake failed CMake Error at /usr/share/cmake-2.8/Modules/CMakeFortranInformation.cmake:27 (get_filename_component): get_filename_component called with incorrect number of arguments Call Stack (most recent call first): CMakeLists.txt:2 (project) CMake Error: CMAKE_Fortran_COMPILER not set, after EnableLanguage CMake Error: Internal CMake error, TryCompile configure of cmake failed CMake Error at /usr/share/cmake-2.8/Modules/CMakeFortranInformation.cmake:27 (get_filename_component): get_filename_component called with incorrect number of arguments Call Stack (most recent call first): CMakeLists.txt:2 (project) CMake Error: CMAKE_Fortran_COMPILER not set, after EnableLanguage CMake Error: Internal CMake error, TryCompile configure of cmake failed CMake Error at /usr/share/cmake-2.8/Modules/CMakeFortranInformation.cmake:27 (get_filename_component): get_filename_component called with incorrect number of arguments Call Stack (most recent call first): CMakeLists.txt:2 (project) CMake Error: CMAKE_Fortran_COMPILER not set, after EnableLanguage CMake Error: Internal CMake error, TryCompile configure of cmake failed Since this seems to be breaking lapack, perhaps we should further simplify package_numpy_1_7 not to use lapack or ATLAS? What do you think Bjoern? I'm a little bit sceptic to remove features only because the Test Utility can not handle it. On the other hand I understand that numpy should just work and that your test failing since month because of my dependencies. I will get rid of it tomorrow. As a result we can setup_virtualenv and install numpy from pip directly I think. Is that ok for you? Cheers, Bjoern http://toolshed.g2.bx.psu.edu/view/iuc/package_numpy_1_7 http://testtoolshed.g2.bx.psu.edu/view/iuc/package_numpy_1_7 Regards, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at:
Re: [galaxy-dev] Adding a tool to Galaxy Main Server
Hello, New tools go into the Tool Shed, links with instructions are here: http://usegalaxy.org/toolshed http://wiki.galaxyproject.org/Tool%20Shed We tweet new tool additions to let the community know about them, info and recent tweets are here: http://wiki.galaxyproject.org/Galaxy%20on%20Twitter If you are working with someone about incorporating a tool onto the public Main server, contacting them direct is generally the best path. Alternatively, after the tool is in the Tool Shed, you can write to us at outre...@galaxyproject.org to ask about adding it. No promises! Many tools in the Tool Shed are specifically intended for use in local/cloud/slipstream Galaxy instances. Next time, sending to just one list is best, these all reach the same Galaxy core team. Galaxy-dev is best for reaching our community that focuses on tool development. http://wiki.galaxyproject.org/Support#Mailing_Lists Best, Jen Galaxy team On 10/7/13 9:36 AM, Hamid Reza Hassanzadeh wrote: Hi everybody, We are going to add our gene prediction tool to Galaxy Main server, anybody know whom we should contact? Thanks -- Hamid Reza Hassanzadeh, PhD Student Graduate Research Assistant, Bioinformatics Lab, Center for Bioinformatics and Computational Genomics Joint Georgia Tech and Emory Wallace H Coulter Department of Biomedical Engineering, Department of Computer Science at Georgia Institute of Technology office: KACB 1343 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ -- Jennifer Hillman-Jackson http://galaxyproject.org ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
