Re: [galaxy-dev] Tool refuses to appear in Toolshed package contents
On Thu, Oct 24, 2013 at 12:09 PM, Lukasse, Pieter pieter.luka...@wur.nl wrote: Hi , I have a toolshed repository where I expect 5 tools to appear in the “Contents of this repository” section but one of them fails to appear in the list (see screenshot). Where can I find out what is going wrong? Thanks, Pieter. Hi Pieter, The first thing to confirm is if the missing tool a valid XML file? Next, does it work in your Galaxy instance (if installed manually). Now if this problem is only in the Tool Shed, which toolshed is it in? Your own local tool shed, the main Penn State public Tool Shed, the Penn State Test Tool Shed, or somewhere else? If it is your own tool shed, you should be able to look at its logging for clues. If it is a public tool shed, please share the URL for the repository. Regards, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed Tests failing with sniffer/ftype problem
On Fri, Oct 11, 2013 at 10:01 AM, Peter Cock p.j.a.c...@googlemail.com wrote: On Thu, Aug 1, 2013 at 8:43 PM, Dave Bouvier d...@bx.psu.edu wrote: Peter, As of 10290:d34d6d1da813, nsltradamus and effectivet3 should start returning valid test results. As for seq_rename, ... --Dave B. Hi Dave, That fix is on galaxy-central and the Test Tool Shed, and my tests have been passing there :) However, the tests are currently still failing on the main Tool Shed: http://toolshed.g2.bx.psu.edu/view/peterjc/nlstradamus http://toolshed.g2.bx.psu.edu/view/peterjc/effectivet3 The commit for your fix is on the galaxy-central default branch, https://bitbucket.org/galaxy/galaxy-dist/commits/d34d6d1da813 https://bitbucket.org/galaxy/galaxy-dist/src/default/lib/galaxy/datatypes/binary.py The fix is not on the stable branch: https://bitbucket.org/galaxy/galaxy-dist/src/stable/lib/galaxy/datatypes/binary.py Nor is the fix on the next-stable branch, https://bitbucket.org/galaxy/galaxy-dist/src/next-stable/lib/galaxy/datatypes/binary.py I'm not quite sure how fixes like this are propagated from galaxy-central to the stable branches on galaxy-dist, but has this been forgotten about? Thanks, Peter It does appear to have been forgotten about - now that the main Tool Shed is running next-stable in preparation for tentative 4 Nov release, I am seeing these sniffer related test failures once again :( http://toolshed.g2.bx.psu.edu/view/peterjc/nlstradamus Tool test results Automated test environment Tests that passed successfully Tool id: nlstradamus Tool version: nlstradamus Test: test_tool_00 (functional.test_toolbox.TestForTool_toolshed.g2.bx.psu.edu/repos/peterjc/nlstradamus/nlstradamus/0.0.8) Tests that failed Tool id: nlstradamus Tool version: nlstradamus Test: test_tool_01 (functional.test_toolbox.TestForTool_toolshed.g2.bx.psu.edu/repos/peterjc/nlstradamus/nlstradamus/0.0.8) Stderr: Traceback: Traceback (most recent call last): File /var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-main-tool-shed-py27/build/test/functional/test_toolbox.py, line 171, in test_tool self.do_it( td, shed_tool_id=shed_tool_id ) File /var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-main-tool-shed-py27/build/test/functional/test_toolbox.py, line 78, in do_it self.run_tool( testdef.tool.id, repeat_name=repeat_name, **page_inputs ) File /var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-main-tool-shed-py27/build/test/base/twilltestcase.py, line 1327, in run_tool self.submit_form( **kwd ) File /var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-main-tool-shed-py27/build/test/base/twilltestcase.py, line 1276, in submit_form raise AssertionError( errmsg ) AssertionError: Attempting to set field 'fasta_file' to value '['empty.fasta']' in form 'tool_form' threw exception: cannot find value/label empty.fasta in list control control: SelectControl(fasta_file=[]) If the above control is a DataToolparameter whose data type class does not include a sniff() method, make sure to include a proper 'ftype' attribute to the tag for the control within the test tag set. http://toolshed.g2.bx.psu.edu/view/peterjc/effectivet3 Tool test results Automated test environment Tests that passed successfully Tool id: effectiveT3 Tool version: effectiveT3 Test: test_tool_00 (functional.test_toolbox.TestForTool_toolshed.g2.bx.psu.edu/repos/peterjc/effectivet3/effectiveT3/0.0.12) Tests that failed Tool id: effectiveT3 Tool version: effectiveT3 Test: test_tool_01 (functional.test_toolbox.TestForTool_toolshed.g2.bx.psu.edu/repos/peterjc/effectivet3/effectiveT3/0.0.12) Stderr: Traceback: Traceback (most recent call last): File /var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-main-tool-shed-py27/build/test/functional/test_toolbox.py, line 171, in test_tool self.do_it( td, shed_tool_id=shed_tool_id ) File /var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-main-tool-shed-py27/build/test/functional/test_toolbox.py, line 78, in do_it self.run_tool( testdef.tool.id, repeat_name=repeat_name, **page_inputs ) File /var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-main-tool-shed-py27/build/test/base/twilltestcase.py, line 1327, in run_tool self.submit_form( **kwd ) File /var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-main-tool-shed-py27/build/test/base/twilltestcase.py, line 1276, in submit_form raise AssertionError( errmsg ) AssertionError: Attempting to set field 'fasta_file' to value '['empty.fasta']' in form 'tool_form' threw exception: cannot find value/label empty.fasta in list control control: SelectControl(fasta_file=[]) If the above control is a DataToolparameter whose data type class does not include a sniff() method, make sure to include a proper 'ftype' attribute to the tag for the control within the test tag set. Traceback (most recent call last): File
Re: [galaxy-dev] Toolshed reset all repository metadata
Hello Simon, On Oct 23, 2013, at 5:01 PM, Simon Guest simon.gu...@agresearch.co.nz wrote: We've noticed a funny about resetting repository metadata from the toolshed web interface. My understanding is that this is used to get the toolshed back in sync with the underlying mercurial repo, for example after pushing updates from hg on the command line. This is not quite correct. When committing a new changeset to a repository (either via hg push from the command line or by using any feature in the Tool Shed's upload component), metadata is reset on the repository automatically, but only back to the previous installable changeset revision if one exists. So for those repositories that have multiple changeset revisions, not all of the repository changelog is inspected with a new commit. The Reset all repository metadata feature, however, does inspect the complete repository changelog and resets metadata from revision 0 through the repository tip. In addition to being useful for the current repository itself, it can also be useful for all other repositories that have a dependency relationship to it. See http://wiki.galaxyproject.org/RepositoryRevisions#Resetting_metadata_on_your_repositories. In general, it is not necessary to use this feature. It seems that only the person who created the repository can reset metadata. I granted authority to make changes to a colleague, which enables him to push updates, but he doesn't see the reset metadata entry in the menu. This seems like a bug to me. Or is it by design? This was the original desing, but a more recent oversight as this feature has evolved. This has been corrected in 4:71c35dbde130, which i committed to the next-stable branch and is now running on both the test and main Galaxy tol sheds. Also, is there a reason why this function is necessary at all? Yes, but it is not generally necessary for it to be used. It used to be restricted to only a Tool Shed admin, but recently it was exposed to repository owners (and noew anyone tha thas write access) because it became very useful for developers like Bjoern Gruening to be able to reset metadata on repositories (especially in the test tool shed) as new repository and tools dependency relationship features were being introduced at the same time as repository types (TOOL_DEPENDENCY_DEFINITION) and he was helping to flesh out the support for the new features. Wouldn't it be better if the toolshed just noticed updates to the underlying mercurial repo, and reset its own metadata automatically? It does, but hopefully my explanations above, as well as the details in the referenced tool shed wiki page will clarify this feature for you. cheers, Simon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Installing toolshed repositories with set_environment dependency
Hello Saskia,
This was indeed a bug (not abut which was a spell checker spelling error)
which has been resolved in 6:83bed9c7dbbc. This fix has been committed to
the next-stable branch and is currently running on both the test and main
Galaxy tool sheds. Thanks for reporting this issue.
Greg Von Kuster
On Oct 23, 2013, at 11:34 AM, S.D. Hiltemann s.hiltem...@erasmusmc.nl wrote:
I updated my Galaxies to the latest version today, and when I try to install
a toolshed repository which defines an environment variable via the
set_environment tags in the tool_dependencies.xml file, I get an error (see
below).
tool_dependencies.xml:
?xml version=1.0?
tool_dependency
set_environment version=1.0
environment_variable name=CONDEL_SCRIPT_PATH
action=set_to$REPOSITORY_INSTALL_DIR/environment_variable
/set_environment
/tool_dependency
(before the update these repositories installed without problems)
..doing the same as an action within a package dependency does still work, so
for now I have just changed the tools I needed to set the environment this
way, e.g.:
tool_dependency
package name=condel version=1
install version=1.0
actions
action type=set_environment
environment_variable name=CONDEL_SCRIPT_PATH
action=set_to$REPOSITORY_INSTALL_DIR/environment_variable
/action
/actions
/install
readme
Set condel environment variable
/readme
/package
/tool_dependency
Is this the preferred way of doing it, and should I change all my tools this
way? or should both ways still be working?
Saskia
Error Traceback:
View as: Interactive | Text | XML (full)
⇝ NameError: global name 'env_var_elem' is not defined
URL:
http://galaxy.trait-ctmm.cloudlet.sara.nl/admin_toolshed/manage_tool_dependencies?repository_id=b847e822bdc195d0operation=install
Module weberror.evalexception.middleware:364 in respond view
app_iter = self.application(environ, detect_start_response)
Module paste.recursive:84 in __call__ view
return self.application(environ, start_response)
Module paste.httpexceptions:633 in __call__ view
return self.application(environ, start_response)
Module galaxy.web.framework.base:132 in __call__ view
return self.handle_request( environ, start_response )
Module galaxy.web.framework.base:190 in handle_request view
body = method( trans, **kwargs )
Module galaxy.web.framework:229 in decorator view
return func( self, trans, *args, **kwargs )
Module galaxy.webapps.galaxy.controllers.admin_toolshed:757 in
manage_tool_dependencies view
self.initiate_tool_dependency_installation( trans,
tool_dependencies_for_installation )
Module galaxy.web.framework:229 in decorator view
return func( self, trans, *args, **kwargs )
Module galaxy.webapps.galaxy.controllers.admin_toolshed:433 in
initiate_tool_dependency_installation view
tool_dependencies=tool_dependencies )
Module tool_shed.util.common_install_util:486 in handle_tool_dependencies
view
env_var_name = env_var_elem.get( 'name', None )
NameError: global name 'env_var_elem' is not defined
From: S.D. Hiltemann
Sent: Friday, August 02, 2013 11:19 AM
To: galaxy-dev@lists.bx.psu.edu
Subject: Toolshed repository update error
I am working on putting a tool into the toolshed. I have a bash script
wrapper. I uploaded the first version fine, but when I wanted to upload a
revision, some unwanted characters are inserted around the revised function
of my code (see bottom) ..the inserted local etc strings are causing
my script to fail of course. How can I prevent this from happening?
Saskia
snippet of the code:
if [[ ! -s $rfile ]]
then
dummycol=${addcols:2}
outputcol=${dummycol//,B./}
local
echo -e
${col_chr_name}\t${col_start_name}\t${col_end_name}\t${col_ref_name}\t${col_obs_name}\t$outputcol
$rfile
cat $rfile
===
numcommas=`echo $addcols | grep -o , | wc -l`
echo numcolums: $numcommas
awk 'BEGIN{FS=\t;OFS=\t}{
if(FNR==1)
print $0,'$outputcol';
else{
printf $0
for(i=0;i='$numcommas'+1;i++)
printf \t
printf \n
}
}END{}' $ofile tempofile
mv tempofile $ofile
return
other
fi
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
Re: [galaxy-dev] Defining $GALAXY_SLOTS for use in tool wrappers
On Thu, Oct 17, 2013 at 8:37 AM, Bjoern Gruening bjoern.gruen...@gmail.com wrote: Am Samstag, den 12.10.2013, 19:42 +0100 schrieb Peter Cock: On Thu, Aug 1, 2013 at 10:27 AM, Nicola Soranzo sora...@crs4.it wrote: Il 2013-07-30 17:18 Peter Cock ha scritto: Hello all, Re: http://lists.bx.psu.edu/pipermail/galaxy-dev/2012-June/010153.html http://lists.bx.psu.edu/pipermail/galaxy-dev/2012-October/011557.html Something I raised during the GCC2013, and we talked about via Twitter as well was a Galaxy environment variable for use within Tool Wrappers setting the number of threads/CPUs to use. The idea is that you can configure a default value, and then override this per runner or per tool etc. Thanks Peter for pushing this idea, I totally support this proposal. In the mean time, I've been using for my tools the solution by Jim Johnson for its CD-HIT wrapper: http://toolshed.g2.bx.psu.edu/view/jjohnson/cdhit But this requires the system administrator to modify both the tool env.sh and job_conf.xml to be in sync. Is there an open Trello card for this? A Trello card would be useful indeed. Nicola Better than a Trello card, we now have a pull request from John: https://bitbucket.org/galaxy/galaxy-central/pull-request/236/job-runner-enhancements-galaxy_slots/diff And thanks to John its merged! Time for testing and migrating our tools :) So far $GALAXY_SLOTS seems to be working nicely for me. However, I am wondering if it would be possible to use it inside the configfile section? Is that run at the time of job creation on the Galaxy server (where determining the number of threads may be hard) or as part of job execution (e.g. on the cluster, when $GALAXY_SLOTS would be known). I have tried using \$GALAXY_SLOTS but it remains in the file generated by configfile as $GALAXY_SLOTS rather than being substituted. Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Defining $GALAXY_SLOTS for use in tool wrappers
So far $GALAXY_SLOTS seems to be working nicely for me. However, I am wondering if it would be possible to use it inside the configfile section? Is that run at the time of job creation on the Galaxy server (where determining the number of threads may be hard) or as part of job execution (e.g. on the cluster, when $GALAXY_SLOTS would be known). Unfortunately, the former, and there is no easy way to change this. One could do some simple variable substitution on the config file when the job runs though (sed would do the job here, not pretty but it works). ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Defining $GALAXY_SLOTS for use in tool wrappers
On Thu, Oct 24, 2013 at 5:53 PM, James Taylor ja...@jamestaylor.org wrote: So far $GALAXY_SLOTS seems to be working nicely for me. However, I am wondering if it would be possible to use it inside the configfile section? Is that run at the time of job creation on the Galaxy server (where determining the number of threads may be hard) or as part of job execution (e.g. on the cluster, when $GALAXY_SLOTS would be known). Unfortunately, the former, and there is no easy way to change this. Ah. That was what I suspected :( One could do some simple variable substitution on the config file when the job runs though (sed would do the job here, not pretty but it works). Yeah - I was thinking about something like that as a work around, although since I have a wrapper Python script anyway I can do the edit there: https://github.com/peterjc/pico_galaxy/tree/master/tools/mira4 Thanks for such a prompt answer, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Toolshed reset all repository metadata
At Thu, 24 Oct 2013 10:03:14 -0400, Greg Von Kuster wrote: Hello Simon, On Oct 23, 2013, at 5:01 PM, Simon Guest simon.gu...@agresearch.co.nz wrote: We've noticed a funny about resetting repository metadata from the toolshed web interface. My understanding is that this is used to get the toolshed back in sync with the underlying mercurial repo, for example after pushing updates from hg on the command line. This is not quite correct. When committing a new changeset to a repository (either via hg push from the command line or by using any feature in the Tool Shed's upload component), metadata is reset on the repository automatically, but only back to the previous installable changeset revision if one exists. So for those repositories that have multiple changeset revisions, not all of the repository changelog is inspected with a new commit. The Reset all repository metadata feature, however, does inspect the complete repository changelog and resets metadata from revision 0 through the repository tip. In addition to being useful for the current repository itself, it can also be useful for all other repositories that have a dependency relationship to it. See http://wiki.galaxyproject.org/RepositoryRevisions#Resetting_metadata_on_your_repositories. In general, it is not necessary to use this feature. Hi Greg, Thanks for the explanation, and the link to that helpful Wiki page. And also that code update 4:71c35dbde130 which will help our tool test/development cycle. I have a remaining question, though, about the recommended working practices for developing a toolshed tool. I didn't find this described on the Wiki. There is obviously a cycle of development to go through, making updates, installing to a test galaxy instance, etc. It seems to us that the only sensible way to handle this is within a local test toolshed, essentially using the repo in the test toolshed as a development repo (despite the warnings not use the toolshed as a source code repo on the Wiki!). During such an iterative process, we don't want to keep bumping the tool version. Therefore it seems necessary to reset the repository meta data after every push, in order to install the just-updated-for-testing tool. Once the tool is working, the files can be uploaded into a clean repo on a production toolshed, so we don't keep all the intermediate development versions. Or is there a better way to do this? cheers, Simon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Toolshed reset all repository metadata
Hello Simon, On Oct 24, 2013, at 4:06 PM, Guest, Simon simon.gu...@agresearch.co.nz wrote: Hi Greg, Thanks for the explanation, and the link to that helpful Wiki page. And also that code update 4:71c35dbde130 which will help our tool test/development cycle. I have a remaining question, though, about the recommended working practices for developing a toolshed tool. I didn't find this described on the Wiki. There is obviously a cycle of development to go through, making updates, installing to a test galaxy instance, etc. It seems to us that the only sensible way to handle this is within a local test toolshed, essentially using the repo in the test toolshed as a development repo (despite the warnings not use the toolshed as a source code repo on the Wiki!). This is certainly a good approach, although using a local tool shed as a component of the devlopment process when developing tools is probably useful only when your repository includes repository dependency definitions or tool dependency definitions. In these cases, the tool shed encironment becomes useful for working out the details of the dependencies and making sure that everything ( all repository and tool dependencies ) installs correctly into your local Galaxy instance. In cases where you are developing simpler tools ( e.g., Text Manipulation tools like filter and sort ) that are completely self contained ( they have no dependencies ), using a local tool shed during the development process is probably not necessary. During such an iterative process, we don't want to keep bumping the tool version. It may ot be necessary to do so - the only time you should be bumping the tool version ( which is a manual process of changing the version string in the tool config ) is when the tool behavior changes such that the same input dataset produces defferent resulting output datasets(s). Therefore it seems necessary to reset the repository meta data after every push, in order to install the just-updated-for-testing tool. If you are using mercurial ersion 2.2.3 or later in your shell environment, then pushing to your local tool shed from the command line should be resetting metadata back to the last installable changeset revsion. If you are not seeing this behavior, make sure you are using that version or later - see http://wiki.galaxyproject.org/ToolShedRepositoryFeatures#Pushing_changes_to_a_repository_using_hg_from_the_command_line Once the tool is working, the files can be uploaded into a clean repo on a production toolshed, so we don't keep all the intermediate development versions. Yes, this is definitely the correct approach if you do find that using a local tool shed during the development process is beneficial. Or is there a better way to do this? I think you have things pretty well figured out. Don't hesitate to continue to ask questions as they arise. cheers, Simon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
