[galaxy-dev] My last day as a core Galaxy Development Team member
Hello Galaxians, I have moved into a new position here at Penn State University, so for the first time in 8 years I am not a member of the core Galaxy development team. This has been a very difficult transition for me - Galaxy has been, and continues to be, extremely important to me. I've really appreciated working with the Galaxy team and the large Galaxy community over these 8 years and I hope I can continue to work with Galaxy in some capacity. I've been fortunate to have found many friends in the bioinformatics community, and I hope those relationsips will continue as well. I feel that Galaxy is far from reaching its peak and that its future remains very bright. I look forward to seeing how far it will go. Warm regards to each of you, Greg Von Kuster ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Main Galaxy Tool Shed is running the next-stable branch
Hi Eric, Sorry for the inconvenience. The tool shed server was out of space. I've corrected the problem and have successfully installed repositories into my local Galaxy instance. Please try again and let us know if you encounter problems. Greg Von Kuster On Aug 6, 2014, at 10:59 AM, Eric Kuyt erick...@gmail.com wrote: Hi Greg, Could there be some problems with https://toolshed.g2.bx.psu.edu/ I have some troubles installing toolshed tools in our galaxy. When I try installing for instance 'suite_samtools_0_1_19', I am presented with a 500 error. I tried then pulling a new stable galaxy from bitbucket, which seems to have the same problem. Maybe I'm doing something wrong both times or toolshed is having problems. I opt for the latter of course. -- part of galaxy traceback -- File '/usr/lib/python2.7/urllib2.py', line 527 in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) HTTPError: HTTP Error 500: Internal Server Error Thanks, Eric On 29 July 2014 23:35, Greg Von Kuster g...@bx.psu.edu wrote: Hello all, The main Galaxy Tool Shed is now running the next-stable branch ( e6813f244a2a (next-stable) tip ) in preparation for next upcoming Galaxy release. Please let us know if you encounter anything strange or broken and we'll get things fixed asap. The main Tool Shed will be updated to the stable branch when it is created for the upcoming Galaxy release. Thanks very much, Greg Von Kuster ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Issues with dependencies for toolshed packages
Hi Iry, Sorry for the inconvenience. The tool shed server was out of space - I've corrected the problem and have successfully installed repositories. Please try again and let us know if you encounter problems. Thanks! Greg Von Kuster On Aug 6, 2014, at 12:20 PM, Iry Witham iry.wit...@jax.org wrote: Hi Team, I have been trying to install package_snpEff_3_5 from the Galaxy main tool shed and package_snpEff_3_6 from the Galaxy test tool shed and have the same issue. Once the installer completes it shows that the dependency snpEff.x.x was not installed. When I try to install the dependency it reports that the directory /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/database/tmp/tmp-toolshed-mtdi48tQV/snpEff does not exist. I have checked to confirm that this is true and discovered that rather then a directory I find a null file named snpEff and the snpEff_v3.x_core.zip file. I checked my shedtool dependencies folder and the directory for these tools are empty. I have since uninstalled the package. I just tried again to install the snpEff tool and am getting the followig error message: Internal Server Error Galaxy was unable to successfully complete your request URL: http://galaxy2/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=76ad6cd032d7117echangeset_revisions=04d2a4f81413 Module galaxy.web.framework.middleware.error:149 in __call__ app_iter = self.application(environ, sr_checker) Module paste.recursive:84 in __call__ return self.application(environ, start_response) Module paste.httpexceptions:633 in __call__ return self.application(environ, start_response) Module galaxy.web.framework.base:132 in __call__ return self.handle_request( environ, start_response ) Module galaxy.web.framework.base:190 in handle_request body = method( trans, **kwargs ) Module galaxy.web.framework:377 in decorator return func( self, trans, *args, **kwargs ) Module galaxy.webapps.galaxy.controllers.admin_toolshed:1035 in prepare_for_install raw_text = common_util.tool_shed_get( trans.app, tool_shed_url, url ) Module tool_shed.util.common_util:310 in tool_shed_get response = urlopener.open( uri ) Module urllib2:395 in open response = meth(req, response) Module urllib2:508 in http_response 'http', request, response, code, msg, hdrs) Module urllib2:427 in error result = self._call_chain(*args) Module urllib2:367 in _call_chain result = func(*args) Module urllib2:603 in http_error_302 return self.parent.open(new) Module urllib2:395 in open response = meth(req, response) Module urllib2:508 in http_response 'http', request, response, code, msg, hdrs) Module urllib2:433 in error return self._call_chain(*args) Module urllib2:367 in _call_chain result = func(*args) Module urllib2:516 in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) HTTPError: HTTP Error 500: Internal Server Error extra data full traceback text version This may be an intermittent problem due to load or other unpredictable factors, reloading the page may address the problem. I appreciate any possible assistance, Regards, __ Iry T. Witham Scientific Applications Administrator Scientific Computing Group Computational Sciences Dept. The Jackson Laboratory 600 Main Street Bar Harbor, ME 04609 Phone: 207-288-6744 email: iry.wit...@jax.org 372D007A-1B00-4668-BA6B-F0527C1F24BE[34][3][1].png The information in this email, including attachments, may be confidential and is intended solely for the addressee(s). If you believe you received this email by mistake, please notify the sender by return email as soon as possible. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Uploads with embedded citations causing red error on Tool Shed
Both Galaxy Tool Sheds have been updated with this fix. Greg On Jul 30, 2014, at 7:14 AM, Peter Cock p.j.a.c...@googlemail.com wrote: Thanks John - is there any point/benefit to re-uploading my tool once the fix is live on the Tool Shed? i.e. Was it a harmless warning? Peter On Wed, Jul 30, 2014 at 12:12 PM, John Chilton jmchil...@gmail.com wrote: Hey Peter, Opps sorry about that and thanks for the bug report. The tool shed code should be fixed with https://bitbucket.org/galaxy/galaxy-central/commits/38ba45d6ba5be65b3b743fc08739e16cd6e0ac8f - it is in next-stable so I think the tool shed should pick up that fix at next tool shed update. -John On Wed, Jul 30, 2014 at 5:43 AM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi John, Following the work at the BOSC 2014 CodeFest to support embedded citations within Galaxy Tool XML files [1], and your work adding this to the BLAST tools as an example [2], I tried uploading a minor tool using this to the Tool Shed. The upload seems to have worked, but there was a scary red error message about metadata... see below (both main and test toolsheds affected). Regards, Peter [1] https://bitbucket.org/galaxy/galaxy-central/pull-request/440/ [2] https://github.com/peterjc/galaxy_blast/commit/9d2e3906915895765ecc3f48421b91fabf2ccd8b -- Uploading on the Test Tool Shed, red error: Metadata may have been defined for some items in revision '796dc2ff8e8e'. Correct the following problems if necessary and reset metadata. blastxml_to_top_descr.xml - 'UniverseApplication' object has no attribute 'citations_manager' https://testtoolshed.g2.bx.psu.edu/view/peterjc/blastxml_to_top_descr -- Uploading on the Tool Shed, red error: Metadata may have been defined for some items in revision 'fe1ed74793c9'. Correct the following problems if necessary and reset metadata. blastxml_to_top_descr.xml - 'UniverseApplication' object has no attribute 'citations_manager' https://toolshed.g2.bx.psu.edu/view/peterjc/blastxml_to_top_descr ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Uploads with embedded citations causing red error on Tool Shed
Yes, it looks like metadata just needs to be regenerated on the affected repositories. Let me know if doing so uncovers something else. Greg On Jul 30, 2014, at 7:20 AM, John Chilton jmchil...@gmail.com wrote: ... that might be a Greg question. The tool shed isn't going to use the citation information in anyway at this time (https://trello.com/c/R8vKH4PQ) - so you won't gain anything from a citation-y perspective by re-uploading. That bug might have interfered with the tool shed metadata generation process though(?) so it might be worth regenerating that when the tool shed is updated(?). That is just a guess though - maybe Greg will weigh in on whether that makes sense. -John On Wed, Jul 30, 2014 at 7:14 AM, Peter Cock p.j.a.c...@googlemail.com wrote: Thanks John - is there any point/benefit to re-uploading my tool once the fix is live on the Tool Shed? i.e. Was it a harmless warning? Peter On Wed, Jul 30, 2014 at 12:12 PM, John Chilton jmchil...@gmail.com wrote: Hey Peter, Opps sorry about that and thanks for the bug report. The tool shed code should be fixed with https://bitbucket.org/galaxy/galaxy-central/commits/38ba45d6ba5be65b3b743fc08739e16cd6e0ac8f - it is in next-stable so I think the tool shed should pick up that fix at next tool shed update. -John On Wed, Jul 30, 2014 at 5:43 AM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi John, Following the work at the BOSC 2014 CodeFest to support embedded citations within Galaxy Tool XML files [1], and your work adding this to the BLAST tools as an example [2], I tried uploading a minor tool using this to the Tool Shed. The upload seems to have worked, but there was a scary red error message about metadata... see below (both main and test toolsheds affected). Regards, Peter [1] https://bitbucket.org/galaxy/galaxy-central/pull-request/440/ [2] https://github.com/peterjc/galaxy_blast/commit/9d2e3906915895765ecc3f48421b91fabf2ccd8b -- Uploading on the Test Tool Shed, red error: Metadata may have been defined for some items in revision '796dc2ff8e8e'. Correct the following problems if necessary and reset metadata. blastxml_to_top_descr.xml - 'UniverseApplication' object has no attribute 'citations_manager' https://testtoolshed.g2.bx.psu.edu/view/peterjc/blastxml_to_top_descr -- Uploading on the Tool Shed, red error: Metadata may have been defined for some items in revision 'fe1ed74793c9'. Correct the following problems if necessary and reset metadata. blastxml_to_top_descr.xml - 'UniverseApplication' object has no attribute 'citations_manager' https://toolshed.g2.bx.psu.edu/view/peterjc/blastxml_to_top_descr ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Main Galaxy Tool Shed is running the next-stable branch
Hello all, The main Galaxy Tool Shed is now running the next-stable branch ( e6813f244a2a (next-stable) tip ) in preparation for next upcoming Galaxy release. Please let us know if you encounter anything strange or broken and we'll get things fixed asap. The main Tool Shed will be updated to the stable branch when it is created for the upcoming Galaxy release. Thanks very much, Greg Von Kuster ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] [galaxy-iuc] writing datatypes
Hi Björn, On Jul 22, 2014, at 6:01 PM, Björn Grüning bjoern.gruen...@gmail.com wrote: Hi Greg, thanks for the clarification. Please see my comments below. On Jul 20, 2014, at 3:22 PM, Peter Cock p.j.a.c...@googlemail.com wrote: On Sun, Jul 20, 2014 at 6:23 PM, Björn Grüning bjo...@gruenings.eu wrote: Hi, single datatype definitions only work if you haven’t defined any converters. Let's assume I have a datatype X and want to ship a X - Y converter (Y - X is also possible), we will end up with a dependency loop, or? The X repository will depend on the Y repository, but Y is depending on X, because we want to include a Y - X converter. Any idea how to solve that? I don't see a problem here, so I'm hoping I'm correctly understanding the issue. If we have: repo_x contains the single datatype X repo_y contains the single datatype Y repo_x_to_y_converter contains a tool that converts datatype X to datatype Y (this repository also defines 2 dependency relationships, one to repo_x and another to repo_y) repo_y_to_x_cenverter contains a tool that converts datatype Y to datatype X (this repository also defines 2 dependency relationships, one to repo_x and another to repo_y) Now if we want to install both the repo_x_to_y_converter and the repo_y_to_x_cenverter automatically whenever either one is installed, we have 2 options: 1) define a 3rd dependency relationshiop for repo_x_to_y_converter to depend on repo_y_to_x_cenverter and, similarly a 3rd dependency relationshiop for repo_y_to_x_cenverter on repo_x_to_y_converter. This does indeed create a circular repository dependency relationship, but the Tool Shed installation process will handle it correctly, installing all 4 repositories with proper dependency relationships created between them Does that mean, circular dependencies will be no problem at all? Yes, the Tool Shed handles circular dependency definitions of any variety, so circular dependency definitions pose no problem. Do you consider including the converters into the datatypes as best-practise? (These converters are implicit-galaxy-converters). I would have only two repositories with circular dependencies. Yes, however, there are some current limitations in the framework detailed on this Trello card: https://trello.com/c/Ho3ra4b9/206-add-support-for-datatype-converters-and-display-applications Tag sets like the following that are defined in a datatypes_conf.xml file contained in a repository should be correctly loaded into the in-memory datatypes registry when the repository is instlled into Galaxy. However, it has been quite a while since I've worked in this area, so let me know if you encounter any issues. The current best practice is probaly that the converters themselved would each individually be in separate repositories (just like all Galaxy tools), but this can certainly be discussed if appropriate. Community thoughts are welcome here! datatype extension=bam type=galaxy.datatypes.binary:Bam mimetype=application/octet-stream display_in_upload=true converter file=bam_to_bai.xml target_datatype=bai/ converter file=bam_to_bigwig_converter.xml target_datatype=bigwig/ display file=ucsc/bam.xml / display file=ensembl/ensembl_bam.xml / display file=igv/bam.xml / display file=igb/bam.xml / /datatype 2) Instead of creating a circlular dependency relationship between repo_x_to_y_converter and repo_y_to_x_cenverter, create an additional suite_definition_x_y repository (of type repository_suite_definition that defines relationships to repo_x_to_y_converter and repo_y_to_x_cenverter, ultimately installing all 4 repositories, but without defining any circular dependency relationships. repo_x_to_y_converter and repo_y_to_x_converter would have dependencies on datatype X and Y, so I do not see the need for a suite_definition ... or it is some collection like the emboss_datatypes … I agree. My scenario is more that the converters are not tools, they are implicit converters and should _not_ be displayed in the tool panel. As far as I know they need to be defined inside the datatypes_conf.xml file. Yes, they must be defined inside the datatypes_conf.xml file. However, converters are just special Galaxy Tools (they are special in the same way that Data Manager tools are special). They are loaded into the in-memory Galaxy tools registry, but not displayed in the tool panel. I think if circular dependencies are not a problem I will try to implement a proof of concept. EMBOSS is now splitted: Sounds goos - circular dependencies should pose no problems. https://github.com/bgruening/galaxytools/tree/master/datatypes/emboss_datatypes Thanks Greg! Bjoern Either of the above 2 scenarios will correctly install the 4 repositories. Let me know if I'm missing something here. Thanks! Greg Excellent example! How to handle
Re: [galaxy-dev] [galaxy-iuc] writing datatypes
Before we go too much further down this path with dataytpes, I'm wondering if some of us should put together a spec of some kind that allows us to all agree on the direction. For example, I'm wondering if datatyps should be versioned and have a name-spaced identifier much like the Tool Shed's guid identifier for tools. I haven't thought too much about whether this would pose backward compatibility issues or not. Discussion is welcomed on this. Greg Von Kuster On Jul 22, 2014, at 7:19 PM, Greg Von Kuster g...@bx.psu.edu wrote: Hi Björn, On Jul 22, 2014, at 6:01 PM, Björn Grüning bjoern.gruen...@gmail.com wrote: Hi Greg, thanks for the clarification. Please see my comments below. On Jul 20, 2014, at 3:22 PM, Peter Cock p.j.a.c...@googlemail.com wrote: On Sun, Jul 20, 2014 at 6:23 PM, Björn Grüning bjo...@gruenings.eu wrote: Hi, single datatype definitions only work if you haven’t defined any converters. Let's assume I have a datatype X and want to ship a X - Y converter (Y - X is also possible), we will end up with a dependency loop, or? The X repository will depend on the Y repository, but Y is depending on X, because we want to include a Y - X converter. Any idea how to solve that? I don't see a problem here, so I'm hoping I'm correctly understanding the issue. If we have: repo_x contains the single datatype X repo_y contains the single datatype Y repo_x_to_y_converter contains a tool that converts datatype X to datatype Y (this repository also defines 2 dependency relationships, one to repo_x and another to repo_y) repo_y_to_x_cenverter contains a tool that converts datatype Y to datatype X (this repository also defines 2 dependency relationships, one to repo_x and another to repo_y) Now if we want to install both the repo_x_to_y_converter and the repo_y_to_x_cenverter automatically whenever either one is installed, we have 2 options: 1) define a 3rd dependency relationshiop for repo_x_to_y_converter to depend on repo_y_to_x_cenverter and, similarly a 3rd dependency relationshiop for repo_y_to_x_cenverter on repo_x_to_y_converter. This does indeed create a circular repository dependency relationship, but the Tool Shed installation process will handle it correctly, installing all 4 repositories with proper dependency relationships created between them Does that mean, circular dependencies will be no problem at all? Yes, the Tool Shed handles circular dependency definitions of any variety, so circular dependency definitions pose no problem. Do you consider including the converters into the datatypes as best-practise? (These converters are implicit-galaxy-converters). I would have only two repositories with circular dependencies. Yes, however, there are some current limitations in the framework detailed on this Trello card: https://trello.com/c/Ho3ra4b9/206-add-support-for-datatype-converters-and-display-applications Tag sets like the following that are defined in a datatypes_conf.xml file contained in a repository should be correctly loaded into the in-memory datatypes registry when the repository is instlled into Galaxy. However, it has been quite a while since I've worked in this area, so let me know if you encounter any issues. The current best practice is probaly that the converters themselved would each individually be in separate repositories (just like all Galaxy tools), but this can certainly be discussed if appropriate. Community thoughts are welcome here! datatype extension=bam type=galaxy.datatypes.binary:Bam mimetype=application/octet-stream display_in_upload=true converter file=bam_to_bai.xml target_datatype=bai/ converter file=bam_to_bigwig_converter.xml target_datatype=bigwig/ display file=ucsc/bam.xml / display file=ensembl/ensembl_bam.xml / display file=igv/bam.xml / display file=igb/bam.xml / /datatype 2) Instead of creating a circlular dependency relationship between repo_x_to_y_converter and repo_y_to_x_cenverter, create an additional suite_definition_x_y repository (of type repository_suite_definition that defines relationships to repo_x_to_y_converter and repo_y_to_x_cenverter, ultimately installing all 4 repositories, but without defining any circular dependency relationships. repo_x_to_y_converter and repo_y_to_x_converter would have dependencies on datatype X and Y, so I do not see the need for a suite_definition ... or it is some collection like the emboss_datatypes … I agree. My scenario is more that the converters are not tools, they are implicit converters and should _not_ be displayed in the tool panel. As far as I know they need to be defined inside the datatypes_conf.xml file. Yes, they must be defined inside the datatypes_conf.xml file. However, converters are just special Galaxy Tools (they are special in the same way that Data Manager tools
Re: [galaxy-dev] writing datatypes
Please see my comments below. On Jul 20, 2014, at 3:22 PM, Peter Cock p.j.a.c...@googlemail.com wrote: On Sun, Jul 20, 2014 at 6:23 PM, Björn Grüning bjo...@gruenings.eu wrote: Hi, single datatype definitions only work if you haven’t defined any converters. Let's assume I have a datatype X and want to ship a X - Y converter (Y - X is also possible), we will end up with a dependency loop, or? The X repository will depend on the Y repository, but Y is depending on X, because we want to include a Y - X converter. Any idea how to solve that? I don't see a problem here, so I'm hoping I'm correctly understanding the issue. If we have: repo_x contains the single datatype X repo_y contains the single datatype Y repo_x_to_y_converter contains a tool that converts datatype X to datatype Y (this repository also defines 2 dependency relationships, one to repo_x and another to repo_y) repo_y_to_x_cenverter contains a tool that converts datatype Y to datatype X (this repository also defines 2 dependency relationships, one to repo_x and another to repo_y) Now if we want to install both the repo_x_to_y_converter and the repo_y_to_x_cenverter automatically whenever either one is installed, we have 2 options: 1) define a 3rd dependency relationshiop for repo_x_to_y_converter to depend on repo_y_to_x_cenverter and, similarly a 3rd dependency relationshiop for repo_y_to_x_cenverter on repo_x_to_y_converter. This does indeed create a circular repository dependency relationship, but the Tool Shed installation process will handle it correctly, installing all 4 repositories with proper dependency relationships created between them 2) Instead of creating a circlular dependency relationship between repo_x_to_y_converter and repo_y_to_x_cenverter, create an additional suite_definition_x_y repository (of type repository_suite_definition that defines relationships to repo_x_to_y_converter and repo_y_to_x_cenverter, ultimately installing all 4 repositories, but without defining any circular dependency relationships. Either of the above 2 scenarios will correctly install the 4 repositories. Let me know if I'm missing something here. Thanks! Greg Excellent example! How to handle versions of datatypes? Extra repositories for stockholm 1.0 and 1.1? If so ... the associated python file (sniffing, splitting ...) should be also versioned, or? What happend if I have two stockholm.py files in my system? Potentially you might need/want to define those as two different Galaxy datatypes? @Peter, can we create a striped-down, python only biopython egg? All parsers should be included, Bio.SeqIO should be sufficient I think. Right now, yes in principle (and this is fine from the licence point of view), but in practise this is a fair chunk of work. However, we are looking at this - see https://github.com/biopython/biopython/issues/349 Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] writing datatypes
Hi John, The general question, I think, is whether reproducibility is important. If it is, then we should not introduce new behavior that adversely impacts it. There are undoubtedly scenarios where reproducibility is not currently absolutely guaranteed, but those area of weakness should be corrected (as time and resources allow) when they are discovered if reproducibility is one of the desired features. Please see my inline comments too. On Jul 18, 2014, at 11:59 AM, John Chilton jmchil...@gmail.com wrote: Does the current implementation really handle datatypes in reproducible manner - if I have a repo which in revision 1 defines foo1 as a text subtype, foo2 as a tabular type and foo3 as a new type in foo.py and then in revision 2 foo1 is defined as a binary subtype , foo2 and foo3 disappear and foo4 is a new type in foo.py (which no longer defines foo3) how could you possibly resolve that in a reproducible manner. So you have: repo_a revision 1: foo1 datatype as text subtype foo2 datatype as tabular foo3 new datatype in foo.py repo_a revision 2: foo1 datatype as binary subtype foo4 new type in foo.py I would say that this is an example of a bad practice on the part of the repository owner, but, of course, this scenario can certainly occur. In this case, the current implementation creates 2 separate installable revisions of repo_a which are loaded into the datatype's registry in a specific order. If repo_a revision 1 was installed first, then it will always be loaded first, and the foo1 and foo4 datatypes contained in repo_a revision 2 will not be loaded because they are currently considered conflicting datatypes. So currently, reproducibility is ensured, but the versions of foo1 and foo4 in revision 2 cannot be used. This may not be ideal, but in order to allow both versions to be used, more than the datatype extensions will be needed in order to defferentiate datatypes (i.e., some named-spaced identifier similar to the Tool Shed's guid for tools). Some of your tools are going to expect foo1 to be one thing - others something else. You are only going to place 1 copy of foo.py on the PYTHONPATH right (or at least python will only load one)? Is it going to define foo3 or foo4? In addition to lacking reproducibility within one instance - if you are somehow trying to preserve all the datatypes a repository has ever defined I feel like after a long stream of such updates - the behavior of the datatypes is going to vary from one installation to another that installed different repository versions. Hence - reproducibility across instances is subtly broken as well? None of this is a solution of course - this problem strikes me as being very difficult. That said - I think correctness and reproduciblity across instances is more important than reproducibility within the same instance over time - so for that reason I think there only being one installable revision of datatypes might be a big step forward relative to the status quo. Intuitively - if we are not namespacing/versioning datatypes - there should only be one definition and it should be the most recently installed one right? It would also resolve this https://trello.com/c/oTq2Kewd problem - where unsniffable binary datatypes are treated as sniffiable if there was ever an installed version that was some sniff-able datatype. -John On Jul 17, 2014 12:35 PM, Greg Von Kuster g...@bx.psu.edu wrote: This would be easy to implement, but could adversely affect reproducibility. If a repository containing datatypes always had only a single installable revision (i.e., the chagelog tip), then any datatypes defined in an early changeset revision that are removed in a later changeset revision would no longer be available. Greg On Jul 17, 2014, at 1:30 PM, Peter Cock p.j.a.c...@googlemail.com wrote: On Thu, Jul 17, 2014 at 6:10 PM, Björn Grüning bjoern.gruen...@gmail.com wrote: ... but the problem will stay the same ... one [datatype definition] repository can have multiple versions ... I like your idea that like tool dependency definitions, this should be a special repository type on the ToolShed: Earlier, Björn Grüning bjoern.gruen...@gmail.com wrote: Imho datatypes should be handled like Tool dependency definitions. There should be only one installable revsion. This is something Greg will have to comment on - there may be ramifications I'm not seeing. Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies
Re: [galaxy-dev] writing datatypes
This would be easy to implement, but could adversely affect reproducibility. If a repository containing datatypes always had only a single installable revision (i.e., the chagelog tip), then any datatypes defined in an early changeset revision that are removed in a later changeset revision would no longer be available. Greg On Jul 17, 2014, at 1:30 PM, Peter Cock p.j.a.c...@googlemail.com wrote: On Thu, Jul 17, 2014 at 6:10 PM, Björn Grüning bjoern.gruen...@gmail.com wrote: ... but the problem will stay the same ... one [datatype definition] repository can have multiple versions ... I like your idea that like tool dependency definitions, this should be a special repository type on the ToolShed: Earlier, Björn Grüning bjoern.gruen...@gmail.com wrote: Imho datatypes should be handled like Tool dependency definitions. There should be only one installable revsion. This is something Greg will have to comment on - there may be ramifications I'm not seeing. Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] writing datatypes
Here's a Trello card for this: https://trello.com/c/s8tfbW4x On Jul 17, 2014, at 1:38 PM, Peter Cock p.j.a.c...@googlemail.com wrote: Good point Greg. Let's refine this slightly then, a new special ToolShed repository type for a *single* datatype definition. That avoids this problem :) (This does not help with suites of very closely related datatypes - like different kinds of BLAST database.) Peter On Thu, Jul 17, 2014 at 6:35 PM, Greg Von Kuster g...@bx.psu.edu wrote: This would be easy to implement, but could adversely affect reproducibility. If a repository containing datatypes always had only a single installable revision (i.e., the chagelog tip), then any datatypes defined in an early changeset revision that are removed in a later changeset revision would no longer be available. Greg On Jul 17, 2014, at 1:30 PM, Peter Cock p.j.a.c...@googlemail.com wrote: On Thu, Jul 17, 2014 at 6:10 PM, Björn Grüning bjoern.gruen...@gmail.com wrote: ... but the problem will stay the same ... one [datatype definition] repository can have multiple versions ... I like your idea that like tool dependency definitions, this should be a special repository type on the ToolShed: Earlier, Björn Grüning bjoern.gruen...@gmail.com wrote: Imho datatypes should be handled like Tool dependency definitions. There should be only one installable revsion. This is something Greg will have to comment on - there may be ramifications I'm not seeing. Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] writing datatypes
Assuming this comment: Finally, we will talk to the devteam to rewrite EMBOSS to depend on our separate data type repositories. refers to the emboss_5 repository owned by devteam, then what is being proposed should work (although I may not be fully understanding what is being proposed). If the emboss datatypes are split up and the emboss_5 repository's repository_dependencies.xml file is altered, then a new installable revision of the emboss_5 repository will be created. This implies that any previous installation cannot be updated to include the split up emboss datatypes. Instead, a new installation of the emboss_5 repository will be required. This new installation may depend on emboss datatypes that conflict with those in the older emboss_5 installation, and the 2nd version of the conflicting datatypes will not be loaded into the Galaxy datatypes registry. However, if the datatypes are the same, this shouldn't be a problem since the 1st version will have been loaded. On Jul 16, 2014, at 4:52 PM, John Chilton jmchil...@gmail.com wrote: Is this going to work? I get that this would be a better design if done from the beginning, but what happens if you install an emboss repository upgrade (on an existing install) that brings in conflicting types from other repositories that already exist and have been previously installed? Does the tool shed have a mechanism to handle that? -John On Wed, Jul 16, 2014 at 9:20 AM, Björn Grüning bjoern.gruen...@gmail.com wrote: Hi Eric, Forgive me, I'm not 100% clear on the custom plugin system used by galaxy, but if I subclass from the text data type, will sniffers I implement override text's and function? The lack of being able to add an entry to the sniffer section (unlike with the tabular example) led me to believe my genbank datatype wouldn't be sniffed. Thats true, if you want to override functions, you need to subclass it on a python level not on the XML level. Additionally, I'd still like to be able to add completely new datatypes, do you know of any working examples of this? As mentioned in my original post, duplicating an existing datatype and changing names on it surprisingly doesn't work. https://github.com/bgruening/galaxytools/tree/master/datatypes/msa_datatypes https://github.com/bgruening/galaxytools/blob/master/chemicaltoolbox/datatypes/datatypes_conf.xml Is that enough, to get started? I'd be lovely to have the emboss datatypes split out. Ok, than lets start :) I will try to fork emboss into my galaxytools/datatypes repository and try to split them. You will get commit access and can improve your genbank datatype (and a few more ;)). Finally, we will talk to the devteam to rewrite EMBOSS to depend on our separate data type repositories. OK? Ciao, Bjoenr Cheers, Eric On July 16, 2014 8:34:55 AM CDT, Peter Cock p.j.a.c...@googlemail.com wrote: Indeed - ideally (once working) we can upload under the IUC ToolShed as a community maintained resource rather than under a personal account which becomes a single point of failure (the bus factor). We (the ICU) have previously discussed doing this so that the EMBOSS datatypes could become more of a meta-entry depending on other smaller specific datatype defining ToolShed repositories. But it hasn't reached the top of my personal TODO list yet ;) Peter On Wed, Jul 16, 2014 at 1:47 PM, Björn Grüning bjoern.gruen...@gmail.com wrote: Hi Eric, please have a look at: https://github.com/bgruening/galaxytools/blob/master/datatypes/msa_datatypes/datatypes_conf.xml You need somthing like: datatype extension=genbank type=galaxy.datatypes.data:Text subclass=True / Lets try to split the EMBOSS datatypes a little bit into small chunks. E.g. sequences_datatypes, msa_datatypes ... and so on ... Cheers, Bjoern Am 14.07.2014 20:31, schrieb Eric Rasche: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I'm trying to add a new datatype to my galaxy instance for genbank files, however I'm running into various issues. I've followed the tutorial (https://wiki.galaxyproject.org/Admin/Datatypes/Adding%20Datatypes) however that example subclasses tabular, and I'd like to subclass Text as they're plain text files, and I'd like to be able to define a sniffer for them (not possible if your type=galaxy.datatypes.data:Text) I figured the call ought to be something like datatype extension=gb type=galaxy.datatypes.data:Genbank subclass=True / however, everything I try fails with Error importing datatype module galaxy.datatypes.data: 'module' object has no attribute 'Genbank' To avoid this particular issue, I tried writing a separate datatype just for genbank files (type=galaxy.datatypes.genbank:Genbank), however that fails with the same error: galaxy.datatypes.registry ERROR 2014-07-14 13:23:23,100 Error importing datatype module
Re: [galaxy-dev] Toolshed : tool_dependency xml, action problem
Are the problems described in this thread occurring with the latest Galaxy release? The references to the code lines do not seem to reflect the latest Galaxy release. Thanks, Greg Von Kuster On Jul 7, 2014, at 10:34 AM, Nicolas Lapalu nicolas.lap...@versailles.inra.fr wrote: Hi There's a bug in the galaxy-dist/lib/tool_shed/galaxy_install/install_manager.py file. line 79 : dir can be equal to None, but os.path.join() raises an error with an arg equal to None To fix it : if dir == None: dir = '' nico Le 07/07/2014 14:38, Kandalaft, Iyad a écrit : That error sounds like it's trying to merge the current working directory with the path supplied in the CHMOD tag. current_dir = os.path.abspath( os.path.join( work_dir, dir ) ) -- join work_dir with dir You have 2 possibilities to try: 1. Eliminate the $INSTALLDIR from the path in the XML if you're already in that path 2. Changedir to the path before trying the CHMOD tag and use . as the path Iyad Kandalaft Microbial Biodiversity Bioinformatics Agriculture and Agri-Food Canada | Agriculture et Agroalimentaire Canada 960 Carling Ave.| 960 Ave. Carling Ottawa, ON| Ottawa (ON) K1A 0C6 E-mail Address / Adresse courriel iyad.kandal...@agr.gc.ca Telephone | Téléphone 613-759-1228 Facsimile | Télécopieur 613-759-1701 Teletypewriter | Téléimprimeur 613-773-2600 Government of Canada | Gouvernement du Canada -Original Message- From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Nicolas Lapalu Sent: Monday, July 07, 2014 5:57 AM To: galaxy-dev@lists.bx.psu.edu Subject: [galaxy-dev] Toolshed : tool_dependency xml, action problem Hi all, I'm trying to package my tool wrapper to release it in the ToolShed. I need to download the associated binary file and change the permissions. I wrote a tool_dependency xml file with a downlod_binary and chmod actions. My tool_dependency_dir is well defined in my universe file. The download is ok, but I get an error during the chmod action. So I tried with your example, faToTwoBit (https://wiki.galaxyproject.org/DownloadingBinaries), and I get the same bug. Do you know the problem ? Do I need to configure someting else ? log : tool_shed.util.tool_dependency_util DEBUG 2014-07-07 11:28:34,346 Updating an existing record for version 0.0.1 of tool dependency faToTwoBit for revision 5ac8ed26842c of repository mytool by updating the status from Never installed to Installing. tool_shed.galaxy_install.tool_dependencies.recipe.step_handler DEBUG 2014-07-07 11:28:34,374 Attempting to download from http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64/faToTwoBit to bin 127.0.0.1 - - [07/Jul/2014:11:28:36 +0200] POST /admin_toolshed/repository_installation_status_updates HTTP/1.1 200 - http://127.0.0.1:8080/admin_toolshed/prepare_for_install; Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140610 Firefox/24.0 Iceweasel/24.6.0 tool_shed.galaxy_install.install_manager ERROR 2014-07-07 11:28:39,038 Error installing tool dependency faToTwoBit version 0.0.1. Traceback (most recent call last): File /home/nlapalu/Galaxy/galaxy-dist/lib/tool_shed/galaxy_install/install_manager.py, line 110, in install_and_build_package_via_fabric tool_dependency = self.install_and_build_package( app, tool_shed_repository, tool_dependency, actions_dict ) File /home/nlapalu/Galaxy/galaxy-dist/lib/tool_shed/galaxy_install/install_manager.py, line 79, in install_and_build_package current_dir = os.path.abspath( os.path.join( work_dir, dir ) ) File /usr/lib/python2.7/posixpath.py, line 75, in join if b.startswith('/'): AttributeError: 'NoneType' object has no attribute 'startswith' Thanks for your help; nico ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Toolshed roadmap
Hello Eric, The Tool Shed wiki is currently undergoing significant changes which will hopefully result in an improved set of documentation. This is a high priority between now and October ( see https://trello.com/c/Gg0jnll7/191-wiki-improvements and https://trello.com/c/uHyAaCLT/176-tasks-for-completion-by-october-2014). In the meantime, you may find my blog posts to be helpful ( see http://gregvonkuster.org ) Greg Von Kuster On Jul 15, 2014, at 4:00 PM, Eric Rasche rasche.e...@yandex.ru wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi Greg, Sorry about that, toolshed development hasn't been very visible in my view of galaxy thus far, so it seemed like asking someone who knew would be the fastest route here. I'll ensure my future inquires go to galaxy-dev. On 07/15/2014 02:18 PM, Greg Von Kuster wrote: Hello Eric, Please use Biostar or the Galaxy dev mail list rather than individual email addresses in order to ensure faster responses. Pleasse see my responses below. On Jul 14, 2014, at 10:20 AM, Eric Rasche rasche.e...@yandex.ru wrote: Howdy Greg, Since there isn't really a public/published roadmap and you're the toolshed guy, I'm emailing to see if I can find out what's in the toolshed's future. The tool shed Trello board at https://trello.com/b/vHqpKpZF/galaxy-tool-shed is the roadmap for the Tool Shed. Just like the Galaxy Development Trello board, you should be able to add new cards to the Inbox, and vote on existing cards to raise their priority. Please let us know if you are not able to do these things. I was not aware that a trello board even existed for the toolshed, that would have answered some of these. (It might be good to have it mentioned more than once on the wiki, perhaps under Toolshed and not just under Issues) Specifically, I'm wondering if there will be API calls for the following things: - creating repositories I've added the following Trello card for the above feature. However, it is currently possible to create new repositories via the API by importing a repository capsule (i.e., ~/api/repositories/import_capsule). https://trello.com/c/cUUWeBCi/234-enhance-the-api-to-enable-creation-of-new-repositories - obtaining a list of categories The above was introduced in the June 2, 2014 release, so, for example, the following will work. https://testtoolshed.g2.bx.psu.edu/api/categories Are any of these API changes documented outside of mercurial/DevNewsBriefs (trello)? My go-to locations for documentation are the wiki and readthedocs. Neither https://wiki.galaxyproject.org/ToolShedApi or http://galaxy-dist.readthedocs.org/en/latest/lib/galaxy.webapps.tool_shed.api.html has up-to-date documentation. Is there another place I should be looking for static, complete toolshed API information? - obtaining category information within the /api/repositories call I've added the following Trello card for this feature. https://trello.com/c/n7P0GE1P/235-enhance-the-api-repositories-feature Thanks! Cheers, Eric Cheers, Eric -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (GNU/Linux) iQIcBAEBAgAGBQJTxYhAAAoJEMqDXdrsMcpV9K0QAKd/KIFxXLmfx1gj208GwCCl WCugrb7F1jKvGMTz+IkyqaFlSsC+d5PjFskETEPIlUjIsXFQ1EwHKQWDVTDRjwEV r9H0huCwfhvS0xz4wlHcrkHBimWCktlfqnbbCzmAkmNuqQ3e2zPquvVajQlcUFQM rVjbn45+c6C2ewy8JeOB9bJUG4JbtIH3jeFWreu8UiBFn8WcfkAsEigDoluLzgpg JuakNPrMuzrVqlsvfJw+Pv0xbYntplrfTOAV1G5F1Qbyc0Zb9KoH5cHKTfIf8bNS E4g2Djvr/ow+ofsf88floBjTfsLUy6ck1/LaAgbwyDIY8b4IVrv3SgtyL6mLx+FX 8K/WvbPgQH7adDykUkMLRwfQcrkTdws1uGYf6EQwqGr0P1KqDcCQufH8y9MH24If Sn51+UlceOR2uQEIi7x8Xm6/tPsIcN+MUrWb//FzrLDitHaGLCZRLGRxrQyACkXS 9IfM9tz6aqR6b8OrEcUGo4E+6azpHsQgi2iTnQz2tVT4g1viKYmwKoPTT58Cm6KF 7h32taJMGabZm618bB/DOT8jxxfZ2VfEfXgMFZaSpkIbMsXnnaS5Pivtwxl7R1nP Sw73Pm2sBNsrnpPjQph39jS2rj1Oqb9PbH8k4EJl36fNiZAksNsi7n5cOCka1P66 2lyxcFTRAwic0CwTJ5Vx =S+xu -END PGP SIGNATURE- ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Control on versioning in toolshed tools
Hello Eric, The public Galaxy test and main Tool Sheds are for sharing validated, functionally correct tools with the Galaxy community, not for developing them. As you've discovered, developing tools using the public Tool Sheds results in undesired behavior since you have restricted control over what can easily be eliminated from sharing. The primary reason for this is the Tool Shed's goal of reproducible analyses in galaxy. Any version of any Galaxy utility uploaded to the public Tool Shed will always be available for installation into Galaxy. This is the behavior you've seen when changing dependency versions, and similar behavior results when any Galaxy utility (tools, custom datatypes, Galaxy Data Managers, etc) version is changed. Instead of using the public Galaxy Tool Sheds for development, you should be using a local development Tool Shed as a complementary component to your local Galaxy development environment. The general steps in the development process are: Set up a local development Tool Shed - see the Hosting Your Own Tool Shed for Developing Galaxy Tools and Utilities section of: http://gregvonkuster.org/galaxy-tool-shed-introduction/ Export repositories containing validated tool dependencies from the main Galaxy Tool Shed into a capsule - see http://gregvonkuster.org/galaxy-tool-shed-leveraging-community-contributions-repository-capsules/ Import the capsule into your local Tool Shed. Create an empty repository in your local Tool Shed to contain the new Galaxy tool. Develop your tool in your local Galaxy environment, using the local Tool Shed as a source code repository if necessary. When development is finished, make sure the finished tool is available in hyour local Tool Shed's repository and execute your local Tool Shed's Install and Test framework to validate the tool - see the Using the Install and Test Framework With a Local Tool Shed section of: http://gregvonkuster.org/galaxy-tool-shed-certifying-repositories-install-test-framework/ Export the repository containing the tool from your local Tool Shed. Import the repository containing the tool into the test and main Galaxy Tool Sheds for additional validation and sharing. Greg Von Kuster On Jun 24, 2014, at 8:15 AM, Eric Kuyt eric.ku...@wur.nl wrote: Hi All, I am playing around with putting a tool in testtoolshed. Now when changes to dependency versions are detected, the toolshed detects a new version and a dropdown is created. but sometimes I do not want this behavior when the first version was erroneous for example. I tried hg strip on the repository and pushing it back to the testtoolshed but sadly it didn't result in a clean repository but a multi-headered mess. Is there a way to cleanup the remote repository and start over. And what would be a cleaner way to develop tools on a toolshed still making use of repository dependencies? Thanks, Eric ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Import of Capsules failing on local toolshed instance
Thanks for the corrections Will, and I'm glad this worked for you. Greg Von Kuster On Jun 18, 2014, at 12:22 PM, Will Holtz who...@lygos.com wrote: Hi Greg, I used the toolshed bootstrapping script and was able to get capsules from testtoolshed to import. Here are a couple of notes that you may find interesting. 1) In a few places you reference the directory ~/lib/tool_shed/scripts/api/bootstrap_from_toolshed/ and these should be changed to ~/lib/tool_shed/scripts/bootstrap_from_toolshed/ 2) I was worried that 'use_remote_user=True' was going to negatively impact the bootstrap process, as accounts are being created that don't exist in LDAP. However, there didn't appear to be any such negative interaction. In user_info.xml I used my email address that maps to my LDAP login (who...@lygos.com), a long gibberish password, and wholtz for my username. Thank you for your help Greg. -Will On Tue, Jun 17, 2014 at 7:51 PM, Greg Von Kuster g...@bx.psu.edu wrote: Hello Will, On Jun 17, 2014, at 6:06 PM, Will Holtz who...@lygos.com wrote: It looks like Dave fixed this issue in 4c58aa19a3d7. Thank you! However I am still having import issues. I am now getting the message: Archive of repository package_bowtie_0_12_7 owned by devteam Import failed: repository owner devteam does not have an account in this Tool Shed. This is on a local toolshed running 9b78595ec11 where I am performing the input from an admin account. I'm guessing the issue is that I have 'use_remote_user=True' for LDAP authentication and that means that a devteam account cannot be automatically created to allow this capsule to be added without modification. Sorry you're still running into problems. No regular development or testing is done using the use_remote_user setting due to resource limitations. Perhaps on import of a capsule (by an administrator) they could be given the option of preserving the existing owners or re-assigning ownership to an existing user (defaulting to self)? This would be non-trivial, and probably would introduce fragility into the process. However, I believe there is a solution (see my next comment) although I haven't tested it with the use_remote_user setting. Of course, what I really want is inter-toolshed dependancies. Maybe I'm missing something, but I'm finding it quite painful just to get a tool development environment setup that makes use of any existing repositories. There was some work done in this area in the June 2, 2014 release. Here is some information for more easily setting up a local development Tool Shed that use new features introduced in the release. Hopefully this will help. Greg Von Kuster Bootstrapping a New Development Tool Shed The June 2, 2014 release introduces the ability to easily bootstrap a new development Tool Shed to prepare it for importing a repository capsule whose contained repositories can be used as the foundation for developing new Galaxy tools and other utilities. This development Tool Shed can be treated as a component in a multi-step process that simplifies and streamlines Galaxy tool development and validation in the local Tool Shed and moving the validated repositories into the test or main public Galaxy Tool Sheds. Tool Shed framework enhancements included in the June 2, 2014 release support this overall process, which will be explained fully in a future article. Here we’ll restrict our discussion to highlights of the enhancements. Several files are included in a new directory named ~/lib/tool_shed/scripts/api/bootstrap_from_toolshed. The file named user_info.xml.sample should be copied to a file with the same name, but eliminating the .sample extension (i.e., user_info.xml). The information in this file is used to automatically create a new “admin” user account in your local development Tool Shed. This should be the account you use in the test and main public Galaxy Tool Sheds if you plan to export your work from your development Tool Shed and import it into one or both of the public Tool Sheds. If you plan to use this new bootstrapping process, make sure your local development Tool Shed environment is pristine: The hgweb.config file must be empty or missing (it will automatically get created if it doesn’t exist) and the configured location for repositories must be an empty directory. The configured database must be new and not yet migrated. Make sure the ~/lib/tool_shed/scripts/api/bootstrap_from_toolshed/user_info.xml file contains the desired account information. The ~/run_tool_shed.sh script, used for starting up a Tool Shed, has been enhanced to enable this bootstrapping process by using a new -bootstrap_from_tool_shed flag. Here’s an example. %sh run_tool_shed.sh -bootstrap_from_tool_shed http://toolshed.g2.bx.psu.edu The above example will initialize a local development Tool Shed (here we’ll assume
Re: [galaxy-dev] Import of Capsules failing on local toolshed instance
Hello Will, On Jun 17, 2014, at 6:06 PM, Will Holtz who...@lygos.com wrote: It looks like Dave fixed this issue in 4c58aa19a3d7. Thank you! However I am still having import issues. I am now getting the message: Archive of repository package_bowtie_0_12_7 owned by devteam Import failed: repository owner devteam does not have an account in this Tool Shed. This is on a local toolshed running 9b78595ec11 where I am performing the input from an admin account. I'm guessing the issue is that I have 'use_remote_user=True' for LDAP authentication and that means that a devteam account cannot be automatically created to allow this capsule to be added without modification. Sorry you're still running into problems. No regular development or testing is done using the use_remote_user setting due to resource limitations. Perhaps on import of a capsule (by an administrator) they could be given the option of preserving the existing owners or re-assigning ownership to an existing user (defaulting to self)? This would be non-trivial, and probably would introduce fragility into the process. However, I believe there is a solution (see my next comment) although I haven't tested it with the use_remote_user setting. Of course, what I really want is inter-toolshed dependancies. Maybe I'm missing something, but I'm finding it quite painful just to get a tool development environment setup that makes use of any existing repositories. There was some work done in this area in the June 2, 2014 release. Here is some information for more easily setting up a local development Tool Shed that use new features introduced in the release. Hopefully this will help. Greg Von Kuster Bootstrapping a New Development Tool Shed The June 2, 2014 release introduces the ability to easily bootstrap a new development Tool Shed to prepare it for importing a repository capsule whose contained repositories can be used as the foundation for developing new Galaxy tools and other utilities. This development Tool Shed can be treated as a component in a multi-step process that simplifies and streamlines Galaxy tool development and validation in the local Tool Shed and moving the validated repositories into the test or main public Galaxy Tool Sheds. Tool Shed framework enhancements included in the June 2, 2014 release support this overall process, which will be explained fully in a future article. Here we’ll restrict our discussion to highlights of the enhancements. Several files are included in a new directory named ~/lib/tool_shed/scripts/api/bootstrap_from_toolshed. The file named user_info.xml.sample should be copied to a file with the same name, but eliminating the .sample extension (i.e., user_info.xml). The information in this file is used to automatically create a new “admin” user account in your local development Tool Shed. This should be the account you use in the test and main public Galaxy Tool Sheds if you plan to export your work from your development Tool Shed and import it into one or both of the public Tool Sheds. If you plan to use this new bootstrapping process, make sure your local development Tool Shed environment is pristine: The hgweb.config file must be empty or missing (it will automatically get created if it doesn’t exist) and the configured location for repositories must be an empty directory. The configured database must be new and not yet migrated. Make sure the ~/lib/tool_shed/scripts/api/bootstrap_from_toolshed/user_info.xml file contains the desired account information. The ~/run_tool_shed.sh script, used for starting up a Tool Shed, has been enhanced to enable this bootstrapping process by using a new -bootstrap_from_tool_shed flag. Here’s an example. %sh run_tool_shed.sh -bootstrap_from_tool_shed http://toolshed.g2.bx.psu.edu The above example will initialize a local development Tool Shed (here we’ll assume its URL is http://localhost:9009) by bootstrapping from the main public Galaxy Tool Shed. The bootstrapping process will perform the following actions in the order listed. Ensure the Tool Shed environment is pristine (e.g., empty hgweb.config file and new database that has not been migrated). Copy all .sample files configured in run_tool_shed.sh. Run the database migration process. Execute the script ~/lib/tool_shed/scripts/bootstrap_tool_shed/create_user_with_api_key.py ~/tool_shed_wsgi.ini to create a new user and an associated API key using the information defined in ~/lib/tool_shed/scripts/bootstrap_tool_shed/user_info.xml. Automatically modify the ~/tool_shed_wsgi.ini file to configure the above user as an admin_user for the Tool Shed. Start the Tool Shed web server. Execute the script ~/lib/tool_shed/scripts/api/create_users.py -a api_key -f http://toolshed.g2.bx.psu.edu -t http://localhost:9009. Execute the script ~/lib/tool_shed/scripts/api/create_categories.py -a api_key -f http://toolshed.g2.bx.psu./edu -t http://localhost:9009
Re: [galaxy-dev] Toolshed upload error message
Hi Vipin, Can you elaborate on the error you are seeing? Thanks, Greg Von Kuster On Jun 10, 2014, at 8:07 PM, Vipin TS vipin...@gmail.com wrote: Hi Greg, When I trying to upload a next release version o my https://toolshed.g2.bx.psu.edu/repos/vipints/fml_gff3togtf converter program to Community Toolshed, I am getting an internal error message. I am uploading a tar.gz file to the page https://toolshed.g2.bx.psu.edu/ but this fails. Could you please pass the error message or let me how I can add new files to the repository and delete the old version. Thanks, Vipin | Rätsch Lab ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] gatk2 wrapper issue
Hello Geert, This issue was corrected yesterday in change sets https://bitbucket.org/galaxy/galaxy-central/commits/96fdb9168f28b2ffa463195d084c9278efb2465a and https://bitbucket.org/galaxy/galaxy-central/commits/58a057a02b896ae6d32571f7141f46ccb9006e54, which are currently available in both the default and stable branches of the galaxy-central repository. The fix will be automatically pushed to the galaxy-dist repository at some point, but I'm not sure when - I believe it will be some time today. If you are tracking the stable branch on the galaxy-central repository, you can: hg pull hg update stable If you are tracking the galaxy-dist repository, you can get the fix later today. Thanks for reporting this, and sorry for the inconvenience. Greg Von Kuster On Jun 4, 2014, at 6:25 AM, Geert Vandeweyer geert.vandewey...@uantwerpen.be wrote: Hi, I installed the GATK2 wrapper from iuc on the lastest galaxy version. It fails to create the GATK2_PATH and GATK2_SITE_OPTIONS structures under the /tool_dependency_dir/environment_settings/ location, without a clear error (says 'never installed'). If I create the files manually, I can see them through : Admin - Manage installed toolshed repos - gatk2 - missing tool dependencies - GATK2_PATH If I click the hyperlink for GATK2_PATH here (this is the install missing dependency page), I can expand and show the GATK2_PATH / env.sh file ghfbhgbd.png Still, running GATK2 tools (eg UG, gives me in the logs: galaxy.jobs.handler INFO 2014-06-04 12:18:07,333 (78998) Job dispatched galaxy.tools.deps WARNING 2014-06-04 12:18:07,928 Failed to resolve dependency on 'gatk2', ignoring galaxy.tools.deps WARNING 2014-06-04 12:18:07,971 Failed to resolve dependency on 'GATK2_PATH', ignoring galaxy.tools.deps WARNING 2014-06-04 12:18:07,974 Failed to resolve dependency on 'GATK2_SITE_OPTIONS', ignoring and tool error: Unable to access jarfile /GenomeAnalysisTK.jar For my mutect_wrapper in test.toolshed, which I initially copied from the gatk2 wrapper, the exact same setup seems to work. Any help would be appreciated. Best, Geert -- Geert Vandeweyer, Ph.D. Department of Medical Genetics University of Antwerp Prins Boudewijnlaan 43 2650 Edegem Belgium Tel: +32 (0)3 275 97 56 E-mail: geert.vandewe...@ua.ac.be http://ua.ac.be/cognitivegenetics http://www.linkedin.com/in/geertvandeweyer ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] devteam packages
Hi Jan, The packages are currently maintained only by the devteam as necessary on a per-repository basis in the Tool Shed itself. The following Trello card is for creating bitbucket repositories for the various Tool Shed repositories owned by the devteam account, but this work won't be done until some time after the upcoming Galaxy Community Conference. If you find issues with any of the devteam packages, let us knwo and we'll get them corrected asap. https://trello.com/c/WdShmMld/159-create-repositories-in-bitbucket-for-all-repositories-in-the-main-tool-shed-that-are-owned-by-devteam Thanks, Greg Von Kuster On Jun 4, 2014, at 9:59 AM, Jan Kanis jan.c...@jankanis.nl wrote: Hi all, Where are the toolshed packages owned by devteam maintained? The galaxy sources don't seem to include those tool shed packages and I was unable to find a related repository on github or a similar place. Jan ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Unable to access admin after install EMBOSS 5 from galaxy test repo
Hello Cristian, I've just instaled the emboss_5 repository along with all dependencies from the test Tool Shed, so I'm not sure what configuration in your Galaxy environment may be causing the behavior you're seeing. Currently the only way to uninstall a repository is via the Galaxy admin UI. There is a Trello card here to enhance the Galaxy API to support uninstalling a repository, but the featue is not yet implemented: https://trello.com/c/4VktpvTd/207-enhance-the-galaxy-api-for-the-tool-shed-to-allow-uninstalling-a-repository Sorry for the inconvenience! Greg Von Kuster On Jun 1, 2014, at 2:32 PM, Cristian Alejandro Rojas alejandro.0...@gmail.com wrote: Hello all, I've select the emboss 5 suite to install from Galaxy test repo. After install I cant access to the admin panel. I think that is beacuse the json value is not complete (len(value):65535), and therefore the json decoder cant decode the value. In attachments I have included the debug data generated by galaxy and the file containing the value variable. Can anybody bring me some help? Is there some way to remove emboss without using the admin panel? Thanks in advance, best regards. Cristian Alejandro Rojas Quintero error_galaxy.txterror_galaxy.xmlerror_galaxy_extra_data.txtjson_value.txt___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Main ToolShed wrong report: Repository does not have a test-data directory.
Hi Peter, I've looked into this as well and again believe it is caused by a problem in the buildbot configuration for the main Tool Shed's Install and Test framework. I've created the following Trello card for this, and we'll get it resolved when Dave B returns next week. https://trello.com/c/9mLrFhTJ/209-install-and-test-framework-does-not-locate-test-data-directory-for-repositories-on-main-tool-shed Greg Von Kuster On May 26, 2014, at 6:54 AM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi guys, I just noticed two of my tools on the main ToolShed were listed under Latest revision: missing tool tests unexpectedly. http://toolshed.g2.bx.psu.edu/view/peterjc/sample_seqs Test runs 2014-05-25 15:12:04 Automated test environment Tools missing tests or test data Tool id: sample_seqs Tool version: 0.0.1 Tool guid: toolshed.g2.bx.psu.edu/repos/peterjc/sample_seqs/sample_seqs/0.0.1 Missing components: Repository does not have a test-data directory. 2014-05-13 01:43:51 Automated test environment Tests that passed successfully Successful installations 2014-05-11 01:38:18 Automated test environment Tests that passed successfully Successful installations 2014-05-09 01:43:58 Automated test environment Tests that passed successfully Successful installations 2014-05-07 01:55:08 Automated test environment Tests that passed successfully Successful installations And, http://toolshed.g2.bx.psu.edu/view/peterjc/get_orfs_or_cdss Test runs 2014-05-25 15:10:53 Automated test environment Tools missing tests or test data Tool id: get_orfs_or_cdss Tool version: 0.0.5 Tool guid: toolshed.g2.bx.psu.edu/repos/peterjc/get_orfs_or_cdss/get_orfs_or_cdss/0.0.5 Missing components: Repository does not have a test-data directory. 2014-05-13 05:11:12 Automated test environment Tests that passed successfully Successful installations 2014-05-11 05:06:15 Automated test environment Tests that passed successfully Successful installations 2014-05-09 05:17:53 Automated test environment Tests that passed successfully Successful installations 2014-05-07 05:52:29 Automated test environment Tests that passed successfully Successful installations In both cases the most recent test run (2014-05-25) has failed in some way, and the ToolShed *wrongly* reports the repository does not have a test-data directory. Regards, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Old Tool Shed URL http://community.g2.bx.psu.edu/ dead
Hi peter, It seems that we have stopped redirecting the old Tool Shed URL http://community.g2.bx.psu.edu/ to the new Tool Shed URL http://toolshed.g2.bx.psu.edu/ . I'm not sure when this happened. Sorry for the inconvenience. Greg Von Kuster On May 21, 2014, at 9:37 AM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi Greg, That Trello card link doesn't work for me, but the old domain is currently not redirecting to the ToolShed. Regards, Peter On Wed, Mar 6, 2013 at 12:32 PM, Greg Von Kuster g...@bx.psu.edu wrote: Hi Peter, I've added this to the same Trello card. These URL redirects lie outside of my control, so I'll bug the right person to get them set up. https://trello.com/card/toolshed-galaxyproject-org-url-redirects/506338ce32ae458f6d15e4b3/697 Thanks! On Mar 6, 2013, at 6:52 AM, Peter Cock wrote: Hello all, Either I made a typo in some old README files, or the old URL for the Tool Shed http://community.g2.bx.psu.edu/ has stopped working. Could this redirect to the current Tool Shed address http://toolshed.g2.bx.psu.edu/ please? Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Old Tool Shed URL http://community.g2.bx.psu.edu/ dead
Peter, sorry, this one is out of my hands. I'm hoping Nate can answer this when he gets a chance. Greg Von Kuster On May 27, 2014, at 11:16 AM, Peter Cock p.j.a.c...@googlemail.com wrote: On Tue, May 27, 2014 at 3:39 PM, Greg Von Kuster g...@bx.psu.edu wrote: Hi peter, It seems that we have stopped redirecting the old Tool Shed URL http://community.g2.bx.psu.edu/ to the new Tool Shed URL http://toolshed.g2.bx.psu.edu/ . I'm not sure when this happened. Sorry for the inconvenience. Greg Von Kuster Can the old domain be restored to fix pre-existing 3rd party links? There could be some in published papers, certainly there are on the mailing list archives. Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] RFC: Citations for tools
Peter, this may be the Trello card you were thinking of. https://trello.com/c/kY7RCnd0/95-tool-shed-citation-for-tools-dois Greg Von Kuster On May 27, 2014, at 1:09 PM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi Eric, I was sure there was a Trello card for this, but I can't find it right now... I've pushed this idea on the mailing list before, and in person at the Galaxy Community Conference too. See also these threads: http://lists.bx.psu.edu/pipermail/galaxy-dev/2011-December/007873.html http://lists.bx.psu.edu/pipermail/galaxy-dev/2012-June/010178.html Are you familiar enough with the area of semantic web/linked data to know what would be the best XML based markup to use for embedding the citations? i.e. We should not reinvent the wheel here ;) Peter On Tue, May 27, 2014 at 5:54 PM, James Taylor ja...@jamestaylor.org wrote: Eric, I'm very much in favor of this feature, and particularly the idea of generating a list of citations from a history or workflow. I imagine the only thing to quibble about will be the syntax. There are already some efforts to represent bibtex in xml (e.g. https://github.com/Zearin/BibTeXML), however they have always struck me as overly verbose. -- jt On Tue, May 27, 2014 at 12:45 PM, Eric Rasche rasche.e...@yandex.ru wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I'd want to open up discussion on a feature I'd like to see. I'll try and implement it if I can find time this summer. I didn't see a trello card for anything like this yet, but please feel free to direct me there if I missed it. I'd like to see citations as a part of every tool. This would happen in the form of a citation block in the XML, which would contain sub-elements with text. These sub-elements could be based off of BibTeX, since they have existing specs for citing things: https://en.wikipedia.org/wiki/BibTeX These citations would then be accessible in the HTML generated tool pages, or via a View/Download button somewhere on the tool page. By storing as an XML tree, we could render these citations as BibTeX entries for the LaTeX users, and I believe there are ways to convert BibTeX to EndNote XML and so on. This could be extended for use in workflows so that when you run workflows, somehow a list of citations for all tools used could be generated. Anyone have thoughts or opinions on this? Using the example bibtex entry from the wikipedia page: @Book{abramowitz+stegun, author= Milton {Abramowitz} and Irene A. {Stegun}, title = Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables, publisher = Dover, year = 1964, address = New York, edition = ninth Dover printing, tenth GPO printing } I imagine it'd look like the following in a real-life tool: tool ... citation type=book authorMilton Abramowitz and Irene A. Stegun/author titleHandbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables/title publisherDover/publisher year1964/year addressNew York/address editionninth Dover printing, tenth GPO printing/edition /citation /tool Cheers, Eric - -- Eric Rasche Programmer II Center for Phage Technology Texas AM University College Station, TX 77843 404-692-2048 e...@tamu.edu rasche.e...@yandex.ru -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (GNU/Linux) iQIcBAEBAgAGBQJThMEUAAoJEMqDXdrsMcpVe9AP/1BjPbP5JdK6KDybOeV2ElvC jvmUGetAjdQzkKO1Ikeuxb46yp0j4abAGGG92AccxlBYALsT3jsPv5dYjm505vcU IwlfTBE7gc5y19x3zx1CuAd7PB11tryODz2LwXNKI75f39bNdX6Qe5aA74Vn/k8a FBeFL/EvkZwI8Z6CvuTGtZrEHvDXVvTFoxMX875f59g2vNy7W8Fh5wQCQ8qgiogi 0SWGIGZ6OJdqX60h2hOjJ06NhzBcTsjQea+AwTo5wAkyGPFJy4DV78jKOvsd5PMG qkk4U6gBcmcFolCeItlLPa/C4uiyOK2wVWHyTnEeKjdpD6dwLuemRmwtZrModbQu PPPFkKzQhUnh5Wgq8TIP4kSc2t1/jSpdV8MHqM0piqOFnR52fExPrwLn82WEFcp8 tXfHHTaZj+6gL+IGswdAgqvxk1V+6QTjH57igWmYtcENpIDbhZVBh+HOD8SS2qdv ohwFg/Vn8VrCeTGGehNSBvkam0HXjXXq7mZ8W4xrOS8vxv8HPAFZHze2sPlUG/Ob WeOdiLgD2HCH95SXYDDBNFG6HwFkLETA3DjpfvgHqPdaLmEfIZW+ks1WF9RG+Pmo jhbWH8OmtTk5A/Zh63uRowMaDe7QSCruPIfIwlrgqLozOj8htMyJPBIW9oZGGkqC 7Vu5UFeL8gcigRmQv5NA =ceAo -END PGP SIGNATURE- ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists
Re: [galaxy-dev] Repository installation on multiple servers
Hello Iyad and Brad, Hello Iyad, Yes, this is the current expected behavior. There is a Trllo card here that tracks progress on this issue. http://trello.com/c/B0pV80d0/1281-messaging-and-task-queue This work is currently in progress with a baseline implementation available in the upcoming Galaxy release (currently scheduled for June 2). I'm not involved in this work, so I'm not quite sure whether this baseline implementation will handle comunication between Galaxy web front-ends like you have (and like we have for our public Galaxy instances), but this information should be clarified with the upcoming release notes. Greg Von Kuster On May 23, 2014, at 8:56 AM, Langhorst, Brad langho...@neb.com wrote: I had the same concern, and heard that this is being worked on. Meanwhile I’m using a rolling restart method from pjbriggs https://github.com/pjbriggs/galaxy-admin-utils/ I’m pretty happy with this (desipite the 2 minute restart per worker) because users do not see any interruption during restarts as they did before. Brad -- Brad Langhorst, Ph.D. Applications and Product Development Scientist On May 23, 2014, at 8:46 AM, Kandalaft, Iyad iyad.kandal...@agr.gc.ca wrote: Hi Everyone This is my first time on the mailing list. I am the administrator for a production installation of Galaxy at Agriculture and Agri-Food Canada in Ottawa, Canada. Currently, I have a galaxy configured to start 3 web servers and 3 handlers. When installing a tool from a toolshed, only one of the three servers sees the installation (the one that processed the installation request). To resolve the issue, galaxy needs to be restarted, which reloads all the tools. Is this the expected behaviour when a proxy server (apache) load balances 3 galaxy web servers? If not, where can I start diagnosing the problem? Your assistance is much appreciated. Thank you. Regards, Iyad Kandalaft Bioinformatics Programmer Microbial Biodiversity Bioinformatics Science Technology Branch Agriculture Agri-Food Canada iyad.kandal...@agr.gc.ca | (613) 759-1228 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] uninstalling tools via web api
Hello Evan, When installing tools into Galaxy from the Tool Shed, you are actually installing Tool Shed repositories that contain tools, among other things (e.g., custom datatypes, exported workflows and other Galaxy utilities). The Galaxy UI does provide the ability to unistall, but it is at the repository level, just like installing is. You'll need to use the Manage installed tool shed repositories option from the Admin perspective. The list of installed repositories each provide a pop-up menu to that includes options to uninstall. Greg Von Kuster On May 23, 2014, at 1:34 PM, Evan Bollig boll0...@umn.edu wrote: I notice that the galaxy web api for installing tools from the toolshed does not include any entrypoint for uninstalling tools. This must be intentional, but what was the motivation to leave this out? Cheers, -Evan Bollig Research Associate | Application Developer | User Support Consultant Minnesota Supercomputing Institute 599 Walter Library 612 624 1447 e...@msi.umn.edu boll0...@umn.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] new_tool_panel_section_label web api
Hello Evan, You're referring to an object we call a ToolSectionLabel, and I don't believe that the Galaxy framework supports creating these objects automatically in real time. You have to manually add them to one of your tool panel configs (e.g., tool_conf.xml, shed_tool_conf.xml, etc) via an editor. For example, the label tag in the following example tool_conf.xml file will display a label with the text Get some data… in the Galaxy tool panel. ?xml version=1.0? toolbox label id=get_data text=Get some data... version= / section id=getext name=Get Data version= tool id=upload1 / /section /toolbox Since the Galaxy framework doesn't support real time addition of section labels in the tool panel, the Tool Shed installation process doesn't either. However, if the ability to do this is needed, we can open a Trello card for it. Just let us know. Thanks, Greg Von Kuster On May 23, 2014, at 1:33 PM, Evan Bollig boll0...@umn.edu wrote: I am using the new_tool_panel_section_label option when installing tools from the toolshed (via scripts/api/install_tool_shed_repository.py. It works, but I'm curious if its producing the wrong result. See the attached image. I have installed a bunch of tools with new_tool_panel_section_label=Other (Main Tool Shed). I see in the tool_conf.xml that the tool is installed under a section, and the section name gives the menu option its string (e.g., Other (Main Tool Shed)). A label on the other hand is contained within a section and gives a subheading string (see expected_section_and_label.png). If the parameter new_tool_panel_section_label specifies the section_name, what option exists to specify the label? Or perhaps was the *_label option intended to provide the subheading and another option (e.g., new_tool_panel_section_name supposed to exist)? Cheers, -Evan Bollig Research Associate | Application Developer | User Support Consultant Minnesota Supercomputing Institute 599 Walter Library 612 624 1447 e...@msi.umn.edu boll0...@umn.edu toolshed_installed.pngexpected_section_and_label.png___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] uninstalling tools via web api
Hi Evan, Sorry I somehow missed that in your original message - here is a Trello card for enhancing the Galaxy API for the Tool Shed. I'll get to this as soon as I can. https://trello.com/c/4VktpvTd/207-enhance-the-galaxy-api-for-the-tool-shed-to-allow-uninstalling-a-repository Greg Von Kuster On May 23, 2014, at 2:04 PM, Evan Bollig boll0...@umn.edu wrote: Hey Greg, I see that option in the galaxy web UI, but I am working from the command line and scripting against your toolshed web api. Its the api that is missing the uninstall option for repositories/tools. https://wiki.galaxyproject.org/ToolShedApi Cheers, -E -Evan Bollig Research Associate | Application Developer | User Support Consultant Minnesota Supercomputing Institute 599 Walter Library 612 624 1447 e...@msi.umn.edu boll0...@umn.edu On Fri, May 23, 2014 at 12:40 PM, Greg Von Kuster g...@bx.psu.edu wrote: Hello Evan, When installing tools into Galaxy from the Tool Shed, you are actually installing Tool Shed repositories that contain tools, among other things (e.g., custom datatypes, exported workflows and other Galaxy utilities). The Galaxy UI does provide the ability to unistall, but it is at the repository level, just like installing is. You'll need to use the Manage installed tool shed repositories option from the Admin perspective. The list of installed repositories each provide a pop-up menu to that includes options to uninstall. Greg Von Kuster On May 23, 2014, at 1:34 PM, Evan Bollig boll0...@umn.edu wrote: I notice that the galaxy web api for installing tools from the toolshed does not include any entrypoint for uninstalling tools. This must be intentional, but what was the motivation to leave this out? Cheers, -Evan Bollig Research Associate | Application Developer | User Support Consultant Minnesota Supercomputing Institute 599 Walter Library 612 624 1447 e...@msi.umn.edu boll0...@umn.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] TestToolShed NameError: global name 'suc' is not defined
Hi Peter, I've been doing some refactoring in the central branch which caused this issuee. It's been fixed in 13582:581e7c8e353e, which is now running on the test tool shed, ao this bad behavior should be resolved. Thanks so much for reporting this. Greg Von Kuster On May 22, 2014, at 8:39 AM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi all, I just hit a bug on the TestToolShed, (1) Goto http://testtoolshed.g2.bx.psu.edu/ (2) Login (3) Click Latest revision: failing tool tests or similar links (4) Observer error page, NameError: global name 'suc' is not defined This is working on the main ToolShed. Regards, Peter -- Internal Server Error Galaxy was unable to successfully complete your request URL: http://testtoolshed.g2.bx.psu.edu/repository/browse_my_writable_repositories_with_failing_tool_tests Module galaxy.web.framework.middleware.error:149 in __call__ app_iter = self.application(environ, sr_checker) Module paste.debug.prints:106 in __call__ environ, self.app) Module paste.wsgilib:543 in intercept_output app_iter = application(environ, replacement_start_response) Module paste.recursive:84 in __call__ return self.application(environ, start_response) Module paste.httpexceptions:633 in __call__ return self.application(environ, start_response) Module galaxy.web.framework.base:132 in __call__ return self.handle_request( environ, start_response ) Module galaxy.web.framework.base:190 in handle_request body = method( trans, **kwargs ) Module galaxy.webapps.tool_shed.controllers.repository:239 in browse_my_writable_repositories_with_failing_tool_tests return self.my_writable_repositories_with_failing_tool_tests_grid( trans, **kwd ) Module galaxy.web.framework.helpers.grids:80 in __call__ query = self.build_initial_query( trans, **kwargs ) Module tool_shed.grids.repository_grids:1083 in build_initial_query grids_util.filter_by_latest_downloadable_changeset_revision_that_has_failing_tool_tests( trans, repository ) Module tool_shed.grids.util:89 in filter_by_latest_downloadable_changeset_revision_that_has_failing_tool_tests repository_metadata = get_latest_downloadable_repository_metadata_if_it_includes_tools( trans, repository ) Module tool_shed.grids.util:196 in get_latest_downloadable_repository_metadata_if_it_includes_tools repository_metadata = get_latest_downloadable_repository_metadata( trans, repository ) Module tool_shed.grids.util:180 in get_latest_downloadable_repository_metadata latest_downloadable_revision = suc.get_previous_metadata_changeset_revision( repository, repo, tip_ctx, downloadable=True ) NameError: global name 'suc' is not defined extra data full traceback text version This may be an intermittent problem due to load or other unpredictable factors, reloading the page may address the problem. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] error: flexbar dependency is not installed by the toolshed
Hello Rui, The problem here is that flexbar repository on the main Galaxy Tool Shed does not define a relationship between the flexbar tool contained within the repository and a recipe for installing the flexbar binary dependency. The flexbar tool config includes the following requirements tag set: requirements requirement type=binary version=2.4flexbar/requirement /requirements But there is no information about how to install the flexbar binary. The tool should have a a relationship to a recipt for installing the flexbar binary using the approach described here: https://wiki.galaxyproject.org/ToolsWitDependenciesInSeparateRepositories You can contact the tool author to report this problem. In the meantime, to use the tool, you'll have to install the flexbar dependency manually and make sure it can be found in the environment in which the tool is executed. Greg Von Kuster On May 19, 2014, at 10:07 PM, ruiwang.sz ruiwang...@gmail.com wrote: Hi Guys, I tried to install flexbar from toolshed, but upon running, error said it could not find it. I checked and found idtoolshed.g2.bx.psu.edu/repos/jtilman/flexbar/flexbar/2.4/id in the config file, but this path does not exist. Is it that I need to install the dependency on my own? I had the impression that toolshed will do it for me. Please correct me if I'm wrong. Thanks, Rui ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] The main Tool Shed is now tracking the next-stable branch in preparation for the upcoming Galaxy release
Hello all, The upcoming Galaxy release is currently scheduled for Monday, June 2. In preparation for this release, the main Galaxy Tool Shed is now tracking the next-stable branch and is currently running changeset 13542:11403745e7ce. The main Tool Shed will be regularly updated with any additions made to the next-stable branch between now and the upcoming Galaxy release. Please let us know if you encounter any issues. Thanks very much, Greg Von Kuster ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] New tool listed under Latest revision: failing tool tests yet not tested
Hi Peter, I doscovered an error in the query filters that are used for rendering these various Latest revision: lists. The problem was that the filters were not correctly determining whether the revisions had been tested yet by the install and test framework. This problem should be fixed in 13496:26b9fe985ee7, which is now running on the test tool shed. Thanks for reporting this problem, and please let us know if you discover any other problems related to this. Greg Von Kustet On May 16, 2014, at 7:22 AM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi Dave et al, Yesterday I uploaded a new tool to the TestToolShed: http://testtoolshed.g2.bx.psu.edu/view/peterjc/blast_rbh Development repository: https://github.com/peterjc/galaxy_blast/tree/master/tools/blast_rbh It is currently listed under Latest revision: failing tool tests, Name Latest Installable Revision Owner blast_rbh 1 (2014-05-15) peterjc On viewing the tool, there are no test results at all (yet). Is this a corner case false-positive, or was there really a failed test run which is not being shown? Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Packaging pycurl on toolshed
Hello Saket, I'm not a system-level expert, so please ignore my attempt at helping if it is incorrect. Your recipe for installing and compiling pycurl may be assuming that certain dependencies exist in the Galaxy environment into which it is being installed for testing, but they do not. Galaxy tools and tool dependency recipes can only assume the following dependencies are available in the Galaxy environment into which they will be installed (this list is available in the Tool Shed wiki on this page: https://wiki.galaxyproject.org/InstallAndTestCertification). In your case, a recipe should probably also be installing the libcurl-devel package (and possibly others??). Recipes need to be created for dependencies that are not on the following list, and then relationships can be defined between repositories that contain them and repositories that contain tools that depend upon them. The reason the following list is short is because starting up a new Galaxy instance should be as simple as cloning it and starting it up with run.sh. autoconf automake autotools-dev build-essential cmake git-core libatlas-base-dev libblas-dev liblapack-dev libc6-dev mercurial python2.6 python2.6-dev pkg-config subversion python-dev python-pip Greg Von Kuster On Apr 29, 2014, at 4:34 PM, Saket Choudhary sake...@gmail.com wrote: Hi Bjoern, There is no script installed, I generally just do this on my local system $ python setup.py build --with-ssl $ python setup.py install The documentation mentions the following options: PycURL Unix options: --curl-config=/path/to/curl-config use specified curl-config binary --openssl-dir=/path/to/openssl/dir path to OpenSSL headers and libraries --with-ssl libcurl is linked against OpenSSL --with-gnutls libcurl is linked against GnuTLS --with-nss libcurl is linked against NSS Saket On 29 April 2014 23:31, Björn Grüning bjoern.gruen...@gmail.com wrote: Hi Saket, is there any script installed with that package? --install-scripts may be needed. Also I'm wondering if there is no dependency on curl-lib? Cheers, Bjoern Am 29.04.2014 19:16, schrieb Saket Choudhary: Any comments on this? http://testtoolshed.g2.bx.psu.edu/view/saketkc/package_pycurl_7_19_3_1 On 17 April 2014 19:54, Saket Choudhary sake...@gmail.com wrote: My attempt to 'package' pycurl on testtoolshed fails with the following error: src/pycurl.c: In function ‘do_multi_info_read’: src/pycurl.c:3549:15: warning: call to ‘_curl_easy_getinfo_err_string’ declared with attribute warning: curl_easy_getinfo expects a pointer to char * for this info [enabled by default] src/pycurl.c: In function ‘do_curl_getinfo’: src/pycurl.c:2888:19: warning: call to ‘_curl_easy_getinfo_err_curl_slist’ declared with attribute warning: curl_easy_getinfo expects a pointer to struct curl_slist * for this info [enabled by default] error: could not create '/usr/local/share/doc': Permission denied A local install worked fine. Is this because of libcurl-devel missing on toolshed instance? ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] problem with bioc_qvalue, Version: 1.34.0
Hi Tony, It may be easiest to just uninstall / reinstall the repository from the Manage installed tool shed repositories option in the admin menu. Greg Von Kuster On Apr 22, 2014, at 4:21 PM, Tony Kusalik kusa...@cs.usask.ca wrote: Thanks for the response. I have made some progress, but I don't seem to be out-of-the-woods yet. I realized that when I had invoked sh ./scripts/migrate_tools/0010_tools.sh I had done it as root, rather than as user galaxy. Hence I did a chmod -R galaxy:galaxy on both the shed_tools/ and tool_dependencies/ directories. Then in the Galaxy Admin interface I selected the Installed tool shed repositories option. I clicked on update tool shed status in the upper right corner. I waited for that update to complete. About 15 lines down the window is the line for compute_q_values. It's installation status was given as Installed, missing tool dependencies. At the very left, it was marked with a green check mark (This is the latest installable revision of this repository) and an attention symbol (Updates are available in the Tool Shed for this revision). I choose get updates from the pull-down menu associated with compute_q_values. The update window appeared, and I toggled on Install. The installation SEEMED to be okay. Now I went back to the Installed tool shed repositories window. compute_q_values was marked with just a green check mark. The revision column/field is given as 62f7b9c20c9b. The only problem is that the Installation Status is still marked with Installed, missing tool dependencies. Looking at the last 100 or so lines of paster.log, it appears that the install of compute_q_values was indeed successful. Nothing about missing dependencies reported there. Selecting update tool shed status does not change anything. Suggestions? Should I try running sh ./scripts/migrate_tools/0010_tools.sh again? Or might that just make matters worse? On 2014-04-22, at 7:56 , Dave Bouvier d...@bx.psu.edu wrote: David, Tony, I have not yet been able to reproduce the behavior you've reported here. I will continue to investigate, but if you go to Manage installed tool shed repositories in your Galaxy admin interface and select the Update tool shed status option on the compute_q_values repository, your Galaxy instance should detect that there is an update available and give you the option to update the repository and install its dependencies. Your report did help me uncover a potentially related issue with determining dependencies of repositories that have been updated after being added to the migrate_tools definition, which has now been resolved. --Dave B. On 04/17/2014 04:35 PM, David Hoover wrote: I saw the same phenomenon. I think it stems from the current version of compute_q_values pointing to unavailable revisions of both R and the bioc_qvalue R package. Whoever maintains the compute_q_values needs to straighten out the revision ids. David Hoover Helix Systems Staff National Institutes of Health On Apr 17, 2014, at 1:38 PM, Tony Kusalik kusa...@cs.usask.ca wrote: Hi, I am having a problem with updating our local Galaxy server, and I am hoping someone can help me. Yesterday I performed a 'hg incoming' command. A bunch of updates were applied, from changeset 12443:ec9d31a8bc04 to changeset 13068:c05752549163. I restarted the server (in daemon mode), but it halted. The content of paster.log instructed me to run sh ./scripts/migrate_tools/0010_tools.sh I did that. That script encountered an error: The following error occurred from the InstallManager while installing tool dependency bioc_qvalue : Error installing tool dependency package bioc_qvalue version 1.34.0: Unable to locate required tool shed repository named package_bioc_qvalue_1_34_0 owned by devteam with revision 11735242a19e. How do I get around this problem? I checked the mailing lists for anyone reporting a problem with bioc_qvalue or 11735242a19e but nothing was turned up. Tony Kusalik ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client
Re: [galaxy-dev] Using Galaxy ToolShed API for uploading a tar-ball
Hi Peter, This is not yet available, but we'll certainly implement it. I've created this Trello card for it. https://trello.com/c/fykf8IPO/197-enhance-api-to-enable-tar-archive-uploads Thanks! Greg Von Kuster On Apr 10, 2014, at 7:50 AM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi all, I've just skimmed Greg's latest blog post which is about the REST API for the ToolShed: http://gregvonkuster.org/galaxy-tool-shed-restful-api/ I would be interested in automating pushing updates to the ToolShed, which I currently do manually by uploading a tar-ball. Is it possible yet to upload a tar-ball via the API? Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Package/Module RPY
Hello Curt, I believe that version 1.0.3 of the rpy package should be installing successfully from the main tool shed if you have a repository that contains a tool that defines a dependency to the rpy repository owned by devteam at http://toolshed.g2.bx.psu.edu/view/devteam/package_rpy_1_0_3 Here is the last test run for the tool shed's install and test framework for that repository. I believe tis should install correctly using the dist repository, but if you're seeing problems, let us know. Also, the next release is coming soon ( tentatively a week or so from now ).. Test runs 2014-04-07 17:25:49 Automated test environment Successful installations Repository dependencies - installation of these additional repositories is required Tool shed NameOwner Changeset revision toolshed.g2.bx.psu.edu package_r_2_11_0devteam 5824d2b3bc8b Tool dependencies TypeNameVersion R package 2.11.0 Installation directory /ToolDepsMain/R/2.11.0/devteam/package_rpy_1_0_3/82170c94ca7c TypeNameVersion rpy package 1.0.3 Installation directory /ToolDepsMain/rpy/1.0.3/devteam/package_rpy_1_0_3/82170c94ca7c On Apr 8, 2014, at 3:28 PM, Hayes, Curt curt.ha...@seattlechildrens.org wrote: Hi there all, Does anyone have RPY working? I install via toolshed and the folder is empty, and shows missing dependencies. I have searched previous posts, nothing since 2010. I have a local install (production config – postgres, etc – on CENTOS 6.5). I posted last week and it fell off the list. From Galaxy User interface: Dataset generation errors Dataset 81: Build base quality distribution on data 9 Tool execution generated the following error message: Traceback (most recent call last): File /apps/galaxy/galaxy-dist/tools/metag_tools/short_reads_figure_score.py, line 12, in module from rpy import * ImportError: No module named rpy I have attempted to download/install the scripts but they fail with multiple errors (most posts out there have a bad URL) - http://getgalaxyp.org/install.html says use: curlhttp://rpy.svn.sourceforge.net/viewvc/rpy/trunk/rpy/?view=tar rpy.tar.gz if you view that it redirects you to another URL. Via shell: R is installed: R version 2.13.0 (2011-04-13) Can someone provide me some help please to get it working via Galaxy? I understand it works in Central. From Toolshed: Name Version Type Installation status rpy 1.0.3 package Error Toolshed Error: In file included from src/rpymodule2110.c:51: src/RPy.h:56:20: error: Python.h: No such file or directory In file included from /usr/lib64/python2.6/site-packages/numpy/core/include/numpy/ndarrayobject.h:68, from /usr/lib64/python2.6/site-packages/numpy/core/include/numpy/arrayobject.h:14, from src/RPy.h:89, from src/rpymodule2110.c:51: /usr/lib64/python2.6/site-packages/numpy/core/include/numpy/npy_common.h:79:2: error: #error Must use Python with unicode enabled. truncated Thanks, Curt CONFIDENTIALITY NOTICE: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information protected by law. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Reg: Issue while adding simple repository dependencies for custom tool
Hello Björn, I've enhanced the error message displayed in 12959:64d677b1e16a, which is available in the next-stable branch, so it will be included in the upcoming release. This message is tricky, because it is only displayed when visiting the Tool Shed from Galaxy to install a repository that has an invalid repository dependency definition. This scenario does not provide the context for the precise cause of the invalid definition. Visiting the Tool Shed directlry (not from Galaxy) and dispaying the repositoryu should provide a more detailed message regarding the problem. Thanks! Greg Von Kuster On Apr 3, 2014, at 10:24 AM, Björn Grüning bjoern.gruen...@gmail.com wrote: Hi, thanks for fixing it Greg. Would it also be possible to get a more concrete error message in such cases where the repository_dependencies.xml file is somehow broken? Thanks, Bjoern Am 03.04.2014 16:22, schrieb Greg Von Kuster: I should have handled the import in the same commit, but did so in the following changeset. It will eliminate the problematic prior_installation_required=False attribute from the repository tag when a capsule is being imported. changeset: 12944:59bd7349a128 tag: tip user:greg date:Thu Apr 03 10:19:46 2014 -0400 summary: Upon import, handle problematic prior_installation_required attribute incorrectly added (and set to False) to the repository tag when a repository was exported. Greg Von Kuster On Apr 3, 2014, at 9:53 AM, Janaki Rama Rao Gollapudi janakiram.gollap...@india.semanticbits.com wrote: Thanks Greg, I will verify with new changes. Thanks, JanakiRam On Thu, Apr 3, 2014 at 7:05 PM, Greg Von Kuster g...@bx.psu.edu wrote: Hello Janaki and Björn, This problematic behavior is a result of a bug in the repository export process which has been fixed in the following changeset which is currently available in the Galaxy central repo. Thanks very much for reporting this. https://bitbucket.org/galaxy/galaxy-central/commits/773578e65580e3488fadc144739d82f86d9b30f3 Greg Von Kuster On Apr 3, 2014, at 7:53 AM, Greg Von Kuster g...@bx.psu.edu wrote: Yes, if that is the behavior, that is definitely a bug. I'll take a look and get back to you on this. Thanks! On Apr 3, 2014, at 7:35 AM, Björn Grüning bjoern.gruen...@gmail.com wrote: Hi JanakiRam, I will take Greg into CC. He is the main Tool Shed developer, maybe we spotted a bug in populating prior_installation_required in repository_dependencies.xml file. He probably knows if that is supposed to work. Greg, to summarize our findings. It seems that if JanakiRam uploads a repository_dependencies.xml file into a local toolshed the prior_installation_required='False' will be inserted. If he tries to install it, Galaxy complains about a wrong syntax inside the repository_dependecies.xml file. Imho prior_installation_required does not make sense in repository_dependencies.xml or? Is that an bug? JanakiRam can you confirm my summary? Thanks, Bjoern Am 03.04.2014 12:49, schrieb Janaki Rama Rao Gollapudi: Hi, I am using latest checkout from https://bitbucket.org/galaxy/galaxy-dist/(Just now I also updated my local galaxy code base). Yes, without repository dependency everything working fine. Thanks, JanakiRam On Thu, Apr 3, 2014 at 3:39 PM, Björn Grüning bjoern.gruen...@gmail.comwrote: Hi, I just tested it in the test-toolshed and for me 'prior_installation_required' is not inserted. Which Galaxy version do you use? Please note, that I'm not sure if that is really causing the trouble you have seen. If you remove the repository dependency everything is working as expected? Am 03.04.2014 12:01, schrieb Janaki Rama Rao Gollapudi: Hi Bjoern, Please see my comments inline. On Thu, Apr 3, 2014 at 3:14 PM, Björn Grüning bjoern.gruen...@gmail.com wrote: Hi Janaki, can you try to remove this prior_installation_required completely? I didn't added 'prior_installation_required' in the repository_dependencies.xml(as this is optional), it got added when I exported repository files from tool shed. Actual repository_dependencies.xml file has below content. ?xml version=1.0? repositories description=test description repository name=string_occurrence owner=janakiram-t1 / /repositories Also why do you need a repository dependency, I think that is not needed or? This is the first time I working with repository dependencies, so for testing I am trying to add simple repository dependency to my custom tool. Cheers, Bjoern Am 03.04.2014 09:30, schrieb Janaki Rama Rao Gollapudi: Hi Bjoern, I also exported both the repositories from local toolshed. Please find the attached files. Thanks, JanakiRam On Thu, Apr 3, 2014 at 12:50 PM, Janaki Rama Rao Gollapudi janakiram.gollap...@india.semanticbits.com
[galaxy-dev] The main Galaxy Tool Shed is now running the next-stable branch
For those interested, The main Galaxy Tool Shed is now running the next-stable branch in preparation for the upcoming Galaxy release currently scheduled for about 2 weeks from now. We'll continue to update the Tool Shed as new changeset are pushed to the next-stable branch. We'll alert you wnen the main Tool Shed is updated to the stable branch when it is tagged for the next Galaxy release. As usual, the test Galaxy Tool Shed continues to track the default branch. Thanks! Greg Von Kuster ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Reg: Issue while adding simple repository dependencies for custom tool
Yes, if that is the behavior, that is definitely a bug. I'll take a look and get back to you on this. Thanks! On Apr 3, 2014, at 7:35 AM, Björn Grüning bjoern.gruen...@gmail.com wrote: Hi JanakiRam, I will take Greg into CC. He is the main Tool Shed developer, maybe we spotted a bug in populating prior_installation_required in repository_dependencies.xml file. He probably knows if that is supposed to work. Greg, to summarize our findings. It seems that if JanakiRam uploads a repository_dependencies.xml file into a local toolshed the prior_installation_required='False' will be inserted. If he tries to install it, Galaxy complains about a wrong syntax inside the repository_dependecies.xml file. Imho prior_installation_required does not make sense in repository_dependencies.xml or? Is that an bug? JanakiRam can you confirm my summary? Thanks, Bjoern Am 03.04.2014 12:49, schrieb Janaki Rama Rao Gollapudi: Hi, I am using latest checkout from https://bitbucket.org/galaxy/galaxy-dist/(Just now I also updated my local galaxy code base). Yes, without repository dependency everything working fine. Thanks, JanakiRam On Thu, Apr 3, 2014 at 3:39 PM, Björn Grüning bjoern.gruen...@gmail.comwrote: Hi, I just tested it in the test-toolshed and for me 'prior_installation_required' is not inserted. Which Galaxy version do you use? Please note, that I'm not sure if that is really causing the trouble you have seen. If you remove the repository dependency everything is working as expected? Am 03.04.2014 12:01, schrieb Janaki Rama Rao Gollapudi: Hi Bjoern, Please see my comments inline. On Thu, Apr 3, 2014 at 3:14 PM, Björn Grüning bjoern.gruen...@gmail.com wrote: Hi Janaki, can you try to remove this prior_installation_required completely? I didn't added 'prior_installation_required' in the repository_dependencies.xml(as this is optional), it got added when I exported repository files from tool shed. Actual repository_dependencies.xml file has below content. ?xml version=1.0? repositories description=test description repository name=string_occurrence owner=janakiram-t1 / /repositories Also why do you need a repository dependency, I think that is not needed or? This is the first time I working with repository dependencies, so for testing I am trying to add simple repository dependency to my custom tool. Cheers, Bjoern Am 03.04.2014 09:30, schrieb Janaki Rama Rao Gollapudi: Hi Bjoern, I also exported both the repositories from local toolshed. Please find the attached files. Thanks, JanakiRam On Thu, Apr 3, 2014 at 12:50 PM, Janaki Rama Rao Gollapudi janakiram.gollap...@india.semanticbits.com wrote: Hi, Thanks for quick reply. Please find the attached .tar file of both the repositories. Thanks, JanakiRam On Thu, Apr 3, 2014 at 12:45 PM, Björn Grüning bjoern.gruen...@gmail.comwrote: This all seems to be fine, can you send me a tarball with both repositories ... Thanks, Bjoern Am 03.04.2014 09:05, schrieb Janaki Rama Rao Gollapudi: Hi, Did anyone faced this issue ?. I struck here. Please suggest me. Thanks, JanakiRam On Wed, Apr 2, 2014 at 3:10 PM, Janaki Rama Rao Gollapudi janakiram.gollap...@india.semanticbits.com wrote: Hi, I have implemented a custom tool called 'barcode-parse' and uploaded this custom to into my local tool shed(which is running at http://localhost:9009). Also able to install this custom tool from galaxy. Then I have added simple repository dependency. I have created a xml file 'repository_dependencies.xml'. Then I created a new repository in the toolShed called 'Test' and uploaded all necessary files(.py, definition xml file and repository dependency xml file) into this repository. (I have created another repository called 'string_occurrence' and uploaded necessary files into toolShed which is used as dependency repository) ?xml version=1.0? repositories description= repository toolshed=http://localhost:9009; name=string_occurrence owner=janakiram-t1 / /repositories I tried to install custom tool 'barcode-parse' from galaxy, but I got below error message. The repository dependency definitions for this repository are invalid and will be ignored. I followed the steps mentioned in the galaxy wikihttps://wiki. galaxyproject.org/SimpleRepositoryDependencies Am I doing anything wrong? Please suggest me on this. Thanks, JanakiRam ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists
Re: [galaxy-dev] Reg: Issue while adding simple repository dependencies for custom tool
Hello Janaki and Björn, This problematic behavior is a result of a bug in the repository export process which has been fixed in the following changeset which is currently available in the Galaxy central repo. Thanks very much for reporting this. https://bitbucket.org/galaxy/galaxy-central/commits/773578e65580e3488fadc144739d82f86d9b30f3 Greg Von Kuster On Apr 3, 2014, at 7:53 AM, Greg Von Kuster g...@bx.psu.edu wrote: Yes, if that is the behavior, that is definitely a bug. I'll take a look and get back to you on this. Thanks! On Apr 3, 2014, at 7:35 AM, Björn Grüning bjoern.gruen...@gmail.com wrote: Hi JanakiRam, I will take Greg into CC. He is the main Tool Shed developer, maybe we spotted a bug in populating prior_installation_required in repository_dependencies.xml file. He probably knows if that is supposed to work. Greg, to summarize our findings. It seems that if JanakiRam uploads a repository_dependencies.xml file into a local toolshed the prior_installation_required='False' will be inserted. If he tries to install it, Galaxy complains about a wrong syntax inside the repository_dependecies.xml file. Imho prior_installation_required does not make sense in repository_dependencies.xml or? Is that an bug? JanakiRam can you confirm my summary? Thanks, Bjoern Am 03.04.2014 12:49, schrieb Janaki Rama Rao Gollapudi: Hi, I am using latest checkout from https://bitbucket.org/galaxy/galaxy-dist/(Just now I also updated my local galaxy code base). Yes, without repository dependency everything working fine. Thanks, JanakiRam On Thu, Apr 3, 2014 at 3:39 PM, Björn Grüning bjoern.gruen...@gmail.comwrote: Hi, I just tested it in the test-toolshed and for me 'prior_installation_required' is not inserted. Which Galaxy version do you use? Please note, that I'm not sure if that is really causing the trouble you have seen. If you remove the repository dependency everything is working as expected? Am 03.04.2014 12:01, schrieb Janaki Rama Rao Gollapudi: Hi Bjoern, Please see my comments inline. On Thu, Apr 3, 2014 at 3:14 PM, Björn Grüning bjoern.gruen...@gmail.com wrote: Hi Janaki, can you try to remove this prior_installation_required completely? I didn't added 'prior_installation_required' in the repository_dependencies.xml(as this is optional), it got added when I exported repository files from tool shed. Actual repository_dependencies.xml file has below content. ?xml version=1.0? repositories description=test description repository name=string_occurrence owner=janakiram-t1 / /repositories Also why do you need a repository dependency, I think that is not needed or? This is the first time I working with repository dependencies, so for testing I am trying to add simple repository dependency to my custom tool. Cheers, Bjoern Am 03.04.2014 09:30, schrieb Janaki Rama Rao Gollapudi: Hi Bjoern, I also exported both the repositories from local toolshed. Please find the attached files. Thanks, JanakiRam On Thu, Apr 3, 2014 at 12:50 PM, Janaki Rama Rao Gollapudi janakiram.gollap...@india.semanticbits.com wrote: Hi, Thanks for quick reply. Please find the attached .tar file of both the repositories. Thanks, JanakiRam On Thu, Apr 3, 2014 at 12:45 PM, Björn Grüning bjoern.gruen...@gmail.comwrote: This all seems to be fine, can you send me a tarball with both repositories ... Thanks, Bjoern Am 03.04.2014 09:05, schrieb Janaki Rama Rao Gollapudi: Hi, Did anyone faced this issue ?. I struck here. Please suggest me. Thanks, JanakiRam On Wed, Apr 2, 2014 at 3:10 PM, Janaki Rama Rao Gollapudi janakiram.gollap...@india.semanticbits.com wrote: Hi, I have implemented a custom tool called 'barcode-parse' and uploaded this custom to into my local tool shed(which is running at http://localhost:9009). Also able to install this custom tool from galaxy. Then I have added simple repository dependency. I have created a xml file 'repository_dependencies.xml'. Then I created a new repository in the toolShed called 'Test' and uploaded all necessary files(.py, definition xml file and repository dependency xml file) into this repository. (I have created another repository called 'string_occurrence' and uploaded necessary files into toolShed which is used as dependency repository) ?xml version=1.0? repositories description= repository toolshed=http://localhost:9009; name=string_occurrence owner=janakiram-t1 / /repositories I tried to install custom tool 'barcode-parse' from galaxy, but I got below error message. The repository dependency definitions for this repository are invalid and will be ignored. I followed the steps mentioned in the galaxy wikihttps://wiki. galaxyproject.org/SimpleRepositoryDependencies Am I doing anything wrong? Please suggest me
Re: [galaxy-dev] Reg: Issue while adding simple repository dependencies for custom tool
I should have handled the import in the same commit, but did so in the following changeset. It will eliminate the problematic prior_installation_required=False attribute from the repository tag when a capsule is being imported. changeset: 12944:59bd7349a128 tag: tip user:greg date:Thu Apr 03 10:19:46 2014 -0400 summary: Upon import, handle problematic prior_installation_required attribute incorrectly added (and set to False) to the repository tag when a repository was exported. Greg Von Kuster On Apr 3, 2014, at 9:53 AM, Janaki Rama Rao Gollapudi janakiram.gollap...@india.semanticbits.com wrote: Thanks Greg, I will verify with new changes. Thanks, JanakiRam On Thu, Apr 3, 2014 at 7:05 PM, Greg Von Kuster g...@bx.psu.edu wrote: Hello Janaki and Björn, This problematic behavior is a result of a bug in the repository export process which has been fixed in the following changeset which is currently available in the Galaxy central repo. Thanks very much for reporting this. https://bitbucket.org/galaxy/galaxy-central/commits/773578e65580e3488fadc144739d82f86d9b30f3 Greg Von Kuster On Apr 3, 2014, at 7:53 AM, Greg Von Kuster g...@bx.psu.edu wrote: Yes, if that is the behavior, that is definitely a bug. I'll take a look and get back to you on this. Thanks! On Apr 3, 2014, at 7:35 AM, Björn Grüning bjoern.gruen...@gmail.com wrote: Hi JanakiRam, I will take Greg into CC. He is the main Tool Shed developer, maybe we spotted a bug in populating prior_installation_required in repository_dependencies.xml file. He probably knows if that is supposed to work. Greg, to summarize our findings. It seems that if JanakiRam uploads a repository_dependencies.xml file into a local toolshed the prior_installation_required='False' will be inserted. If he tries to install it, Galaxy complains about a wrong syntax inside the repository_dependecies.xml file. Imho prior_installation_required does not make sense in repository_dependencies.xml or? Is that an bug? JanakiRam can you confirm my summary? Thanks, Bjoern Am 03.04.2014 12:49, schrieb Janaki Rama Rao Gollapudi: Hi, I am using latest checkout from https://bitbucket.org/galaxy/galaxy-dist/(Just now I also updated my local galaxy code base). Yes, without repository dependency everything working fine. Thanks, JanakiRam On Thu, Apr 3, 2014 at 3:39 PM, Björn Grüning bjoern.gruen...@gmail.comwrote: Hi, I just tested it in the test-toolshed and for me 'prior_installation_required' is not inserted. Which Galaxy version do you use? Please note, that I'm not sure if that is really causing the trouble you have seen. If you remove the repository dependency everything is working as expected? Am 03.04.2014 12:01, schrieb Janaki Rama Rao Gollapudi: Hi Bjoern, Please see my comments inline. On Thu, Apr 3, 2014 at 3:14 PM, Björn Grüning bjoern.gruen...@gmail.com wrote: Hi Janaki, can you try to remove this prior_installation_required completely? I didn't added 'prior_installation_required' in the repository_dependencies.xml(as this is optional), it got added when I exported repository files from tool shed. Actual repository_dependencies.xml file has below content. ?xml version=1.0? repositories description=test description repository name=string_occurrence owner=janakiram-t1 / /repositories Also why do you need a repository dependency, I think that is not needed or? This is the first time I working with repository dependencies, so for testing I am trying to add simple repository dependency to my custom tool. Cheers, Bjoern Am 03.04.2014 09:30, schrieb Janaki Rama Rao Gollapudi: Hi Bjoern, I also exported both the repositories from local toolshed. Please find the attached files. Thanks, JanakiRam On Thu, Apr 3, 2014 at 12:50 PM, Janaki Rama Rao Gollapudi janakiram.gollap...@india.semanticbits.com wrote: Hi, Thanks for quick reply. Please find the attached .tar file of both the repositories. Thanks, JanakiRam On Thu, Apr 3, 2014 at 12:45 PM, Björn Grüning bjoern.gruen...@gmail.comwrote: This all seems to be fine, can you send me a tarball with both repositories ... Thanks, Bjoern Am 03.04.2014 09:05, schrieb Janaki Rama Rao Gollapudi: Hi, Did anyone faced this issue ?. I struck here. Please suggest me. Thanks, JanakiRam On Wed, Apr 2, 2014 at 3:10 PM, Janaki Rama Rao Gollapudi janakiram.gollap...@india.semanticbits.com wrote: Hi, I have implemented a custom tool called 'barcode-parse' and uploaded this custom to into my local tool shed(which is running at http://localhost:9009). Also able to install this custom tool from galaxy. Then I have added simple repository dependency
Re: [galaxy-dev] How to write test cases for custom tool
As Janaki replied below, see: http://wiki.galaxyproject.org/Admin/RunningTests?action=showredirect=Admin%2FRunning+Tests On Mar 27, 2014, at 7:15 AM, Janaki Rama Rao Gollapudi janakiram.gollap...@india.semanticbits.com wrote: Hi, I have added tests to tool dependency xml, but how should I run these tests, can you please help me to run test cases. . tests test param name=input1 value=some string/ param name=input2 value=expected format/ output name=output file=output.txt/ /test /tests .. Thanks, G.JanakiRam On Thu, Mar 27, 2014 at 3:27 PM, Janaki Rama Rao Gollapudi janakiram.gollap...@india.semanticbits.com wrote: Hi, I found below resources: Running test Writing Functional tests Thanks, JanakiRam On Thu, Mar 27, 2014 at 2:53 PM, Janaki Rama Rao Gollapudi janakiram.gollap...@india.semanticbits.com wrote: Hi, I have implemented a custom tool which will parse a given string in to expected format. Installed this custom tool into galaxy and was able to run successfully. I would like to add test cases to this tool, can you please provide me resources to write test cases for this custom tool. Is this tool need functional, unit, integration test cases ? I am not sure about which test cases needed for this tool. Thanks, JanakiRam ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] How to write test cases for custom tool
Hello Janaki, Thanks for clarifying that you have installed your tools from a Tool Shed. In this case, functional tests do not use Galaxy's tool_conf.xml.sample, so changing it will make no difference. For information about running functional tests on tools installed from the Tool Shed, see: https://wiki.galaxyproject.org/TestingInstalledTools Basically, you'll be doing the following. The functional test framework is not currently set up to test a specific installed tool using a -id flag. You'll have to use just the -installed flagg which will end up testing all installed tools. export GALAXY_TOOL_DEPENDENCY_DIR=tool_dependencies; sh run_functional_tests.sh -installed Greg Von Kuster On Mar 27, 2014, at 8:58 AM, Janaki Rama Rao Gollapudi janakiram.gollap...@india.semanticbits.com wrote: Hi, Thanks for reply. I composed a details email for better clarity. when run grep command in galaxy home folder ( ./run_functional_tests.sh -list | grep barcode-parse), I got no results. What I did was: Implemented a custom tool (It has one python script and tool definition file. These files are located in ../tools/Mytools/customToolName/) Then run the ToolShed in my local which is running on port 9009 (I have) Created a new repository in the ToolShed (which is running on 9009) and uploaded .py and .xml files in to it Now I run the galaxy (running on port 8080) and browse the my custom tool from my local galaxy and install the custom tool successfully And shed_tool_conf.xml file updated with new section: section id=mTools name=MyTools version= tool file=xxx.xx.x.xx/repos/janakiram-t1/barcode_parse_1/b6a60d02b1a2/barcode_parse_1/barcode- parse.xml guid=xxx.xx.x.xx:9009/repos/janakiram-t1/barcode_parse_1/barcode-parse/1.0.0 tool_shedxxx.xxx.x.xx:9009/tool_shed repository_namebarcode_parse_1/repository_name repository_ownerjanakiram-t1/repository_owner installed_changeset_revisionb6a60d02b1a2/installed_changeset_revision idxxx.xxx.x.xx:9009/repos/janakiram-t1/barcode_parse_1/barcode-parse/1.0.0/id version1.0.0/version /tool /section My tool definition file look likes below: tool id=barcode-parse name=Barcode parse descriptionsome description/description command interpreter=pythonbarcode-parse.py $input1 $input2 $output/command inputs param format=input name=input1 type=text size=50 label=barcode help=Enter barcode validator type=empty_field message=please enter barcode/ /param param format=input name=input2 type=text size=50 label=Expected format help=Enter format to parse barcode validator type=empty_field message=please enter expected format barcode/ /param /inputs outputs data format=txt name=output / /outputs tests test param name=input1 value=test-01-sample-test-02-sampl1-test-03-sample3/ param name=input2 value=name-id/ output name=output file=barcode-parse.txt/ /test /tests help Parse the barcode into expected format /help /tool As galaxy using tool_conf.xml.sample, this file has no section for my custom tool. I have added a section manually like below: section name=MyTools id=mTools tool file=myTools/barcode-parse/barcode-parse.xml / /section Then my tool id found when I run the grep command. After that Then I run the below command to run the test cases: sh run_functional_tests.sh -id barcode-parse But I got below out put (from the run_functional_tests.html) AttributeError: 'module' object has no attribute 'test_toolbox:TestForTool_barcode-parse' Am I doing anything wrong. Thanks, JanakiRam On Thu, Mar 27, 2014 at 6:16 PM, Greg Von Kuster g...@bx.psu.edu wrote: As Janaki replied below, see: http://wiki.galaxyproject.org/Admin/RunningTests?action=showredirect=Admin%2FRunning+Tests On Mar 27, 2014, at 7:15 AM, Janaki Rama Rao Gollapudi janakiram.gollap...@india.semanticbits.com wrote: Hi, I have added tests to tool dependency xml, but how should I run these tests, can you please help me to run test cases. . tests test param name=input1 value=some string/ param name=input2 value=expected format/ output name=output file=output.txt/ /test /tests .. Thanks, G.JanakiRam On Thu, Mar 27, 2014 at 3:27 PM, Janaki Rama Rao Gollapudi janakiram.gollap...@india.semanticbits.com wrote: Hi, I found below resources: Running test Writing Functional tests Thanks, JanakiRam On Thu, Mar 27, 2014 at 2:53 PM, Janaki Rama Rao Gollapudi janakiram.gollap...@india.semanticbits.com wrote: Hi, I have implemented a custom tool which will parse a given string in to expected format
Re: [galaxy-dev] Galaxy startup takes very long. Normal?
Hi Geert, The setting should be in your universe_wsgi.ini.sample, and you woud have to manually edit your universe_wsgi.ini to add it. I would say that 125 installed packages is probably what is causing the slow starts. This feature is not currently useful, so it can be set to not function with no problems. In the future I'll introduce additional benefits for this feature, and I'll look at ways to improve the startup speed. Greg Von Kuster On Mar 26, 2014, at 3:43 AM, Geert Vandeweyer geert.vandewey...@uantwerpen.be wrote: hi Greg, I don't have that setting in my universe. Should I just add it? The output of hg summary is (hg incoming doesn't show any available updates): parent: 12276:dc067a95261d tip Added tag release_2014.02.10 for changeset 5e605ed6069f branch: stable commit: 22 modified, 1 deleted, 127 unknown update: (current) I currently have 125 packages from toolsheds. Is that considered to be many? Most of them are the migrated tools and the devteam mappers/gatk/tophat stuff. Geert On 03/25/2014 05:16 PM, Greg Von Kuster wrote: Hello Geert, Do you have a lot of repositories installed from the Tool Shed into you Galaxy instance? if so, the time you're experience may be due to loading the in-memory installed repository registry. Using this registry is optional, but the default configuration setting is to use it. You can set the following to entry to False in your universe_wsgi.ini file and restart your Galaxy server to see if that is the problem. This registry is not currently used by anythin except to display the set of dependent repositories that will be affected if a repository is uninstalled. If this is ot what is causing the slow statup, then I'm not sure where else to look. # Enable use of an in-memory registry with bi-directional relationships # between repositories (i.e., in addition to lists of dependencies for a # repository, keep an in-memory registry of dependent items for each repository. #manage_dependency_relationships = True Greg Von Kuster On Mar 25, 2014, at 11:08 AM, Geert Vandeweyer geert.vandewey...@uantwerpen.be wrote: Dear all, I'm wondering if the following behaviour is normal. Since I reinstalled the latest galaxy distribution, every restart hangs/loads for up to 10 minutes at the following lines: galaxy.model.migrate.check INFO 2014-03-25 15:56:57,965 At database version 118 tool_shed.galaxy_install.migrate.check INFO 2014-03-25 15:56:58,477 At migrate_tools version 9 galaxy.config INFO 2014-03-25 15:56:58,694 Install database targetting Galaxy's database configuration. After several minutes of no heavy processing (cpu/database is almost idle), the startup continues. Best, Geert -- Geert Vandeweyer, Ph.D. Department of Medical Genetics University of Antwerp Prins Boudewijnlaan 43 2650 Edegem Belgium Tel: +32 (0)3 275 97 56 E-mail: geert.vandewe...@ua.ac.be http://ua.ac.be/cognitivegenetics http://www.linkedin.com/in/geertvandeweyer ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ -- Geert Vandeweyer, Ph.D. Department of Medical Genetics University of Antwerp Prins Boudewijnlaan 43 2650 Edegem Belgium Tel: +32 (0)3 275 97 56 E-mail: geert.vandewe...@ua.ac.be http://ua.ac.be/cognitivegenetics http://www.linkedin.com/in/geertvandeweyer ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Galaxy startup takes very long. Normal?
The changeset that made this a configurable setting was committed to the stable branch in d270d3c, which was committed on 2014-02-19 https://bitbucket.org/galaxy/galaxy-central/commits/d270d3cf7627a42b6fac2aa69701deadb89c8ffc If you are tracking the stable branch in the galaxy central repo (which it looks like you are doing), then you should be able to pull it from there. Greg Von Kuster On Mar 26, 2014, at 8:04 AM, Geert Vandeweyer geert.vandewey...@uantwerpen.be wrote: Hi Greg, The setting was not in my universe_sample file. I added it, set log_info to DEBUG and restarted galaxy. The startup time remained the same, and there were many entries in the log file indicating that the lag is indeed the creation of the dependency system: tool_shed.galaxy_install.installed_repository_manager DEBUG 2014-03-26 13:01:01,176 Adding an entry for version 0.1.18 of package samtools to runtime_tool_dependencies_of_installed_tool_dependencies. tool_shed.galaxy_install.installed_repository_manager DEBUG 2014-03-26 13:01:02,645 Adding an entry for version 0.1.18 of package samtools to installed_runtime_dependent_tool_dependencies_of_installed_tool_dependencies. tool_shed.galaxy_install.installed_repository_manager DEBUG 2014-03-26 13:01:03,874 Adding an entry for version 0.1.18 of package samtools to runtime_tool_dependencies_of_installed_tool_dependencies. tool_shed.galaxy_install.installed_repository_manager DEBUG 2014-03-26 13:01:05,383 Adding an entry for version 0.1.18 of package samtools to installed_runtime_dependent_tool_dependencies_of_installed_tool_dependencies. tool_shed.galaxy_install.installed_repository_manager DEBUG 2014-03-26 13:01:06,533 Adding an entry for version 2.11.0 of package R to runtime_tool_dependencies_of_installed_tool_dependencies. tool_shed.galaxy_install.installed_repository_manager DEBUG 2014-03-26 13:01:08,054 Adding an entry for version 2.11.0 of package R to installed_runtime_dependent_tool_dependencies_of_installed_tool_dependencies. tool_shed.galaxy_install.installed_repository_manager DEBUG 2014-03-26 13:01:09,126 Adding an entry for version 2.11.0 of package R to runtime_tool_dependencies_of_installed_tool_dependencies. tool_shed.galaxy_install.installed_repository_manager DEBUG 2014-03-26 13:01:10,596 Adding an entry for version 2.11.0 of package R to installed_runtime_dependent_tool_dependencies_of_installed_tool_dependencies. tool_shed.galaxy_install.installed_repository_manager DEBUG 2014-03-26 13:01:11,803 Adding an entry for version 1.02.00 of package lastz to runtime_tool_dependencies_of_installed_tool_dependencies. tool_shed.galaxy_install.installed_repository_manager DEBUG 2014-03-26 13:01:13,332 Adding an entry for version 1.02.00 of package lastz to installed_runtime_dependent_tool_dependencies_of_installed_tool_dependencies. tool_shed.galaxy_install.installed_repository_manager DEBUG 2014-03-26 13:01:14,552 Adding an entry for version 3.0.1 of package R_3_0_1 to runtime_tool_dependencies_of_installed_tool_dependencies. tool_shed.galaxy_install.installed_repository_manager DEBUG 2014-03-26 13:01:15,943 Adding an entry for version 3.0.1 of package R_3_0_1 to installed_runtime_dependent_tool_dependencies_of_installed_tool_dependencies. tool_shed.galaxy_install.installed_repository_manager DEBUG 2014-03-26 13:01:17,162 Adding an entry for version 0.1.18 of package samtools to runtime_tool_dependencies_of_installed_tool_dependencies. tool_shed.galaxy_install.installed_repository_manager DEBUG 2014-03-26 13:01:18,594 Adding an entry for version 0.1.18 of package samtools to installed_runtime_dependent_tool_dependencies_of_installed_tool_dependencies. etc... I also note many double entries in the log files. Do I need a specific changeset for the parameter setting to take effect? Universe_wsgi.ini setting: # Enable use of an in-memory registry with bi-directional relationships between repositories manage_dependency_relationships = False Best, Geert On 03/26/2014 11:53 AM, Greg Von Kuster wrote: Hi Geert, The setting should be in your universe_wsgi.ini.sample, and you woud have to manually edit your universe_wsgi.ini to add it. I would say that 125 installed packages is probably what is causing the slow starts. This feature is not currently useful, so it can be set to not function with no problems. In the future I'll introduce additional benefits for this feature, and I'll look at ways to improve the startup speed. Greg Von Kuster On Mar 26, 2014, at 3:43 AM, Geert Vandeweyer geert.vandewey...@uantwerpen.be wrote: hi Greg, I don't have that setting in my universe. Should I just add it? The output of hg summary is (hg incoming doesn't show any available updates): parent: 12276:dc067a95261d tip Added tag release_2014.02.10 for changeset 5e605ed6069f branch: stable commit: 22 modified, 1 deleted, 127 unknown update
Re: [galaxy-dev] prims workflows
Hello Pieter, When you install a repository into Galaxy that contains workflows, you can import the workflow into Galaxy. At the time the workflow is imported, the import process checks to see if all required tools are available in Galaxy. If not, a page is displayed with links to accessible Tool Sheds allowing the user to search for missing required tools. I believe this still works, although there have been some changes to the workflow framework over the past several months, and I have not checked to see if this is still available. See https://wiki.galaxyproject.org/ToolShedWorkflowSharing for more details. This approach is basically a hack that I implemented several years ago. A much better approach would be to enhance the workflow UI to include a feature allowing the user to search Tool Sheds to discover and install missing tools required by any workflow currently abvailable in the Galaxy instance. Hopefully at some point soon something like this will be introduced. Greg Von Kuster On Mar 25, 2014, at 6:47 AM, Lukasse, Pieter pieter.luka...@wur.nl wrote: Hi Bjoern, I think you may have misunderstood me. What I mean is : instead of me having to add a tool_dependencies or repository_dependencies.xml to the workflows package, let Galaxy Tool Shed find these dependencies based on metadata available in the workflow itself. Best regards, Pieter. -Original Message- From: Björn Grüning [mailto:bjoern.gruen...@gmail.com] Sent: dinsdag 25 maart 2014 11:43 To: Lukasse, Pieter; Greg Von Kuster (g...@bx.psu.edu) Subject: Re: FW: prims workflows Hi Pieter, if I'm not misunderstood you, thats already possible: Have a look at: https://github.com/bgruening/galaxytools/tree/master/workflows/glimmer3 For an example. Cheers, Bjoern Am 25.03.2014 11:08, schrieb Lukasse, Pieter: Hi Greg, Do you have any plans of resolving the dependencies automatically when it comes to workflows? E.g. I have a Tool Shed repository where I would like to add workflows that are constructed using tools from different repositories. It would be nice if you could let Tool Shed install the tools automatically upon installation of the workflow. Any plans? Best regards, Pieter -Original Message- From: Björn Grüning [mailto:bjoern.gruen...@gmail.com] Sent: zaterdag 22 maart 2014 16:25 To: Lukasse, Pieter Subject: prims workflows Hi Pieter, thanks for your prims Galaxy contributions. Any chance to add a repository_dependencies.xml file to the workflow repository? Thanks, Bjoern ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] tophat2 install
If you installe dit from the main Tool Shed, then this repository has the recipe for installing the package. http://toolshed.g2.bx.psu.edu/view/devteam/package_tophat2_2_0_9 Greg Von Kuster On Mar 25, 2014, at 9:25 AM, Briand, Sheldon sheldon.bri...@ssc-spc.gc.ca wrote: Hi, Not sure if you saw my last email. Updating to the latest patch version doesn't seem to help. I was wondering where I could get the galaxy package version of tophat2 and put it where it needs to be manually. Then I could try the uninstall/install and see if that helps. Thanks! -Sheldon -Original Message- From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Briand, Sheldon Sent: Thursday, March 20, 2014 3:11 PM To: 'Greg Von Kuster' Cc: galaxy-dev@lists.bx.psu.edu Dev Subject: Re: [galaxy-dev] tophat2 install No difference. Same error -Original Message- From: Greg Von Kuster [mailto:g...@bx.psu.edu] Sent: Thursday, March 20, 2014 2:56 PM To: Briand, Sheldon Cc: galaxy-dev@lists.bx.psu.edu Dev Subject: Re: [galaxy-dev] tophat2 install Sorry, should have advised this: hg pull https://bitbucket.org/galaxy/galaxy-central#stable hg update stable See the end ot this thread: http://dev.list.galaxyproject.org/Persistent-jobs-in-cluster-queue-even-after-canceling-job-in-galaxy-td4663719.html On Mar 20, 2014, at 1:51 PM, Björn Grüning bjoern.gruen...@gmail.com wrote: Hi, you need to track galaxy-central to make use of the patches that are applied as bugfixes after the release. You can do that easily with changing the path in .hg/hgrc ... but I do not know if that is the right way to do ;) For me it worked. Cheers, Bjoern Am 20.03.2014 18:46, schrieb Briand, Sheldon: pulling from https://bitbucket.org/galaxy/galaxy-dist searching for changes no changes found From: Greg Von Kuster [mailto:g...@bx.psu.edu] Sent: Thursday, March 20, 2014 2:36 PM To: Briand, Sheldon Cc: 'galaxy-dev@lists.bx.psu.edu' Subject: Re: [galaxy-dev] tophat2 install There have been several fixes released to the stable branch that you do not have in 12441. I would recommend doing the following: hg pull hg update stable Then uninstall the tophat2 repository and reinstall it. See if that makes a difference. On Mar 20, 2014, at 1:30 PM, Briand, Sheldon sheldon.bri...@ssc-spc.gc.camailto:sheldon.bri...@ssc-spc.gc.ca wrote: $ hg summary parent: 12441:dc067a95261d From the tool_dependencies.xml: package name=tophat2 version=2.0.9 repository changeset_revision=8549fd545473 name=package_tophat2_2_0_9 owner=devteam prior_installation_required=False toolshed =http://toolshed.g2.bx.psu.edu; / From: Greg Von Kuster [mailto:g...@bx.psu.eduhttp://bx.psu.edu] Sent: Thursday, March 20, 2014 2:23 PM To: Briand, Sheldon Cc: 'galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu' Subject: Re: [galaxy-dev] tophat2 install What version of Galaxy are you running? You want to look in the tophat2 repository directory, not its package dependency. There should be a tool_dependencies.xml file in the tophat2's installation directory hierarchy somewhere. On Mar 20, 2014, at 1:12 PM, Briand, Sheldon sheldon.bri...@ssc-spc.gc.camailto:sheldon.bri...@ssc-spc.gc.ca wrote: Hi, dependancies/tophat2/2.0.9/devteam/package_tophat2_2_0_9/8549fd545473 contains only: env.sh If that isn't the right place to look let me know. -Sheldon From: Greg Von Kuster [mailto:g...@bx.psu.eduhttp://bx.psu.edu] Sent: Thursday, March 20, 2014 2:04 PM To: Briand, Sheldon Cc: 'galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu' Subject: Re: [galaxy-dev] tophat2 install It looks like your database and file system are out of sync. Your database seems to think you have an installed repository but your file system does not seem to have the repository's installation directory. Can you confirm that this is the case? On Mar 20, 2014, at 11:53 AM, Briand, Sheldon sheldon.bri...@ssc-spc.gc.camailto:sheldon.bri...@ssc-spc.gc.ca wrote: Hi, Here is the error from the paster.log: Traceback (most recent call last): File /BigData/galaxy/galaxy-dist/lib/tool_shed/util/common_install_util.py, line 496, in handle_tool_dependencies from_install_manager=from_install_manager ) File /BigData/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/tool_dependencies/install_util.py, line 318, in install_package from_install_manager=from_install_manager ) File /BigData/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/tool_dependencies/install_util.py, line 245, in handle_complex_repository_d ependency_for_package tool_dependencies_config = get_absolute_path_to_file_in_repository( repo_files_dir, 'tool_dependencies.xml' ) File /BigData/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/tool_dependencies/install_util.py
Re: [galaxy-dev] Galaxy startup takes very long. Normal?
Hello Geert, Do you have a lot of repositories installed from the Tool Shed into you Galaxy instance? if so, the time you're experience may be due to loading the in-memory installed repository registry. Using this registry is optional, but the default configuration setting is to use it. You can set the following to entry to False in your universe_wsgi.ini file and restart your Galaxy server to see if that is the problem. This registry is not currently used by anythin except to display the set of dependent repositories that will be affected if a repository is uninstalled. If this is ot what is causing the slow statup, then I'm not sure where else to look. # Enable use of an in-memory registry with bi-directional relationships # between repositories (i.e., in addition to lists of dependencies for a # repository, keep an in-memory registry of dependent items for each repository. #manage_dependency_relationships = True Greg Von Kuster On Mar 25, 2014, at 11:08 AM, Geert Vandeweyer geert.vandewey...@uantwerpen.be wrote: Dear all, I'm wondering if the following behaviour is normal. Since I reinstalled the latest galaxy distribution, every restart hangs/loads for up to 10 minutes at the following lines: galaxy.model.migrate.check INFO 2014-03-25 15:56:57,965 At database version 118 tool_shed.galaxy_install.migrate.check INFO 2014-03-25 15:56:58,477 At migrate_tools version 9 galaxy.config INFO 2014-03-25 15:56:58,694 Install database targetting Galaxy's database configuration. After several minutes of no heavy processing (cpu/database is almost idle), the startup continues. Best, Geert -- Geert Vandeweyer, Ph.D. Department of Medical Genetics University of Antwerp Prins Boudewijnlaan 43 2650 Edegem Belgium Tel: +32 (0)3 275 97 56 E-mail: geert.vandewe...@ua.ac.be http://ua.ac.be/cognitivegenetics http://www.linkedin.com/in/geertvandeweyer ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] tool_dependencies.xml Environmental Variable Question
Hello Michael, Environment variables properly defined in a tool's associated tool_dependencies.xml file contained in an installed Tool Shed repository should be available to the tool's environment at runtime. Do you have an example where you are not seeing this? Greg Von Kuster On Mar 21, 2014, at 1:22 PM, Michael E. Cotterell mepcotter...@gmail.com wrote: Is the expected behavior that environmental variables setup in a tool’s tool_dependencies.xml file are not available in the tool’s python virtualenv during runtime? It’s seems odd that we can specify modules (installable from pip) that are available during runtime, but that the environmental variables are not available during runtime. Thanks! Sincerely, Michael E. Cotterell Ph.D. Student in Computer Science, University of Georgia Instructor of Record, Graduate RA TA, University of Georgia Department Liaison, CS Graduate Student Association, University of Georgia mepcotter...@gmail.com (mailto:mepcotter...@gmail.com) mepc...@uga.edu (mailto:mepc...@uga.edu) m...@cs.uga.edu (mailto:m...@cs.uga.edu) http://michaelcotterell.com/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] tophat2 install
Hi Sheldon, I recommend not manually changing any database entries. You should be able to install the repository and all of its dependencies using the Repair Repository feature. See https://wiki.galaxyproject.org/RepairingInstalledRepositories Greg Von Kuster On Mar 20, 2014, at 8:59 AM, Briand, Sheldon sheldon.bri...@ssc-spc.gc.ca wrote: Hi, Just wondering if I can still use the following to fix up my messed up tophat2 install (the install had hung up on me, for 2 hours, and I tried to remove/repair it via the interface). I haven’t had problems installing any other tools or packages. I am using postgres but this advice was given 2 years ago to another user and fields may have changed. Thanks for any help! -Sheldon 1. Manually remove the installed repository subdirectory hierarchy from disk. 2. If the repository included any tools, manually remove entries for each of them from the shed_tool-conf.xml file ( or the equivalent file you have configured for handling installed repositories ) 3. Manually update the database using the following command (assuming your installed repository is named 'bcftools_view' and it is the only repository you have installed with that name) - letter capitalization is required: The following assumes you're using postgres: update tool_shed_repository set deleted=True, uninstalled=True, status='Uninstalled', error_message=Null where name = 'bcftools_view'; From: Briand, Sheldon Sent: Wednesday, March 12, 2014 11:44 AM To: galaxy-dev@lists.bx.psu.edu Subject: tophat2 install Hi, I ran into a problem installing tophat2 here: tophat2 version 2.0.9: coercing to Unicode: need string or buffer, NoneType found I have only used the interface to do the installation (no manual deletes). The rest of the dependencies for tophat2 installed just fine. Repairing the repository doesn’t make any difference. Thanks! -Sheldon Sheldon Briand NRC Research Computing Support Analyst Research Computing Support / Soutien Informartique a la Recherche Operations, Science Portfolio / Operations, Portefeuil des sciences SSC-NRC / SPC-CNRC Rm 329A, 1411 Oxford Street / Piece 329A, 1411 Rue Oxford Halifax, NS B3H 3Z1 902 426-1677 sheldon.bri...@ssc-spc.gc.ca ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] tophat2 install
You should be able to click on the missing tool dependency and see its installation log. It should show you whatever error is occuring that is not allowing successful installation. Greg Von Kuster On Mar 20, 2014, at 9:46 AM, Briand, Sheldon sheldon.bri...@ssc-spc.gc.ca wrote: Hi, The repair operation doesn’t seem to make any difference (I already tried that before emailing). Screenshot of the result attached. -Sheldon From: Greg Von Kuster [mailto:g...@bx.psu.edu] Sent: Thursday, March 20, 2014 10:26 AM To: Briand, Sheldon Cc: 'galaxy-dev@lists.bx.psu.edu' Subject: Re: [galaxy-dev] tophat2 install Hi Sheldon, I recommend not manually changing any database entries. You should be able to install the repository and all of its dependencies using the Repair Repository feature. See https://wiki.galaxyproject.org/RepairingInstalledRepositories Greg Von Kuster On Mar 20, 2014, at 8:59 AM, Briand, Sheldon sheldon.bri...@ssc-spc.gc.ca wrote: Hi, Just wondering if I can still use the following to fix up my messed up tophat2 install (the install had hung up on me, for 2 hours, and I tried to remove/repair it via the interface). I haven’t had problems installing any other tools or packages. I am using postgres but this advice was given 2 years ago to another user and fields may have changed. Thanks for any help! -Sheldon 1. Manually remove the installed repository subdirectory hierarchy from disk. 2. If the repository included any tools, manually remove entries for each of them from the shed_tool-conf.xml file ( or the equivalent file you have configured for handling installed repositories ) 3. Manually update the database using the following command (assuming your installed repository is named 'bcftools_view' and it is the only repository you have installed with that name) - letter capitalization is required: The following assumes you're using postgres: update tool_shed_repository set deleted=True, uninstalled=True, status='Uninstalled', error_message=Null where name = 'bcftools_view'; From: Briand, Sheldon Sent: Wednesday, March 12, 2014 11:44 AM To: galaxy-dev@lists.bx.psu.edu Subject: tophat2 install Hi, I ran into a problem installing tophat2 here: tophat2 version 2.0.9: coercing to Unicode: need string or buffer, NoneType found I have only used the interface to do the installation (no manual deletes). The rest of the dependencies for tophat2 installed just fine. Repairing the repository doesn’t make any difference. Thanks! -Sheldon Sheldon Briand NRC Research Computing Support Analyst Research Computing Support / Soutien Informartique a la Recherche Operations, Science Portfolio / Operations, Portefeuil des sciences SSC-NRC / SPC-CNRC Rm 329A, 1411 Oxford Street / Piece 329A, 1411 Rue Oxford Halifax, NS B3H 3Z1 902 426-1677 sheldon.bri...@ssc-spc.gc.ca ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ Screenshot.png ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] tophat2 install
It looks like your database and file system are out of sync. Your database seems to think you have an installed repository but your file system does not seem to have the repository's installation directory. Can you confirm that this is the case? On Mar 20, 2014, at 11:53 AM, Briand, Sheldon sheldon.bri...@ssc-spc.gc.ca wrote: Hi, Here is the error from the paster.log: Traceback (most recent call last): File /BigData/galaxy/galaxy-dist/lib/tool_shed/util/common_install_util.py, line 496, in handle_tool_dependencies from_install_manager=from_install_manager ) File /BigData/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/tool_dependencies/install_util.py, line 318, in install_package from_install_manager=from_install_manager ) File /BigData/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/tool_dependencies/install_util.py, line 245, in handle_complex_repository_d ependency_for_package tool_dependencies_config = get_absolute_path_to_file_in_repository( repo_files_dir, 'tool_dependencies.xml' ) File /BigData/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/tool_dependencies/install_util.py, line 126, in get_absolute_path_to_file_i n_repository for root, dirs, files in os.walk( repo_files_dir ): File /opt/local/python-2.7.5/lib/python2.7/os.py, line 276, in walk names = listdir(top) TypeError: coercing to Unicode: need string or buffer, NoneType found -Sheldon From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Briand, Sheldon Sent: Thursday, March 20, 2014 10:52 AM To: 'Greg Von Kuster' Cc: 'galaxy-dev@lists.bx.psu.edu' Subject: Re: [galaxy-dev] tophat2 install Hi, Yes the error is: Error installing tool dependency package tophat2 version 2.0.9: coercing to Unicode: need string or buffer, NoneType found -Sheldon From: Greg Von Kuster [mailto:g...@bx.psu.edu] Sent: Thursday, March 20, 2014 10:49 AM To: Briand, Sheldon Cc: 'galaxy-dev@lists.bx.psu.edu' Subject: Re: [galaxy-dev] tophat2 install You should be able to click on the missing tool dependency and see its installation log. It should show you whatever error is occuring that is not allowing successful installation. Greg Von Kuster On Mar 20, 2014, at 9:46 AM, Briand, Sheldon sheldon.bri...@ssc-spc.gc.ca wrote: Hi, The repair operation doesn’t seem to make any difference (I already tried that before emailing). Screenshot of the result attached. -Sheldon From: Greg Von Kuster [mailto:g...@bx.psu.edu] Sent: Thursday, March 20, 2014 10:26 AM To: Briand, Sheldon Cc: 'galaxy-dev@lists.bx.psu.edu' Subject: Re: [galaxy-dev] tophat2 install Hi Sheldon, I recommend not manually changing any database entries. You should be able to install the repository and all of its dependencies using the Repair Repository feature. See https://wiki.galaxyproject.org/RepairingInstalledRepositories Greg Von Kuster On Mar 20, 2014, at 8:59 AM, Briand, Sheldon sheldon.bri...@ssc-spc.gc.ca wrote: Hi, Just wondering if I can still use the following to fix up my messed up tophat2 install (the install had hung up on me, for 2 hours, and I tried to remove/repair it via the interface). I haven’t had problems installing any other tools or packages. I am using postgres but this advice was given 2 years ago to another user and fields may have changed. Thanks for any help! -Sheldon 1. Manually remove the installed repository subdirectory hierarchy from disk. 2. If the repository included any tools, manually remove entries for each of them from the shed_tool-conf.xml file ( or the equivalent file you have configured for handling installed repositories ) 3. Manually update the database using the following command (assuming your installed repository is named 'bcftools_view' and it is the only repository you have installed with that name) - letter capitalization is required: The following assumes you're using postgres: update tool_shed_repository set deleted=True, uninstalled=True, status='Uninstalled', error_message=Null where name = 'bcftools_view'; From: Briand, Sheldon Sent: Wednesday, March 12, 2014 11:44 AM To: galaxy-dev@lists.bx.psu.edu Subject: tophat2 install Hi, I ran into a problem installing tophat2 here: tophat2 version 2.0.9: coercing to Unicode: need string or buffer, NoneType found I have only used the interface to do the installation (no manual deletes). The rest of the dependencies for tophat2 installed just fine. Repairing the repository doesn’t make any difference. Thanks! -Sheldon Sheldon Briand NRC Research Computing Support Analyst Research Computing Support / Soutien Informartique a la Recherche Operations, Science Portfolio / Operations, Portefeuil des sciences SSC-NRC / SPC-CNRC Rm 329A, 1411 Oxford Street / Piece 329A, 1411 Rue Oxford
Re: [galaxy-dev] tophat2 install
What version of Galaxy are you running? You want to look in the tophat2 repository directory, not its package dependency. There should be a tool_dependencies.xml file in the tophat2's installation directory hierarchy somewhere. On Mar 20, 2014, at 1:12 PM, Briand, Sheldon sheldon.bri...@ssc-spc.gc.ca wrote: Hi, dependancies/tophat2/2.0.9/devteam/package_tophat2_2_0_9/8549fd545473 contains only: env.sh If that isn’t the right place to look let me know. -Sheldon From: Greg Von Kuster [mailto:g...@bx.psu.edu] Sent: Thursday, March 20, 2014 2:04 PM To: Briand, Sheldon Cc: 'galaxy-dev@lists.bx.psu.edu' Subject: Re: [galaxy-dev] tophat2 install It looks like your database and file system are out of sync. Your database seems to think you have an installed repository but your file system does not seem to have the repository's installation directory. Can you confirm that this is the case? On Mar 20, 2014, at 11:53 AM, Briand, Sheldon sheldon.bri...@ssc-spc.gc.ca wrote: Hi, Here is the error from the paster.log: Traceback (most recent call last): File /BigData/galaxy/galaxy-dist/lib/tool_shed/util/common_install_util.py, line 496, in handle_tool_dependencies from_install_manager=from_install_manager ) File /BigData/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/tool_dependencies/install_util.py, line 318, in install_package from_install_manager=from_install_manager ) File /BigData/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/tool_dependencies/install_util.py, line 245, in handle_complex_repository_d ependency_for_package tool_dependencies_config = get_absolute_path_to_file_in_repository( repo_files_dir, 'tool_dependencies.xml' ) File /BigData/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/tool_dependencies/install_util.py, line 126, in get_absolute_path_to_file_i n_repository for root, dirs, files in os.walk( repo_files_dir ): File /opt/local/python-2.7.5/lib/python2.7/os.py, line 276, in walk names = listdir(top) TypeError: coercing to Unicode: need string or buffer, NoneType found -Sheldon From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Briand, Sheldon Sent: Thursday, March 20, 2014 10:52 AM To: 'Greg Von Kuster' Cc: 'galaxy-dev@lists.bx.psu.edu' Subject: Re: [galaxy-dev] tophat2 install Hi, Yes the error is: Error installing tool dependency package tophat2 version 2.0.9: coercing to Unicode: need string or buffer, NoneType found -Sheldon From: Greg Von Kuster [mailto:g...@bx.psu.edu] Sent: Thursday, March 20, 2014 10:49 AM To: Briand, Sheldon Cc: 'galaxy-dev@lists.bx.psu.edu' Subject: Re: [galaxy-dev] tophat2 install You should be able to click on the missing tool dependency and see its installation log. It should show you whatever error is occuring that is not allowing successful installation. Greg Von Kuster On Mar 20, 2014, at 9:46 AM, Briand, Sheldon sheldon.bri...@ssc-spc.gc.ca wrote: Hi, The repair operation doesn’t seem to make any difference (I already tried that before emailing). Screenshot of the result attached. -Sheldon From: Greg Von Kuster [mailto:g...@bx.psu.edu] Sent: Thursday, March 20, 2014 10:26 AM To: Briand, Sheldon Cc: 'galaxy-dev@lists.bx.psu.edu' Subject: Re: [galaxy-dev] tophat2 install Hi Sheldon, I recommend not manually changing any database entries. You should be able to install the repository and all of its dependencies using the Repair Repository feature. See https://wiki.galaxyproject.org/RepairingInstalledRepositories Greg Von Kuster On Mar 20, 2014, at 8:59 AM, Briand, Sheldon sheldon.bri...@ssc-spc.gc.ca wrote: Hi, Just wondering if I can still use the following to fix up my messed up tophat2 install (the install had hung up on me, for 2 hours, and I tried to remove/repair it via the interface). I haven’t had problems installing any other tools or packages. I am using postgres but this advice was given 2 years ago to another user and fields may have changed. Thanks for any help! -Sheldon 1. Manually remove the installed repository subdirectory hierarchy from disk. 2. If the repository included any tools, manually remove entries for each of them from the shed_tool-conf.xml file ( or the equivalent file you have configured for handling installed repositories ) 3. Manually update the database using the following command (assuming your installed repository is named 'bcftools_view' and it is the only repository you have installed with that name) - letter capitalization is required: The following assumes you're using postgres: update tool_shed_repository set deleted=True, uninstalled=True, status='Uninstalled', error_message=Null where name = 'bcftools_view'; From: Briand, Sheldon Sent: Wednesday, March 12, 2014 11:44 AM
Re: [galaxy-dev] tophat2 install
There have been several fixes released to the stable branch that you do not have in 12441. I would recommend doing the following: hg pull hg update stable Then uninstall the tophat2 repository and reinstall it. See if that makes a difference. On Mar 20, 2014, at 1:30 PM, Briand, Sheldon sheldon.bri...@ssc-spc.gc.ca wrote: $ hg summary parent: 12441:dc067a95261d From the tool_dependencies.xml: package name=tophat2 version=2.0.9 repository changeset_revision=8549fd545473 name=package_tophat2_2_0_9 owner=devteam prior_installation_required=False toolshed =http://toolshed.g2.bx.psu.edu; / From: Greg Von Kuster [mailto:g...@bx.psu.edu] Sent: Thursday, March 20, 2014 2:23 PM To: Briand, Sheldon Cc: 'galaxy-dev@lists.bx.psu.edu' Subject: Re: [galaxy-dev] tophat2 install What version of Galaxy are you running? You want to look in the tophat2 repository directory, not its package dependency. There should be a tool_dependencies.xml file in the tophat2's installation directory hierarchy somewhere. On Mar 20, 2014, at 1:12 PM, Briand, Sheldon sheldon.bri...@ssc-spc.gc.ca wrote: Hi, dependancies/tophat2/2.0.9/devteam/package_tophat2_2_0_9/8549fd545473 contains only: env.sh If that isn’t the right place to look let me know. -Sheldon From: Greg Von Kuster [mailto:g...@bx.psu.edu] Sent: Thursday, March 20, 2014 2:04 PM To: Briand, Sheldon Cc: 'galaxy-dev@lists.bx.psu.edu' Subject: Re: [galaxy-dev] tophat2 install It looks like your database and file system are out of sync. Your database seems to think you have an installed repository but your file system does not seem to have the repository's installation directory. Can you confirm that this is the case? On Mar 20, 2014, at 11:53 AM, Briand, Sheldon sheldon.bri...@ssc-spc.gc.ca wrote: Hi, Here is the error from the paster.log: Traceback (most recent call last): File /BigData/galaxy/galaxy-dist/lib/tool_shed/util/common_install_util.py, line 496, in handle_tool_dependencies from_install_manager=from_install_manager ) File /BigData/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/tool_dependencies/install_util.py, line 318, in install_package from_install_manager=from_install_manager ) File /BigData/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/tool_dependencies/install_util.py, line 245, in handle_complex_repository_d ependency_for_package tool_dependencies_config = get_absolute_path_to_file_in_repository( repo_files_dir, 'tool_dependencies.xml' ) File /BigData/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/tool_dependencies/install_util.py, line 126, in get_absolute_path_to_file_i n_repository for root, dirs, files in os.walk( repo_files_dir ): File /opt/local/python-2.7.5/lib/python2.7/os.py, line 276, in walk names = listdir(top) TypeError: coercing to Unicode: need string or buffer, NoneType found -Sheldon From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Briand, Sheldon Sent: Thursday, March 20, 2014 10:52 AM To: 'Greg Von Kuster' Cc: 'galaxy-dev@lists.bx.psu.edu' Subject: Re: [galaxy-dev] tophat2 install Hi, Yes the error is: Error installing tool dependency package tophat2 version 2.0.9: coercing to Unicode: need string or buffer, NoneType found -Sheldon From: Greg Von Kuster [mailto:g...@bx.psu.edu] Sent: Thursday, March 20, 2014 10:49 AM To: Briand, Sheldon Cc: 'galaxy-dev@lists.bx.psu.edu' Subject: Re: [galaxy-dev] tophat2 install You should be able to click on the missing tool dependency and see its installation log. It should show you whatever error is occuring that is not allowing successful installation. Greg Von Kuster On Mar 20, 2014, at 9:46 AM, Briand, Sheldon sheldon.bri...@ssc-spc.gc.ca wrote: Hi, The repair operation doesn’t seem to make any difference (I already tried that before emailing). Screenshot of the result attached. -Sheldon From: Greg Von Kuster [mailto:g...@bx.psu.edu] Sent: Thursday, March 20, 2014 10:26 AM To: Briand, Sheldon Cc: 'galaxy-dev@lists.bx.psu.edu' Subject: Re: [galaxy-dev] tophat2 install Hi Sheldon, I recommend not manually changing any database entries. You should be able to install the repository and all of its dependencies using the Repair Repository feature. See https://wiki.galaxyproject.org/RepairingInstalledRepositories Greg Von Kuster On Mar 20, 2014, at 8:59 AM, Briand, Sheldon sheldon.bri...@ssc-spc.gc.ca wrote: Hi, Just wondering if I can still use the following to fix up my messed up tophat2 install (the install had hung up on me, for 2 hours, and I tried to remove/repair it via the interface). I haven’t had problems installing any other tools or packages. I am using postgres but this advice was given 2 years ago to another user and fields may
Re: [galaxy-dev] tophat2 install
Sorry, should have advised this: hg pull https://bitbucket.org/galaxy/galaxy-central#stable hg update stable See the end ot this thread: http://dev.list.galaxyproject.org/Persistent-jobs-in-cluster-queue-even-after-canceling-job-in-galaxy-td4663719.html On Mar 20, 2014, at 1:51 PM, Björn Grüning bjoern.gruen...@gmail.com wrote: Hi, you need to track galaxy-central to make use of the patches that are applied as bugfixes after the release. You can do that easily with changing the path in .hg/hgrc ... but I do not know if that is the right way to do ;) For me it worked. Cheers, Bjoern Am 20.03.2014 18:46, schrieb Briand, Sheldon: pulling from https://bitbucket.org/galaxy/galaxy-dist searching for changes no changes found From: Greg Von Kuster [mailto:g...@bx.psu.edu] Sent: Thursday, March 20, 2014 2:36 PM To: Briand, Sheldon Cc: 'galaxy-dev@lists.bx.psu.edu' Subject: Re: [galaxy-dev] tophat2 install There have been several fixes released to the stable branch that you do not have in 12441. I would recommend doing the following: hg pull hg update stable Then uninstall the tophat2 repository and reinstall it. See if that makes a difference. On Mar 20, 2014, at 1:30 PM, Briand, Sheldon sheldon.bri...@ssc-spc.gc.camailto:sheldon.bri...@ssc-spc.gc.ca wrote: $ hg summary parent: 12441:dc067a95261d From the tool_dependencies.xml: package name=tophat2 version=2.0.9 repository changeset_revision=8549fd545473 name=package_tophat2_2_0_9 owner=devteam prior_installation_required=False toolshed =http://toolshed.g2.bx.psu.edu; / From: Greg Von Kuster [mailto:g...@bx.psu.eduhttp://bx.psu.edu] Sent: Thursday, March 20, 2014 2:23 PM To: Briand, Sheldon Cc: 'galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu' Subject: Re: [galaxy-dev] tophat2 install What version of Galaxy are you running? You want to look in the tophat2 repository directory, not its package dependency. There should be a tool_dependencies.xml file in the tophat2's installation directory hierarchy somewhere. On Mar 20, 2014, at 1:12 PM, Briand, Sheldon sheldon.bri...@ssc-spc.gc.camailto:sheldon.bri...@ssc-spc.gc.ca wrote: Hi, dependancies/tophat2/2.0.9/devteam/package_tophat2_2_0_9/8549fd545473 contains only: env.sh If that isn't the right place to look let me know. -Sheldon From: Greg Von Kuster [mailto:g...@bx.psu.eduhttp://bx.psu.edu] Sent: Thursday, March 20, 2014 2:04 PM To: Briand, Sheldon Cc: 'galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu' Subject: Re: [galaxy-dev] tophat2 install It looks like your database and file system are out of sync. Your database seems to think you have an installed repository but your file system does not seem to have the repository's installation directory. Can you confirm that this is the case? On Mar 20, 2014, at 11:53 AM, Briand, Sheldon sheldon.bri...@ssc-spc.gc.camailto:sheldon.bri...@ssc-spc.gc.ca wrote: Hi, Here is the error from the paster.log: Traceback (most recent call last): File /BigData/galaxy/galaxy-dist/lib/tool_shed/util/common_install_util.py, line 496, in handle_tool_dependencies from_install_manager=from_install_manager ) File /BigData/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/tool_dependencies/install_util.py, line 318, in install_package from_install_manager=from_install_manager ) File /BigData/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/tool_dependencies/install_util.py, line 245, in handle_complex_repository_d ependency_for_package tool_dependencies_config = get_absolute_path_to_file_in_repository( repo_files_dir, 'tool_dependencies.xml' ) File /BigData/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/tool_dependencies/install_util.py, line 126, in get_absolute_path_to_file_i n_repository for root, dirs, files in os.walk( repo_files_dir ): File /opt/local/python-2.7.5/lib/python2.7/os.py, line 276, in walk names = listdir(top) TypeError: coercing to Unicode: need string or buffer, NoneType found -Sheldon From: galaxy-dev-boun...@lists.bx.psu.edumailto:galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edumailto:dev-boun...@lists.bx.psu.edu] On Behalf Of Briand, Sheldon Sent: Thursday, March 20, 2014 10:52 AM To: 'Greg Von Kuster' Cc: 'galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu' Subject: Re: [galaxy-dev] tophat2 install Hi, Yes the error is: Error installing tool dependency package tophat2 version 2.0.9: coercing to Unicode: need string or buffer, NoneType found -Sheldon From: Greg Von Kuster [mailto:g...@bx.psu.edu] Sent: Thursday, March 20, 2014 10:49 AM To: Briand, Sheldon Cc: 'galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu' Subject: Re: [galaxy-dev] tophat2 install You should be able to click on the missing tool dependency and see its
Re: [galaxy-dev] Installing homer into galaxy
Hi Pete, Here is the recipe for installing homer from this repository: http://toolshed.g2.bx.psu.edu/view/kevyin/homer ?xml version=1.0? tool_dependency package name=homer version=4.1 install version=4.1 actions action type=download_by_urlhttp://biowhat.ucsd.edu/homer/configureHomer.pl/action action type=shell_commandperl ./configureHomer.pl -install/action !--action type=shell_commandperl ./configureHomer.pl -install hg19/action-- action type=move_directory_files source_directory.//source_directory destination_directory$INSTALL_DIR/destination_directory /action action type=set_environment environment_variable name=PATH action=prepend_to$INSTALL_DIR/bin/environment_variable /action /actions /install readme I'm sorry but this does not work /readme /package /tool_dependency The problem is tha the following tag: install version=4.1 must be: install version=1.0 I'm pretty sure a message was displayed when the file was uploaded, but it's been a while since I looked at thie relevant code, so I'm not positive. Gfreg Von Kuster On Mar 20, 2014, at 2:52 PM, Pete Schmitt peter.r.schm...@dartmouth.edu wrote: I've installed homer 4.5 on the server that hosts the latest galaxy version, and ensured that it is in galaxy's path (in run.sh) when it starts up. When installing the wrapper from the main toolshed I get the following error which I don't understand: Tool shed repository: homer Tool shed repository changeset revision: f0b5827b6051 Tool dependency status: Error Tool dependency installation error: Error installing tool dependency package homer version 4.1: Only install version 1.0 is currently supported (i.e., change your tag to be ). Tool dependency installation directory: /galaxy/tools/homer/4.1/kevyin/homer/f0b5827b6051 The following is the README from the homer wrapper that I installed: Homer wrapper for Galaxy The homer tools will need to be accessible from command line Code repo: https://bitbucket.org/gvl/homer =: LICENSE for this wrapper: =: Kevin Ying Garvan Institute: http://www.garvan.org.au GVL: https://genome.edu.au/wiki/GVL http://opensource.org/licenses/mit-license.php ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Installing homer into galaxy
If you make the correction in your local version it should install. However, getting updates (if they become available) may require merging. On Mar 20, 2014, at 3:19 PM, Pete Schmitt peter.r.schm...@dartmouth.edu wrote: Can I just edit this file to point to the existing homer install on the server? I found this in the tool shed tree. On 3/20/14, 3:03 PM, Greg Von Kuster wrote: Hi Pete, Here is the recipe for installing homer from this repository: http://toolshed.g2.bx.psu.edu/view/kevyin/homer ?xml version=1.0? tool_dependency package name=homer version=4.1 install version=4.1 actions action type=download_by_urlhttp://biowhat.ucsd.edu/homer/configureHomer.pl/action action type=shell_commandperl ./configureHomer.pl -install/action !--action type=shell_commandperl ./configureHomer.pl -install hg19/action-- action type=move_directory_files source_directory.//source_directory destination_directory$INSTALL_DIR/destination_directory /action action type=set_environment environment_variable name=PATH action=prepend_to$INSTALL_DIR/bin/environment_variable /action /actions /install readme I'm sorry but this does not work /readme /package /tool_dependency The problem is tha the following tag: install version=4.1 must be: install version=1.0 I'm pretty sure a message was displayed when the file was uploaded, but it's been a while since I looked at thie relevant code, so I'm not positive. Gfreg Von Kuster On Mar 20, 2014, at 2:52 PM, Pete Schmitt peter.r.schm...@dartmouth.edu wrote: I've installed homer 4.5 on the server that hosts the latest galaxy version, and ensured that it is in galaxy's path (in run.sh) when it starts up. When installing the wrapper from the main toolshed I get the following error which I don't understand: Tool shed repository: homer Tool shed repository changeset revision: f0b5827b6051 Tool dependency status: Error Tool dependency installation error: Error installing tool dependency package homer version 4.1: Only install version 1.0 is currently supported (i.e., change your tag to be ). Tool dependency installation directory: /galaxy/tools/homer/4.1/kevyin/homer/f0b5827b6051 The following is the README from the homer wrapper that I installed: Homer wrapper for Galaxy The homer tools will need to be accessible from command line Code repo: https://bitbucket.org/gvl/homer =: LICENSE for this wrapper: =: Kevin Ying Garvan Institute: http://www.garvan.org.au GVL: https://genome.edu.au/wiki/GVL http://opensource.org/licenses/mit-license.php ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ -- Pete Schmitt Technical Director: Discovery Cluster NH INBRE Grid Computational Genetics Lab Institute for Quantitative Biomedical Sciences Dartmouth College, HB 6203 L12 Berry/Baker Library Hanover, NH 03755 Phone: 603-646-8109 http://discovery.dartmouth.edu http://columbia.dartmouth.edu/grid http://www.epistasis.org http://iQBS.org dartmouth.png ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] http://toolshed.g2.bx.psu.edu/ is down?
Yes, both the test and main Galaxy Tool Sheds are temporarily unavailable due to a hardware move that is in progress. I just learned of this this morning, so I didn't get a chance to send out an announcement. Sorry for the inconvenience! We'll send another announcement as soon as the Tool Sheds are available. Greg Von Kuster On Mar 17, 2014, at 1:17 PM, changyu changyu_...@mail.dfci.harvard.edu wrote: Is http://toolshed.g2.bx.psu.edu/ down? ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] http://toolshed.g2.bx.psu.edu/ is down?
Both the test and main Galaxy Tool Sheds should now be available. Please let us know if you encounter issues. Thanks, Greg Von Kuster On Mar 17, 2014, at 1:31 PM, Greg Von Kuster g...@bx.psu.edu wrote: Yes, both the test and main Galaxy Tool Sheds are temporarily unavailable due to a hardware move that is in progress. I just learned of this this morning, so I didn't get a chance to send out an announcement. Sorry for the inconvenience! We'll send another announcement as soon as the Tool Sheds are available. Greg Von Kuster On Mar 17, 2014, at 1:17 PM, changyu changyu_...@mail.dfci.harvard.edu wrote: Is http://toolshed.g2.bx.psu.edu/ down? ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] tool_dependency_dir (Installed, missing tool dependencies)
Hello, It's difficult to determine the cause of the problem in your environment based on the details you've provided. If you have defined your tool_dependeny_dir configuration setting in your universe_wsgi.ini to be ../tool_dependencies, then the tool shed installation process will install tool dependencies there. If you see them installed elsewhere, then that setting must have been set differently at the time they were installed. You can try setting it to the location where your tool dependencies have been installed (i.e., the Galaxy installation directory), restat your server and uninstall them. Then you can reset it to the more appropriate location (e.g., ../tool_dependencies), restart yiour server and reinstall them. If you have tool dependencies installed in multiple locations, you'll have to be careful to use a process that will eventually result in all tool dependencies installed into a single location. Greg Von Kuster On Mar 6, 2014, at 5:41 PM, Wang, Xiaofei xfw...@ku.edu wrote: Dear there, I am trying to install some tools from toolshed on my local galaxy instance, like bwa_wrappers, package_picard_1_56_0, package_samtools_0_1_18, and sam_to_bam. For the first time, I got a warning before installing, something like this set the value of your tool_dependency_dir setting in universe_wsgi_ini and restart before installing. Then, I followed the answer on this thread (Problem installing tool_ Galaxy local) as below. I edited the universe_wsgi_ini like this: #Path to the directory in which managed tool dependencies are placed. To use # the dependency system, see the documentation at: # http://wiki.g2.bx.psu.edu/Admin/Config/Tool%20Dependencies #tool_dependency_dir = None tool_dependency_dir = ../tool_dependencies Then, I restarted and installed the tool again. Galaxy create a directory named tool_dependencies at the same level in local file system hierarchy as the galaxy installation directory (same level as galaxy-dist). Some softwares such as bwa and picard are installed automatically here. But, when I checked in the admin manage installed tool shed repositories, the installation status is Installed, missing tool dependencies for the four tools I mentioned above. I do not know why this happened in my case? I tried to figure it out follow this thread The migrated BWA installed but missing dependencies. But, I did not get the idea. Could you tell me why this happen and how to fix it up? Thank you so much! -- Your setting for tool_dependency_dir should not be as you've stated: tool_dependency_dir = tool_dependencies It should be an actual path on your local file system, something like: tool_dependency_dir = ../tool_dependencies If you use the above setting, then Galaxy should create a directory named tool_dependency_dir at the same level in your local file system hierarchy as your Galaxy installation directory. I cannot guarantee that this change will solve your problem, but it will be a step in the right direction. After making this change, let us know if you still see problems. Greg Von Kuster ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] tool trouble
Hello Sheldon, On Mar 7, 2014, at 2:14 PM, Briand, Sheldon sheldon.bri...@ssc-spc.gc.ca wrote: Hi, I've finally figured out what is happening with galaxy on my end regarding tools not showing up in the menu. I moved my install to a new larger file system and started from scratch wiping out the database to rule out my having messed up the galaxy instance. The first tool I tried was one I had successfully installed on my previous setup. I tried to install package_cufflinks_2_1_1. One thing I noticed during the initial install setup screens is that it didn't ask me where I wanted to put the cufflinks tool in my menu. After the package had successfully installed the tool does not show up in the Analyze data tool list. The above repository contains only a recipe for installing a tool dependency, specifically version 2.1.1 of the cufflinks package. This installed package can be used by Galaxy tools that are containe din other repositories you may install. The reason you were not asked which tool sections to use is because the repository does not contain any tools. I then tried to install the cufflinks tool which has the repository dependency of package_cufflinks_2_1_1. During the initial install screens I'm given the option of putting the tool in one of my existing tool panel sections or adding a new one. I installed and the tool shows up in the Analyze Data tool list where it's expected. Again, the above behavior is due to the fact that you installed a repository that contained a tool, not just a tool dependency. I'm just wondering if this is expected behavior or it's something I've configured wrong. It is expected behavior. Usually you would have just elected to install the repository that contains the cufflinks tool itself and checked the checkbox to install all of its repository and tool dependencies rather than doing this in separate steps. In cases such as bowtie 1 (I could use bowtie 0_12_7 and use the bowtie wrapper), I only have the package option and so I can't get the tool to appear in my menu. Again, you'll see the page asking you what tool panel section you want tools contained in the repository to show up in only if the repository contains tools for display in the tool panel. I think you may be confusing Galaxy tools with tool dependencies. Thanks! -Sheldon -Original Message- From: Briand, Sheldon Sent: Monday, March 03, 2014 2:16 PM To: 'Greg Von Kuster' Subject: RE: [galaxy-dev] tool trouble Hi, Not sure if you were still scratching your head about this one. I am going to be moving my galaxy install to a new larger file system and put it in production so I thought I would do a new install as my current one was just testing. If you were still going to be looking at it don't bother. Thanks for all you had done on it up to now. Cheers! -Sheldon -Original Message- From: Briand, Sheldon Sent: Friday, February 28, 2014 6:38 PM To: 'Greg Von Kuster' Cc: galaxy-dev@lists.bx.psu.edu Subject: RE: [galaxy-dev] tool trouble Hi Greg, Also for bowtie2. I can confirm that it doesn't have an entry in the shed_tool_conf.xml file. Sorry I didn't mention these before. Its been a long week and there obviously was a problem between my chair and keyboard. :) -Sheldon From: galaxy-dev-boun...@lists.bx.psu.edu [galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Briand, Sheldon [sheldon.bri...@ssc-spc.gc.ca] Sent: Friday, February 28, 2014 6:02 PM To: 'Greg Von Kuster' Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] tool trouble Hi, So those are the ones that work. One of the ones that doesn't work is bowtie2: tool_shed.galaxy_install.installed_repository_manager DEBUG 2014-02-28 16:03:14,476 Adding an entry for revision 017a00c265f1 of repository package_bowtie2_2_1_0 owned by devteam to repository_dependencies_of_installed_repositories. tool_shed.galaxy_install.installed_repository_manager DEBUG 2014-02-28 16:03:14,490 Adding an entry for revision 017a00c265f1 of reposito ry package_bowtie2_2_1_0 owned by devteam to installed_repository_dependencies_of_installed_repositories. tool_shed.galaxy_install.installed_repository_manager DEBUG 2014-02-28 16:03:14,502 Adding an entry for revision 017a00c265f1 of reposito ry package_bowtie2_2_1_0 owned by devteam to tool_dependencies_of_installed_repositories. tool_shed.galaxy_install.installed_repository_manager DEBUG 2014-02-28 16:03:14,516 Adding an entry for revision 017a00c265f1 of reposito ry package_bowtie2_2_1_0 owned by devteam to installed_tool_dependencies_of_installed_repositories. tool_shed.galaxy_install.installed_repository_manager DEBUG 2014-02-28 16:03:45,046 Adding an entry for version 2.1.0 of package bowtie2 to installed_runtime_dependent_tool_dependencies_of_installed_tool_dependencies
Re: [galaxy-dev] Nightly testing status on the (Test) Tool Shed
Hello Peter, On Feb 20, 2014, at 10:20 AM, Greg Von Kuster g...@bx.psu.edu wrote: On Thu, Feb 20, 2014 at 9:59 AM, Peter Cock p.j.a.c...@googlemail.com wrote: On Thu, Feb 20, 2014 at 2:56 AM, Greg Von Kuster g...@bx.psu.edu wrote: Your installation recipe for mira attempts to download a binary and if that fails, it echoes an error, but still performs set_environment actions.. Ah - I can see that now, I need a fall back action tag which either tries to compile MIRA or raises an explicit error. Thanks! Sorry, slightly confused: there was a fallback action tag which was and is meant to raise an error. It was accidentally raising an error due to a bash syntax error in my unquoted echo (which that detailed log you posted alerted me to), now fixed in my repository and uploaded to the Test Tool Shed: https://github.com/peterjc/pico_galaxy/commit/4b003cf9b5e6baa15c75bb65723c5af78e40c18d http://testtoolshed.g2.bx.psu.edu/view/peterjc/mira4_assembler The BLAST+ packages have the same glitch in their fall-back (so I will need to update them on the main and test Tool Shed, although I will delay that pending your reply to the problem below): https://github.com/peterjc/galaxy_blast/commit/4b2fa1fbf485c4630cb6dc0beaf7286cb103bd71 So, what happens is first the arch/os specific actions is tried, here actions os=linux architecture=x86_64 That was failing with MIRA4 due to the symlink bug (now fixed). Because the platform specific action failed, the generic actions was used - where I deliberately try to raise an error to signal the installation failed. What is surprising me is that the Tool Shed see the error, logs it, but still continues on (setting my environment variables, and reporting success). Why is that? I'll need some time to investigate this behavior - if I discover a framework issue, I'll provide a fix. The above issue should be corrected in https://bitbucket.org/galaxy/galaxy-central/commits/f5a119ac52a957204ab968a78638fe2a942c6893 which is currently running on the Test Tool Shed. I'm going to wait for tonight's Install and Test run on the Test Tool Shed to make sure all is well with the fix. If things look good we'll graft the fix to the stable branch tomorrow. Thanks for reporting this! As we briefly discussed earlier, your mira4 recipe is not currently following best practices. Although you uncovered a problem in the framework which has now been corrected, your recipe's fall back actions tag set should be the recipe for installing mira4 from source ( http://sourceforge.net/projects/mira-assembler/ ) since there is no licensing issues for doing so. This would be a more ideal approach than echoing the error messages. Thanks very much for helping us discover this problem though! Greg Von Kuster Thanks, Peter P.S. The assert file exists action I suggested previously would be a neat way to catch some failing installs, e.g. here I expected the executables $MIRA4/mira and $MIRA4/mirabait etc to exist. I've filed a Trello card for this enhancement idea: https://trello.com/c/xBUhFvj0/1446-add-assert-file-exists-and-directory-exists-actions-to-tool-dependencies-xml Thanks - I've moved it to the Tool Shed Trello Board - its in Project in Planning. Greg Von Kuster ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Nightly testing status on the (Test) Tool Shed
Hi Peter, On Mar 6, 2014, at 1:07 PM, Peter Cock p.j.a.c...@googlemail.com wrote: On Thu, Mar 6, 2014 at 5:24 PM, Peter Cock p.j.a.c...@googlemail.com wrote: On Thu, Mar 6, 2014 at 4:53 PM, Greg Von Kuster g...@bx.psu.edu wrote: On Feb 20, 2014, at 10:20 AM, Greg Von Kuster g...@bx.psu.edu wrote: On Thu, Feb 20, 2014 at 9:59 AM, Peter Cock p.j.a.c...@googlemail.com wrote: ... So, what happens is first the arch/os specific actions is tried, here actions os=linux architecture=x86_64 That was failing with MIRA4 due to the symlink bug (now fixed). Because the platform specific action failed, the generic actions was used - where I deliberately try to raise an error to signal the installation failed. What is surprising me is that the Tool Shed see the error, logs it, but still continues on (setting my environment variables, and reporting success). Why is that? I'll need some time to investigate this behavior - if I discover a framework issue, I'll provide a fix. The above issue should be corrected in https://bitbucket.org/galaxy/galaxy-central/commits/f5a119ac52a957204ab968a78638fe2a942c6893 which is currently running on the Test Tool Shed. I'm going to wait for tonight's Install and Test run on the Test Tool Shed to make sure all is well with the fix. If things look good we'll graft the fix to the stable branch tomorrow. Thanks for reporting this! ... Hi Greg, No problem - I'm good at discovering problems ;) If the download approach failed, it it most likely due to a transient error (e.g. network issues with download). Here I would much prefer Galaxy aborted and reported this as an error (and does not attempt the default action). Is that what you just fixed? Re-reading the commit, and in particular the comment, it sounds like you have not fixed the larger issue I am trying to flag, quoting the commit comment: Fixes to handle case where a tool dependency recipe defines an actions_group tag set for installing pre-compiled binaries which fail to install, so the fall back recipe for installing and compileing from souce is used. Prior to this change set, the recipe from source would not work, but the install process would finish in such a way that the recipe seemed to work. i.e. The fall back action in my MIRA4 and BLAST+ recipes where I use false to raise an error should now be treated as an error (rather than as before being a silent failure). That's progress :) Yes, thanks! Given the current behaviour of switching to the default action if the platform specific action fails (which I think is unwise), It's fine if you want to display errors rather than an install from source recipe. You have ample justification for that approach in this case. does the Tool Shed clean up the failed platform specific installation before attempting the default action? Yes. e.g. The failed action may have downloaded and unzipped some files which could interfere. The tool shed will inspect and remove the contents of the installation directory before attempting to install from source. Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Test ToolShed misfiling missing tool tests
Hi peter, Thanks for reposrting this. I've created the follwoing Trello card for this issue, and we'll get it taken care of asap. https://trello.com/c/rArkn49z/173-incorrect-tool-test-results-for-tools-missing-tool-test-components Gfreg Von Kuster On Mar 3, 2014, at 8:38 AM, Peter Cock p.j.a.c...@googlemail.com wrote: Hello all, Currently the Test Tool Shed seems not to be flagging all tools with missing tests. For example, the following currently lack tests yet are not listed under Latest revision: missing tool tests but instead under Latest revision: all tool tests pass: http://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus/9fc8b8a639c9 http://testtoolshed.g2.bx.psu.edu/view/peterjc/mira4_assembler/4c6e4d583b8f Viewing these repositories they are shown to have tools without tests. Note that some tools are correctly listed under Latest revision: missing tool tests, e.g. http://testtoolshed.g2.bx.psu.edu/view/peterjc/clc_assembly_cell http://testtoolshed.g2.bx.psu.edu/view/peterjc/mira_assembler Regards, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Exporting Importing Tool Shed repository is just great!
Many thanks, Björn for your very kind words. The Tool Shed would not be what it is without your hard work as well. We hoped the export/import capsule feature would be beneficial, so it's great to hear confirmation that it is. Also, please don't hesitate to continue to contact us with newly discovered issues! Greg Von Kuster On Mar 3, 2014, at 12:02 PM, Björn Grüning bjoern.gruen...@gmail.com wrote: Hi Tool Shed Team! I know I'm always finding bugs and keeping you busy, but this mail is different! The Tool Shed exporting and importing feature is just great, its save so much time and is a pleasure to use. Everyone should try it! So thanks for the hard work you all put into the Tool Shed over the last month. I really enjoy to put my tools into it! Thanks, Bjoern ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Exporting Importing Tool Shed repository is just great!
On Mar 3, 2014, at 1:13 PM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi Guys, Is this stuff on the wiki? Not completely yet. Presumably the goal is helping to move repositories between Tool Shed instances (e.g. from a development/private Tool Shed to a production/public Tool Shed)? Yes. I can see Import repository capsule as a new action at the bottom of the left hand column (under Create new repository). Is there an Export repository capsule entry somewhere? I can see a Export this revision action on each repository, is that it? Yes. Thanks, Peter On Mon, Mar 3, 2014 at 5:58 PM, Greg Von Kuster g...@bx.psu.edu wrote: Many thanks, Björn for your very kind words. The Tool Shed would not be what it is without your hard work as well. We hoped the export/import capsule feature would be beneficial, so it's great to hear confirmation that it is. Also, please don't hesitate to continue to contact us with newly discovered issues! Greg Von Kuster On Mar 3, 2014, at 12:02 PM, Björn Grüning bjoern.gruen...@gmail.com wrote: Hi Tool Shed Team! I know I'm always finding bugs and keeping you busy, but this mail is different! The Tool Shed exporting and importing feature is just great, its save so much time and is a pleasure to use. Everyone should try it! So thanks for the hard work you all put into the Tool Shed over the last month. I really enjoy to put my tools into it! Thanks, Bjoern ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] tool trouble
Hello Briand, This may take some back-and-forth with you to understand the steps you have taken so far. Successfully installing a repository that contains tools usually results in the tools being displayed in the Galaxy tool panel. However, there may be issues that result in the tools not properly loading. This often occurs when you stop and restart your Galaxy instance. Your Galaxy pasxter log will include information about any tool that does not properly load. Uninstalling and reinstalling the repository will usually not have any affect on this. On Feb 28, 2014, at 9:17 AM, Briand, Sheldon sheldon.bri...@ssc-spc.gc.ca wrote: Hi, I’ve been poking around a bit more. It looks as though galaxy isn’t actually downloading and installing the tools. The tool status changes to installed in the web interface and postgres but the tool is not present on disk. What is present on disk? If you install a repository with tools, the repository revision's contents should be on disk. However, if the repository is uninstalled, disk files are deleted. Do you have repositories that are installed but do not have any files on disk? The log below shows no download and make of the tool and dependancies. Hopefully the log gives a bit more information. Sorry for the second email. tool_shed.util.shed_util_common DEBUG 2014-02-28 09:58:48,567 Updating an existing row for repository 'tophat2' in the tool_shed_repository table, status set to 'New'. tool_shed.util.repository_dependency_util DEBUG 2014-02-28 09:58:48,714 Creating repository dependency objects... The log looks ok at first inspection, but it's not useful without more information. From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Briand, Sheldon Sent: Thursday, February 27, 2014 8:18 PM To: galaxy-dev@lists.bx.psu.edu Subject: [galaxy-dev] tool trouble Hi, I’m having trouble with some of my tools not appearing in the Analyze Data menu. I’ve successfully installed the tools via the manage installed tool shed repositories(so they are installed to disk). I have tried uninstalling and installing again. The above should not be necessary. When you install, are you checking the optional checkboxes to install repository dependencies and tool dependencies? If so, are they installing as well? The tool installs but does not appear in the menu. Do you have an example of a repository that contains tools that installls corruptly, but the tools do not load into the tool panel? Other tools install the menu updates just fine. Do you have an example of a repsoitory that contains tools that do show up in the tool panel after successfule repository installation? This is probably due to me corrupting the misbehaving tools installation somehow. I’m guessing that I need to uninstall the tool and wipe some postgres entries. I strongly advise not manually changing inaything in your database suing sql commands. If you have done this, it could be the cause of the behavor you are describing. The Galaxy UI or API should be used. Greg Von Kuster My shed tool path: toolbox tool_path=../shed_tools My current version: parent: 12441:dc067a95261d Added tag release_2014.02.10 for changeset 5e605ed6069f branch: stable Could someone point me in the right direction? Thanks, -Sheldon Sheldon Briand NRC Research Computing Support Analyst Research Computing Support / Soutien Informartique a la Recherche Operations, Science Portfolio / Operations, Portefeuil des sciences SSC-NRC / SPC-CNRC Rm 329A, 1411 Oxford Street / Piece 329A, 1411 Rue Oxford Halifax, NS B3H 3Z1 902 426-1677 sheldon.bri...@ssc-spc.gc.ca ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] tool trouble
Hello Sheldon, What is the configuration setting you are using for the location to install repositories? This is defined by the tool_path attribute of the toolbox tag in your shed_tools_conf.xml file. The default is: toolbox tool_path=../shed_tools On Feb 28, 2014, at 3:47 PM, Briand, Sheldon sheldon.bri...@ssc-spc.gc.ca wrote: Hi, No joy on the migration strategy. Deseq is example of a tool that I installed successfully yesterday. Do you mean that you installed the deseq repository from the main tool shed: http://toolshed.g2.bx.psu.edu/view/nikhil-joshi/deseq_and_sam2counts and that its contained deseq tool DE Seq (version 1.0.0) was properly displayed in your Galaxy tool panel? If so, what is now happening? Bowtie2 is an example of a tool that isn’t showing up even though the web interface indicates that it is successfully installed. Can you uninstall the repsository and reinstall it? If so, what happens? Here is my directory structure relating to bowtie2: $ find . -name *bowtie2* ./shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/bowtie2 ./shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/package_bowtie2_2_1_0 ./shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/package_bowtie2_2_1_0/017a00c265f1/package_bowtie2_2_1_0 ./shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/tophat2/ffa30bedbee3/tophat2/.hg/store/data/tool-data/bowtie2__indices.loc.sample.i ./shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/tophat2/ffa30bedbee3/tophat2/tool-data/bowtie2_indices.loc.sample ./galaxy-dist/.hg/store/data/tool-data/bowtie2__indices.loc.sample.i ./galaxy-dist/.hg/store/data/tools/sr__mapping/bowtie2__wrapper.xml.i ./galaxy-dist/.hg/store/data/tools/sr__mapping/bowtie2__wrapper.py.i ./galaxy-dist/.hg/store/data/test-data/bowtie2 ./galaxy-dist/tool-data/bowtie2_indices.loc.sample ./galaxy-dist/tool-data/bowtie2_indices.loc ./galaxy-dist/tool-data/toolshed.g2.bx.psu.edu/repos/devteam/tophat2/ffa30bedbee3/bowtie2_indices.loc ./galaxy-dist/dependancies/samtools/0.1.18/devteam/bowtie2 ./galaxy-dist/dependancies/samtools/0.1.19/iuc/package_samtools_0_1_19/00e17a794a2e/bin/bowtie2sam.pl ./galaxy-dist/dependancies/bowtie2 ./galaxy-dist/dependancies/bowtie2/2.1.0/devteam/bowtie2 ./galaxy-dist/dependancies/bowtie2/2.1.0/devteam/package_bowtie2_2_1_0 ./galaxy-dist/dependancies/bowtie2/2.1.0/devteam/package_bowtie2_2_1_0/017a00c265f1/bowtie2-align ./galaxy-dist/dependancies/bowtie2/2.1.0/devteam/package_bowtie2_2_1_0/017a00c265f1/bowtie2-build-debug ./galaxy-dist/dependancies/bowtie2/2.1.0/devteam/package_bowtie2_2_1_0/017a00c265f1/bowtie2-inspect-debug ./galaxy-dist/dependancies/bowtie2/2.1.0/devteam/package_bowtie2_2_1_0/017a00c265f1/bowtie2 ./galaxy-dist/dependancies/bowtie2/2.1.0/devteam/package_bowtie2_2_1_0/017a00c265f1/bowtie2-inspect ./galaxy-dist/dependancies/bowtie2/2.1.0/devteam/package_bowtie2_2_1_0/017a00c265f1/bowtie2-align-debug ./galaxy-dist/dependancies/bowtie2/2.1.0/devteam/package_bowtie2_2_1_0/017a00c265f1/bowtie2-build [galaxy@eugene galaxy]$ cd shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/ I seem to be missing the .py and .xml files in the shed_tools directory. -Sheldon From: Briand, Sheldon Sent: Friday, February 28, 2014 2:11 PM To: 'Greg Von Kuster' Subject: RE: [galaxy-dev] tool trouble Hi, Thanks for getting back to me. I didn’t get your first reply (email trouble here) when I sent my second email. I have not played with the sql tables. I think my problem is that I need to run tool migration scripts. The reinstall log I sent was for tophat 2. It didn’t show up in shed_tool_conf.xml after the update so I thought it had once again failed. After I sent the email I noticed that tophat2 had shown up in the tool menu (not where I wanted it but it was there). So I looked around and there was an entry in migrated_tools_conf.xml. I think the tools I am missing are because I likely missed the migration step. I’ll try that first and report back. Cheers, -Sheldon From: Greg Von Kuster [mailto:g...@bx.psu.edu] Sent: Friday, February 28, 2014 11:04 AM To: Briand, Sheldon Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] tool trouble Hello Briand, This may take some back-and-forth with you to understand the steps you have taken so far. Successfully installing a repository that contains tools usually results in the tools being displayed in the Galaxy tool panel. However, there may be issues that result in the tools not properly loading. This often occurs when you stop and restart your Galaxy instance. Your Galaxy pasxter log will include information about any tool that does not properly load. Uninstalling and reinstalling the repository will usually not have any affect on this. On Feb 28, 2014, at 9:17 AM, Briand, Sheldon sheldon.bri...@ssc-spc.gc.ca wrote: Hi, I’ve been poking around a bit more. It looks as though
Re: [galaxy-dev] contributing patches to toolshed repos...
Hi Brad and Björn, Tickets can currently only be added to the Galaxy Trello Board, so please add them there and we'll move them to the Tool Shed Trello board whenever appropriate. Hopefully we'll soon have the ability for tickets to be submitted to multiple boards. Thanks very much! Greg Von Kuster On Feb 25, 2014, at 3:34 PM, Björn Grüning bjoern.gruen...@gmail.com wrote: Hi Brad, interesting. I have CC'ed Greg, maybe he can get help us. In the meantime, please create it under the Galaxy ToolShed. It's probably not really defined in which board we should create Tool issues. Thanks! Bjoern Hi Bjorn: I don’t appear to have permission to create cards on the the galaxy-tool-shed board… I can create cards on the main board via http://galaxyproject.org/trello but I can’t find an analog for the toolshed board. Am I missing something? Brad -- Brad Langhorst, Ph.D. Applications and Product Development Scientist On Feb 25, 2014, at 11:53 AM, Björn Grüning bjoern.gruen...@gmail.com wrote: Hi Brad, unfortunately there is no standard way to contribute back to the repositories. Many of us have a github/bitbucket repositories with the wrappers. There are a few tickets to make that best practise, for example: https://trello.com/c/WdShmMld For the time being, please create a trello card at: https://trello.com/b/vHqpKpZF/galaxy-tool-shed and attach your patch. If you can't attach it, I will do it for you. Thanks a lot for your contribution! Bjoern What’s the right way to contribute additions to other people’s repos? I have extended the picard wrappers to support the rna-seq metrics and downsample sam tools, but I don’t want to start a new repo just for these. I have attached the output from hg outgoing -p in case that’s useful ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Test Tool Shed, nightly tests: Error 'unicode' object has no attribute 'get'
Hello Peter, On Feb 24, 2014, at 3:32 AM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi Greg, Good progress - no sign of the unicode error :) I now have the following three repositories shown under Latest revision: installation errors http://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus (Seems to list both a failed and a successful BLAST+ 2.2.29 install?) http://testtoolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_26 (No error?) http://testtoolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_28 (No error?) On the other hand, http://testtoolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_27 (No test/install results?) http://testtoolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_29 Looks good. So it seems there is still something not quite right here… We've eliminated the use of fabric for installing tool dependencies on the test tool shed, and with this change I've seen the test tool shed fail at the same point the past 3 runs. It fails attempting to compile boost where it times out ( we have abuildbot timeout set at 36000 seconds if nothing occurs). This error occurs about half way through Stage 1 of the test run, so about 1/2 of the tool dependency definition repositories are not tested and no tests are run for repositories with tools. The following log shows the repeaated compilation error. We'll be investigating approaches for handling issues like this. Now that we've eliminated fabric we'll have many more options for handling installation issues like this. Of course, if it turns out that our new code that eliminated fabric caused this, we'll get a fix in. Thanks, Greg Von Kuster fabric_util.STDOUT DEBUG 2014-02-23 20:20:14,740 -- NUMBER_OF_JOBS: 2 (maximal number of concurrent compile jobs) fabric_util.STDOUT DEBUG 2014-02-23 20:20:14,741 -- Extracting BOOST .. fabric_util.STDOUT DEBUG 2014-02-23 20:20:20,006 -- Extracting BOOST .. done fabric_util.STDOUT DEBUG 2014-02-23 20:20:20,006 -- Bootstrapping Boost libraries (./bootstrap.sh --prefix=/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/tool_dependency_definitions/tmp/tmpShf1jI/tmp-toolshed-mtdq8JEyP/OpenMS-1.11.1/contrib --with-libraries=date_time,iostreams,math,regex) ... fabric_util.STDOUT DEBUG 2014-02-23 20:20:37,231 -- Bootstrapping Boost libraries (./bootstrap.sh --prefix=/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/tool_dependency_definitions/tmp/tmpShf1jI/tmp-toolshed-mtdq8JEyP/OpenMS-1.11.1/contrib --with-libraries=iostreams,math,date_time,regex) ... done fabric_util.STDOUT DEBUG 2014-02-23 20:20:37,232 -- Installing Boost libraries (./bjam -j 2 -sZLIB_SOURCE=/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/tool_dependency_definitions/tmp/tmpShf1jI/tmp-toolshed-mtdq8JEyP/OpenMS-1.11.1/contrib/src/zlib-1.2.5 -sBZIP2_SOURCE=/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/tool_dependency_definitions/tmp/tmpShf1jI/tmp-toolshed-mtdq8JEyP/OpenMS-1.11.1/contrib/src/bzip2-1.0.5 link=static cxxflags=-fPIC install --build-type=complete --layout=tagged --threading=single,multi) ... fabric_util.STDOUT DEBUG 2014-02-23 20:36:11,619 -- Installing Boost libraries (./bjam -j 2 -sZLIB_SOURCE=/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/tool_dependency_definitions/tmp/tmpShf1jI/tmp-toolshed-mtdq8JEyP/OpenMS-1.11.1/contrib/src/zlib-1.2.5 -sBZIP2_SOURCE=/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/tool_dependency_definitions/tmp/tmpShf1jI/tmp-toolshed-mtdq8JEyP/OpenMS-1.11.1/contrib/src/bzip2-1.0.5 link=static cxxflags=-fPIC install --build-type=complete --layout=tagged --threading=single,multi) ... failed command timed out: 36000 seconds without output, attempting to kill process killed by signal 9 Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] With regard to Galaxy tools, a repository should only contain one
Hi Peter, On Feb 21, 2014, at 9:10 AM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi all, I've just seen Greg's blog post, The Galaxy Tool Shed: Best Practices for Populating a Repository http://gregvonkuster.org/galaxy-tool-shed-best-practices-populating-repository/ One particular point that caught my attention (which I've used as the email title), which seems to signal a move away from Tool Suites as a single repository (although they can presumably be emulated by a dummy package defining dependencies on the child tools?). With regard to Galaxy tools, a repository should only contain one. I look forward to future posts (or emails?) clarifying this… Full clarification will be posted very soon…. Regards, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Test Tool Shed, nightly tests: Error 'unicode' object has no attribute 'get'
Hello Peter, On Feb 21, 2014, at 4:53 AM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi guys, Yesterday I made a small tweak to the BLAST+ packages to fix a bash syntax error in the fall back action: https://github.com/peterjc/galaxy_blast/commit/4b2fa1fbf485c4630cb6dc0beaf7286cb103bd71 The BLAST+ 2.2.27 package seems to have no test results at all. The above issue was due to a bug that was introduced into the install and test framework late yesterday, causing the test to not run at all. The bug has been corrected in preparation for tonight's test run. The Test environment entry for the above repository was produced by the preparation script. However, the BLAST+ 2.2.28 package has this strange unicode error: Automated tool test results Test runs 2014-02-20 12:48:52 This repository Error 'unicode' object has no attribute 'get' Error 'unicode' object has no attribute 'get' Error 'unicode' object has no attribute 'get' Error 'unicode' object has no attribute 'get' And similarly for BLAST+ 2.2.29, Automated tool test results Test runs 2014-02-20 14:37:58 This repository Error 'unicode' object has no attribute 'get' Error 'unicode' object has no attribute 'get' Error 'unicode' object has no attribute 'get' Error 'unicode' object has no attribute 'get' This is puzzling, since the changes I made to the packages seemed innocuous - making me wonder if something changed in the Test Tool Shed itself? The above 2 issues were the result of a bug in the Install and Test Framework that was exposed only with the changes you made to you repositories. The bug resulted in invalid information being included in the test results database column, and the unicode object has not attribute get error you're seeing above is due to the exception handling for displaying the invalid data in the Tool Shed repository's Test runs container. We've corrected the data in the database as well as the bug that produced it, so tonight's test run should not result in this behavior for these repositories. Thanks very much! Greg Von Kuster ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Nightly testing status on the (Test) Tool Shed
Hi Peter, Regarding youyr previous question about why the more usefule log resutls are not returned for display in the Tool Shed, the reason is that our use of fabric's local command for installing tool dependnecies restricts us from bewing able to capture that information. We are currently working to eliminate the use of fabric for tool dependency installations - the Trello card is here: https://trello.com/c/zJoRROvv/147-eliminate-the-use-of-fabric-as-the-installation-process-for-repositories We should have this workling this week, after which better error logs can be treurned to the Tool Shed and several other Terello cards can be handled. See my inline comments below… On Feb 20, 2014, at 5:30 AM, Peter Cock p.j.a.c...@googlemail.com wrote: On Thu, Feb 20, 2014 at 9:59 AM, Peter Cock p.j.a.c...@googlemail.com wrote: On Thu, Feb 20, 2014 at 2:56 AM, Greg Von Kuster g...@bx.psu.edu wrote: Your installation recipe for mira attempts to download a binary and if that fails, it echoes an error, but still performs set_environment actions.. Ah - I can see that now, I need a fall back action tag which either tries to compile MIRA or raises an explicit error. Thanks! Sorry, slightly confused: there was a fallback action tag which was and is meant to raise an error. It was accidentally raising an error due to a bash syntax error in my unquoted echo (which that detailed log you posted alerted me to), now fixed in my repository and uploaded to the Test Tool Shed: https://github.com/peterjc/pico_galaxy/commit/4b003cf9b5e6baa15c75bb65723c5af78e40c18d http://testtoolshed.g2.bx.psu.edu/view/peterjc/mira4_assembler The BLAST+ packages have the same glitch in their fall-back (so I will need to update them on the main and test Tool Shed, although I will delay that pending your reply to the problem below): https://github.com/peterjc/galaxy_blast/commit/4b2fa1fbf485c4630cb6dc0beaf7286cb103bd71 So, what happens is first the arch/os specific actions is tried, here actions os=linux architecture=x86_64 That was failing with MIRA4 due to the symlink bug (now fixed). Because the platform specific action failed, the generic actions was used - where I deliberately try to raise an error to signal the installation failed. What is surprising me is that the Tool Shed see the error, logs it, but still continues on (setting my environment variables, and reporting success). Why is that? I'll need some time to investigate this behavior - if I discover a framework issue, I'll provide a fix. Thanks, Peter P.S. The assert file exists action I suggested previously would be a neat way to catch some failing installs, e.g. here I expected the executables $MIRA4/mira and $MIRA4/mirabait etc to exist. I've filed a Trello card for this enhancement idea: https://trello.com/c/xBUhFvj0/1446-add-assert-file-exists-and-directory-exists-actions-to-tool-dependencies-xml Thanks - I've moved it to the Tool Shed Trello Board - its in Project in Planning. Greg Von Kuster ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Nightly testing status on the (Test) Tool Shed
Hello Peter, Please see below… On Feb 19, 2014, at 4:41 AM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi Greg, Overnight this has gone back to the previously observed missing data problem (e.g. email of 4th Feb) - no test results at all: http://testtoolshed.g2.bx.psu.edu/view/peterjc/mira4_assembler Test runs 2014-02-18 16:15:52 Automated test environment Time tested: 2014-02-18 16:15:52 System: Architecture: Python version: Galaxy revision: Galaxy database version: Tool shed revision: 12485:64e6873c8825 Tool shed database version: 22 Tool shed mercurial version: 2.2.3 Tools missing tests or test data Tool id: mira_4_0_mapping Tool version: 0.0.2 Tool guid: testtoolshed.g2.bx.psu.edu/repos/peterjc/mira4_assembler/mira_4_0_mapping/0.0.2 Missing components: Functional test definitions missing for mira_4_0_mapping. i.e. No test successes or failures reported. Collapsing the sections a bit: Test runs * 2014-02-18 16:15:52 *Automated test environment This was on Firefox (logged in, or logged out of the Tool Shed). Oddly, switching to Safari (not logged into the Tool Shed) on the same machine, or Chrome on a second machine, I also see an older test run's information where there is test output: Test runs * 2014-02-18 16:15:52 *Automated test environment * Tools missing tests or test data * 2014-02-18 03:40:12 * Automated test environment * Tests that failed * Tools missing tests or test data * Successful installation Might some strange cache or Firefox issue be hiding data? Your above email arrived in my inbox at 4:42 AM EST, and it looks like your repository in question was not yet tested. Using Firefox, I see the following resutls for today's test run, the results of which were updated in your Tool Shed repository at 4:50 AM EST. The nightly Install and Test Framework is finishing at about 8:00 AM EST for hte Test Tool Shed. The runs finish for the Main Tool Shed at about 5:00 AM EST. Test runs 2014-02-19 04:50:21 Automated test environment Time tested: 2014-02-19 04:50:21 System: Linux 3.8.0-30-generic Architecture: x86_64 Python version: 2.7.4 Galaxy revision: 12536:c85f8fb5d63e Galaxy database version: 118 Tool shed revision: 12485:64e6873c8825 Tool shed database version: 22 Tool shed mercurial version: 2.2.3 Tests that failed Tool id: mira_4_0_bait Tool version: mira_4_0_bait Test: test_tool_00 (functional.test_toolbox.TestForTool_testtoolshed.g2.bx.psu.edu/repos/peterjc/mira4_assembler/mira_4_0_bait/0.0.1) Stderr: Fatal error: Exit code 1 () Missing mirabait under $MIRA4, '/ToolDepsTest/MIRA/4.0/peterjc/mira4_assembler/133b863a8a40/mirabait' Folder contained: env.sh Traceback: Traceback (most recent call last): File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py, line 106, in test_tool self.do_it( td ) File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py, line 34, in do_it self._verify_outputs( testdef, test_history, shed_tool_id, data_list, galaxy_interactor ) File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py, line 67, in _verify_outputs galaxy_interactor.verify_output( history, output_data, outfile, attributes=attributes, shed_tool_id=shed_tool_id, maxseconds=maxseconds ) File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/base/interactor.py, line 312, in verify_output self.twill_test_case.verify_dataset_correctness( outfile, hid=hid, attributes=attributes, shed_tool_id=shed_tool_id, maxseconds=maxseconds ) File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/base/twilltestcase.py, line 827, in verify_dataset_correctness self._assert_dataset_state( elem, 'ok' ) File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/base/twilltestcase.py, line 641, in _assert_dataset_state raise AssertionError( errmsg ) AssertionError: Expecting dataset state 'ok', but state is 'error'. Dataset blurb: error Tool id: mira_4_0_bait Tool version: mira_4_0_bait Test: test_tool_01 (functional.test_toolbox.TestForTool_testtoolshed.g2.bx.psu.edu/repos/peterjc/mira4_assembler/mira_4_0_bait/0.0.1) Stderr: Fatal error: Exit code 1 () Missing mirabait under $MIRA4, '/ToolDepsTest/MIRA/4.0/peterjc/mira4_assembler/133b863a8a40/mirabait' Folder contained: env.sh Traceback: Traceback (most recent call last): File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py, line 106, in test_tool
Re: [galaxy-dev] Nightly testing status on the (Test) Tool Shed
Hi Peter, Dave B and I just discovered that issue that causes your tests to fail. The problem lies with our current implementation supporting the move_directory_files action tag, which is used in your recipe. This tag ultimately uses Python's os.listdir via shutil.move. It turns out that certain Python versions produce different results from os.listdir - the order is different. This is a problem with MIRA because during the instlllation it is moving a directory of files where the directory includes symlinks to other files in the directory. When the symlinks are moved after the files to which they link, problems occur. Again, this behavior is intermittent depending on the Pyhton version ( and perhaps the os ). In any case, we have a fix for out move_directory_files which Dave is now committing. So your tests should pass with tonight's run - Let's hope! Thanks Peter! Greg Von Kuster On Feb 19, 2014, at 3:49 PM, Peter Cock p.j.a.c...@googlemail.com wrote: On Wed, Feb 19, 2014 at 8:42 PM, Greg Von Kuster g...@bx.psu.edu wrote: Hello Peter, Please see below... ... Your above email arrived in my inbox at 4:42 AM EST, and it looks like your repository in question was not yet tested. Ah. That would partly explain things - perhaps coupled with a cache effect it might even explain why different browsers seemed to give me different output Could Galaxy include the time zone (EST) in the Tool Shed test time stamps and/or show GMT/UTC? (You have no easy way of knowing the user's local timezone do you?). Alternatively, Test in progress would be nice :) As to the test failure, it seems $INSTALL_DIR or at least $MIRA4 only contains env.sh (which puzzles me). Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Nightly testing status on the (Test) Tool Shed
Hi Peter, On Feb 19, 2014, at 6:09 PM, Peter Cock p.j.a.c...@googlemail.com wrote: On Wed, Feb 19, 2014 at 9:33 PM, Greg Von Kuster g...@bx.psu.edu wrote: Hi Peter, Dave B and I just discovered that issue that causes your tests to fail. The problem lies with our current implementation supporting the move_directory_files action tag, which is used in your recipe. This tag ultimately uses Python's os.listdir via shutil.move. It turns out that certain Python versions produce different results from os.listdir - the order is different. This is a problem with MIRA because during the instlllation it is moving a directory of files where the directory includes symlinks to other files in the directory. When the symlinks are moved after the files to which they link, problems occur. Again, this behavior is intermittent depending on the Pyhton version ( and perhaps the os ). In any case, we have a fix for out move_directory_files which Dave is now committing. So your tests should pass with tonight's run - Let's hope! Thanks Peter! Well done Greg Dave, That explanation makes perfect sense to me - but I can see it took some digging to solve it! There is actually one MIRA binary (mira) and several aliases (e.g. mirabait) which are really symlinks back to mira. Magic. My next query would be: why wasn't the failing action move_directory_files being caught as a failed install? Your installation recipe for mira attempts to download a binary and if that fails, it echoes an error, but still performs set_environment actions.. The tool dependency installation process will fall back to attempting the installation using the action tags to handle install from source and compile if downloading a pre-compiled binary fails. However, your recipe does not have include installing from source, so the installation process simply assumed your set_environment tags were all that was necessary. If the recipe for installing from source is not too complex, maybe you could add it to your recipe. It's probabluy not critical though since we now have the fix to the framework. Here is the entire log of the installtion process that helped us uncover the problem - notice that upon failure of the initial binary installation, the process proceeds with install and compile recipe for tool dependency MIRA. tool_shed.galaxy_install.tool_dependencies.fabric_util DEBUG 2014-02-19 04:50:42,285 Successfully downloaded from url: https://downloads.sourceforge.net/project/mira-assembler/MIRA/stable/mira_4.0_linux-gnu_x86_64_static.tar.bz2 tool_shed.galaxy_install.tool_dependencies.install_util ERROR 2014-02-19 04:50:42,318 Error installing tool dependency MIRA version 4.0. Traceback (most recent call last): File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/lib/tool_shed/galaxy_install/tool_dependencies/install_util.py, line 289, in install_and_build_package_via_fabric tool_dependency = fabric_util.install_and_build_package( app, tool_dependency, actions_dict ) File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/lib/tool_shed/galaxy_install/tool_dependencies/fabric_util.py, line 581, in install_and_build_package destination_dir=os.path.join( action_dict[ 'destination_directory' ] ) ) File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/lib/tool_shed/galaxy_install/tool_dependencies/td_common_util.py, line 309, in move_directory_files shutil.move( source_file, destination_file ) File /usr/lib/python2.7/shutil.py, line 301, in move copy2(src, real_dst) File /usr/lib/python2.7/shutil.py, line 130, in copy2 copyfile(src, dst) File /usr/lib/python2.7/shutil.py, line 82, in copyfile with open(src, 'rb') as fsrc: IOError: [Errno 2] No such file or directory: '/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/repositories_with_tools/tmp/tmpdr4lDp/tmp-toolshed-mtdV8W9PM/mira_4.0_linux-gnu_x86_64_static/bin/mirabait' tool_shed.galaxy_install.tool_dependencies.install_util DEBUG 2014-02-19 04:50:42,541 Error downloading binary for tool dependency MIRA version 4.0: File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/lib/tool_shed/galaxy_install/tool_dependencies/install_util.py, line 289, in install_and_build_package_via_fabric tool_dependency = fabric_util.install_and_build_package( app, tool_dependency, actions_dict ) File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/lib/tool_shed/galaxy_install/tool_dependencies/fabric_util.py, line 581, in install_and_build_package destination_dir=os.path.join( action_dict[ 'destination_directory' ] ) ) File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/lib/tool_shed/galaxy_install
Re: [galaxy-dev] Nightly testing status on the (Test) Tool Shed
Hi Peter, We'll try to get some time to look at your tool_dependencies.xml recipe as soon as possible. Greg Von Kuster On Feb 18, 2014, at 5:47 AM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi Greg, Dave, RE: http://testtoolshed.g2.bx.psu.edu/view/peterjc/mira4_assembler Same as before, although now both the mirabait and mira de novo tests are attempted: Missing mirabait under $MIRA4, '/ToolDepsTest/MIRA/4.0/peterjc/mira4_assembler/133b863a8a40/mirabait' Folder contained: env.sh Missing mira under $MIRA4, '/ToolDepsTest/MIRA/4.0/peterjc/mira4_assembler/133b863a8a40/mira' Folder contained: env.sh My Python wrapper checks the $MIRA4 environment variable, which should be the $INSTALL_DIR location which should hold the MIRA binaries (mira, mirabait, etc). According to the Python wrapper, the $MIRA variable was set to: /ToolDepsTest/MIRA/4.0/peterjc/mira4_assembler/133b863a8a40/ However, the Python wrapper reports this folder only contains one file, env.sh - which leads me to suspect a glitch in my tool_dependencies.xml and/or the installation framework. Regards, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Nightly testing status on the (Test) Tool Shed
Hi Peter, It looks like you made some changes to your tool dependency recipe since the last test run. Dave B and I followed the latest revision of your recipe and manually installed it using https://downloads.sourceforge.net/project/mira-assembler/MIRA/stable/mira_4.0_linux-gnu_x86_64_static.tar.bz2 and it looked ok. Let's see what the restults of tonight's test run is for this repository and if there are still problems after your latest changes we'll track them down. Thanks, Greg Von Kuster On Feb 17, 2014, at 6:15 AM, Peter Cock p.j.a.c...@googlemail.com wrote: On Sat, Feb 15, 2014 at 12:32 PM, Peter Cock p.j.a.c...@googlemail.com wrote: On Sat, Feb 15, 2014 at 12:25 PM, Greg Von Kuster g...@bx.psu.edu wrote: On Feb 3, 2014, at 7:01 PM, Peter Cock p.j.a.c...@googlemail.com wrote: Returning to the MIRA4 problem, I got a meaningful test failure which I fixed (I hadn't set an environment variable in the install script), but I now get only partial test output: http://testtoolshed.g2.bx.psu.edu/view/peterjc/mira4_assembler ... That's all fine, but where are the test results for mira_bait which I would expect to pass? It looks lie the mira_bait tests results are being populated, but the tests are failing, not passing. This doesn't look like an issue with the Tool Shed's install and test framework. Confirmed - the Tool Shed issue where only partial test output was shown is fixed (did you work out why the test output was missing?). I was hoping to see a passing test, but I can now see a failing test (still something not quite right with my dependency installation script - I'll look at this next week hopefully). Hi Greg, I added a little more debug information to the mirabait test, which confirms a problem in the tool_dependencies.xml file: Missing mirabait under $MIRA4, '/ToolDepsTest/MIRA/4.0/peterjc/mira4_assembler/7fcabeeca5df/mirabait' Folder contained: env.sh i.e. $MIRA4 was set to $INSTALL_DIR which is the folder /ToolDepsTest/MIRA/4.0/peterjc/mira4_assembler/7fcabeeca5df/ and (surprisingly) it only contains one file, env.sh Greg: Could you confirm that on the server? i.e. what does this give? $ ls -l /ToolDepsTest/MIRA/4.0/peterjc/mira4_assembler/7fcabeeca5df/ ... Upon downloading mira_4.0_linux-gnu_x86_64_static.tar.bz2 and decompressing it, the tool shed should have changed into the main folder mira_4.0_linux-gnu_x86_64_static and then moved the contents of the bin folder into the $INSTALL_DIR, action type=download_by_urlhttps://downloads.sourceforge.net/project/mira-assembler/MIRA/stable/mira_4.0_linux-gnu_x86_64_static.tar.bz2/action action type=move_directory_files source_directorybin/source_directory destination_directory$INSTALL_DIR/destination_directory /action i.e. Move files mira, mirabait, miraconvert and miramem into $INSTALL_DIR I am somewhat puzzled - other tools like the BLAST+ installers use a very similar setup and work fine (although there I used $PATH instead). Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Nightly testing status on the (Test) Tool Shed
Hi Peter, On Feb 3, 2014, at 7:01 PM, Peter Cock p.j.a.c...@googlemail.com wrote: Moreover there is a similar problem with the matching repository on the Test Tool Shed, http://testtoolshed.g2.bx.psu.edu/view/peterjc/blastxml_to_top_descr 2014-01-04 showing failed tests for proteomics_search_tandem_1 2013-12-20 showing failed tests for proteomics_search_tandem_1 2013-12-18 tests for blastxml_to_top_descr passed 2013-12-17 tests for blastxml_to_top_descr passed Like the same repository on the main tool shed, the above issue were fixed for the blastxml_to_top_descr on the test tool shed in https://bitbucket.org/galaxy/galaxy-central/commits/e3a4d4d813fdb8a34cfd6d596e0f4bbdb2d9e211 -- Returning to the MIRA4 problem, I got a meaningful test failure which I fixed (I hadn't set an environment variable in the install script), but I now get only partial test output: http://testtoolshed.g2.bx.psu.edu/view/peterjc/mira4_assembler Not datestamped (a glitch if there is only one test entry?): Automated tool test results Automated test environment Time tested: 2014-02-03 16:25:08 System: Architecture: Python version: Galaxy revision: Galaxy database version: Tool shed revision: 12332:d9f6f3f24671 Tool shed database version: 22 Tool shed mercurial version: 2.2.3 Tools missing tests or test data Tool id: mira_4_0_mapping Tool version: 0.0.2 Tool guid: testtoolshed.g2.bx.psu.edu/repos/peterjc/mira4_assembler/mira_4_0_mapping/0.0.2 Missing components: Functional test definitions missing for mira_4_0_mapping. Tool id: mira_4_0_de_novo Tool version: 0.0.2 Tool guid: testtoolshed.g2.bx.psu.edu/repos/peterjc/mira4_assembler/mira_4_0_de_novo/0.0.2 Missing components: Functional test definitions missing for mira_4_0_de_novo. That's all fine, but where are the test results for mira_bait which I would expect to pass? It looks lie the mira_bait tests results are being populated, bu the tests are failing, not passing. This doesn't look like an issue with the Tool Shed's install and test framework. Tests that failed Tool id: mira_4_0_bait Tool version: mira_4_0_bait Test: test_tool_00 (functional.test_toolbox.TestForTool_testtoolshed.g2.bx.psu.edu/repos/peterjc/mira4_assembler/mira_4_0_bait/0.0.1) Stderr: Fatal error: Exit code 1 () Missing mirabait under $MIRA4, '/ToolDepsTest/MIRA/4.0/peterjc/mira4_assembler/a6a56440567c/mirabait' Traceback: Traceback (most recent call last): File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py, line 106, in test_tool self.do_it( td ) File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py, line 34, in do_it self._verify_outputs( testdef, test_history, shed_tool_id, data_list, galaxy_interactor ) File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py, line 67, in _verify_outputs galaxy_interactor.verify_output( history, output_data, outfile, attributes=attributes, shed_tool_id=shed_tool_id, maxseconds=maxseconds ) File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/base/interactor.py, line 312, in verify_output self.twill_test_case.verify_dataset_correctness( outfile, hid=hid, attributes=attributes, shed_tool_id=shed_tool_id, maxseconds=maxseconds ) File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/base/twilltestcase.py, line 827, in verify_dataset_correctness self._assert_dataset_state( elem, 'ok' ) File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/base/twilltestcase.py, line 641, in _assert_dataset_state raise AssertionError( errmsg ) AssertionError: Expecting dataset state 'ok', but state is 'error'. Dataset blurb: error Tool id: mira_4_0_bait Tool version: mira_4_0_bait Test: test_tool_01 (functional.test_toolbox.TestForTool_testtoolshed.g2.bx.psu.edu/repos/peterjc/mira4_assembler/mira_4_0_bait/0.0.1) Stderr: Fatal error: Exit code 1 () Missing mirabait under $MIRA4, '/ToolDepsTest/MIRA/4.0/peterjc/mira4_assembler/a6a56440567c/mirabait' Traceback: Traceback (most recent call last): File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py, line 106, in test_tool self.do_it( td ) File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py, line 34, in do_it self._verify_outputs( testdef, test_history, shed_tool_id, data_list, galaxy_interactor ) File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py, line 67, in _verify_outputs galaxy_interactor.verify_output(
Re: [galaxy-dev] WARNING:galaxy.datatypes.registry:Overriding conflicting datatype
I've gone ahead and made the change in the datatypes registry to use log.debu instead of log.warning, so hopefully this issue is resolved. The changeset in central is 12497:0b0018fa5d20, which has also be grafted to stable. Greg Von Kuster On Feb 13, 2014, at 11:51 AM, Jim Johnson johns...@umn.edu wrote: On 2/13/14, 10:22 AM, Greg Von Kuster wrote: Hi JJ, On Feb 13, 2014, at 11:00 AM, Jim Johnson johns...@umn.edu wrote: I now have 2 versions of snpeff installed from the toolshed on my galaxy server. Each snpeff version includes an identical datatypes_conf.xml The galaxy server is setting metadata externally. When any job runs, (in may case I was running a picard tool), the following message is written to the job stderr: WARNING:galaxy.datatypes.registry:Overriding conflicting datatype with extension 'snpeffdb', using datatype from /galaxy/database/tmp/tmpdKGVnQ. Without stdio tags or otherwise catching the stderr, the job state is set to error. I believe I'm responsible for the above warning message, but I'm not sure I like the resulting behavior - I wasn't aware that the job runner now captures warning messages like this and sets the job state to error. At the time I enhance the Galaxy datatypes components to work with the Tool Shed, this was not the behavior. I'm now assuming that best practice would be to put the datatypes in a separate toolshed repository that would be a dependency for the tools repository. Thus the datatypes would not loaded multiple times and avoid overrride conflicts. Yes, the above plan would be a best practice. Though I could still have some issues on my test galaxy instance, since I often have tools from both the toolshed and testtoolshed simultaneously. In the meantime, I have just commented out the warning message. Any other advice on this issue? I'm wondering if we should just eliminate the warning message altogether sicne it now results in jobe errors. Do others agree? Thanks, JJ -- James E. Johnson, Minnesota Supercomputing Institute, University of Minnesota Or we could move it to log.debug level, which wouldn't usually be set on a production server. JJ -- James E. Johnson, Minnesota Supercomputing Institute, University of Minnesota ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Nightly testing status on the (Test) Tool Shed
Hi Peter, The following issue was corrected in change set https://bitbucket.org/galaxy/galaxy-central/commits/e3a4d4d813fdb8a34cfd6d596e0f4bbdb2d9e211 On Jan 29, 2014, at 12:55 PM, Peter Cock p.j.a.c...@googlemail.com wrote: http://toolshed.g2.bx.psu.edu/view/peterjc/blastxml_to_top_descr 2014-01-29 missing blast_datatypes dependency (why?) 2014-01-28 showing failed tests for sambamba_filter (wrong tool) I'll be taking a look at the remaining issues on the following Trello card next. https://trello.com/c/M1rwVWhI/143-nightly-test-runs-1 Greg Von Kuster ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Twill compares wrong file; was: Twill comparisons ignore GALAXY_TEST_NO_CLEANUP
Hello Peter, I'm also sure we could enhance hte twill code to use the dictionary approach used in the API based testing. However, pervasive use of Javascript is being introduced into the Galaxy code base at a fast pace and twill does not work with Javascript. So clearly the use of twill for functional testing in Galaxy will have to be eliminated although I'm not aware of any formal plans to do so. All of the tool shed functional tests and the tool shed's install and test framework use twill as well, so I'll be working to eliminate its use in those frameworks as soon as possible. John has been working on some pieces of the puzzle for eliminating twill (e.g., the API based testing), but I assume there is significant work left to completely eliminate the use of twill. So perhaps we should plan to continue to add enhancements like this to the twill framework even though we'll have to soon eliminate it due to Javascript. I'm not sure if others agree with this though. Greg Von Kuster On Feb 12, 2014, at 5:46 AM, Peter Cock p.j.a.c...@googlemail.com wrote: I think it ought be possible to refactor the Twill test code to use the same dictionary approach used in the API based testing - and so support listing the test output in any order and only testing some of the files. However, I'll wait and see what you guys have to say. Peter. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Twill compares wrong file; was: Twill comparisons ignore GALAXY_TEST_NO_CLEANUP
Hi Björn, I think twill is going to be around for a while yet, but it is quickly becoming more critical to eliminate it. I haven't yet looked at Johns tests using the API, but my understanding is that they are focused on tools. We're using twill a lot in other areas besides tools, so it will require some work to eliminate it completely. Some enhancements to our twill framework will undoubtedly be necessaruy form some time. However, non-trivial enhancements should be thought about carefully. Thanks! Greg Von Kuster On Feb 13, 2014, at 5:46 PM, Björn Grüning bjoern.gruen...@gmail.com wrote: Hi Dan, Hi all, I just wanted to point out that there are several different ways to compare the history output item to a test file beyond the default “diff”, including contains, re_match, sim_size, etc, this is set by the “compare” attribute in the xml tag for the test output. If you really don’t care about the contents of an output, you could, for example, use contains against an empty file. That is indeed a nice trick! It is potentially a way to only compare the first 1000 lines, assuming that the rest is identical. Unfortunately, I'm not able to complete the MACS2 tests, because I was not able to get conditionals of conditionals working. Is that a known bug? As I understood Greg correctly, we should not invest much more time in the twill testing and are waiting for the new API based tests? Is that correct? Thanks, Bjoern Sorry that I didn’t provide more specific examples, as I am at a conference this week. Thanks for using Galaxy, Dan On Feb 13, 2014, at 5:06 AM, Peter Cock p.j.a.c...@googlemail.com wrote: On Thu, Feb 13, 2014 at 7:51 AM, Björn Grüning bjoern.gruen...@gmail.com wrote: Am Mittwoch, den 12.02.2014, 10:46 + schrieb Peter Cock: ... However, at only 5kb the FASTA file is small and fine to bundle - but the BAM (49kb), log (123kb) and MAF (764kb) quickly add up so I would prefer not to have to include these (under source code control, and for the Tool Shed upload). Also, as you might guess from the large number of allowed differences, the log file is quite variable (lots of time stamps plus Galaxy's temp filenames appear). Do you have any advise which filesize is feasible and at which point we should skip the tests? I'm working currently on MACS2 wrappers and to have reasonable tests I need up to 50mb ob test files. I try to use small files under say 10kb if possible - partly as smaller unit tests are often easier to diagnose when they break, but also as noted above to reduce overhead for version control, and the ToolShed upload/download time. 50mb does sound excessive, but if that's the minimum maybe it is still better than no test at all? Providing gz input files would be one option to get it down to 10mb, but that does not work and it's still 10mb ... I thought the test framework would allow you to provide gzipped input since it uses the upload tool internally which would handle this. If that breaks I'd consider it a bug. Gzipped output files might be more difficult, since the validation is based on direct file comparison. Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Nightly testing status on the (Test) Tool Shed
Hi Peter, I've created the following Trello card for this. I know what the problems are, but I won't have time to get fixes for them for a while. Please continue to reposrt issues you discover and I'll make sure to get fixes as soon as I get time. https://trello.com/c/M1rwVWhI/143-nightly-test-runs-1 Thanks Greg Von Kuster On Feb 3, 2014, at 7:01 PM, Peter Cock p.j.a.c...@googlemail.com wrote: On Thu, Jan 30, 2014 at 4:32 PM, Greg Von Kuster g...@bx.psu.edu wrote: On Jan 29, 2014, at 1:18 PM, Greg Von Kuster g...@bx.psu.edu wrote: Peter wrote: http://toolshed.g2.bx.psu.edu/view/peterjc/blastxml_to_top_descr 2014-01-29 missing blast_datatypes dependency (why?) 2014-01-28 showing failed tests for sambamba_filter (wrong tool) I'll have to take a look at the above issue. I'm completely swamped though, so it will be a while. I believe I have the above issue resolved in 12331:fd4936ae8219 ( at least I took a step in the right direction ). We'll have to wait for tonight's test run to make sure everything is handled correctly. Hi again Greg, That one is still weird - was that fix deployed to the main toolshed? http://toolshed.g2.bx.psu.edu/view/peterjc/blastxml_to_top_descr Moreover there is a similar problem with the matching repository on the Test Tool Shed, http://testtoolshed.g2.bx.psu.edu/view/peterjc/blastxml_to_top_descr 2014-01-04 showing failed tests for proteomics_search_tandem_1 2013-12-20 showing failed tests for proteomics_search_tandem_1 2013-12-18 tests for blastxml_to_top_descr passed 2013-12-17 tests for blastxml_to_top_descr passed -- Returning to the MIRA4 problem, I got a meaningful test failure which I fixed (I hadn't set an environment variable in the install script), but I now get only partial test output: http://testtoolshed.g2.bx.psu.edu/view/peterjc/mira4_assembler Not datestamped (a glitch if there is only one test entry?): Automated tool test results Automated test environment Time tested: 2014-02-03 16:25:08 System: Architecture: Python version: Galaxy revision: Galaxy database version: Tool shed revision: 12332:d9f6f3f24671 Tool shed database version: 22 Tool shed mercurial version: 2.2.3 Tools missing tests or test data Tool id: mira_4_0_mapping Tool version: 0.0.2 Tool guid: testtoolshed.g2.bx.psu.edu/repos/peterjc/mira4_assembler/mira_4_0_mapping/0.0.2 Missing components: Functional test definitions missing for mira_4_0_mapping. Tool id: mira_4_0_de_novo Tool version: 0.0.2 Tool guid: testtoolshed.g2.bx.psu.edu/repos/peterjc/mira4_assembler/mira_4_0_de_novo/0.0.2 Missing components: Functional test definitions missing for mira_4_0_de_novo. That's all fine, but where are the test results for mira_bait which I would expect to pass? Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Nightly testing status on the (Test) Tool Shed
Hi Peter, I've created the following Trello card for this. I know what the problems are, but I won't have time to get fixes for them for a while. Please continue to reposrt issues you discover and I'll make sure to get fixes as soon as I get time. https://trello.com/c/M1rwVWhI/143-nightly-test-runs-1 Thanks Greg Von Kuster On Feb 3, 2014, at 7:01 PM, Peter Cock p.j.a.c...@googlemail.com wrote: On Thu, Jan 30, 2014 at 4:32 PM, Greg Von Kuster g...@bx.psu.edu wrote: On Jan 29, 2014, at 1:18 PM, Greg Von Kuster g...@bx.psu.edu wrote: Peter wrote: http://toolshed.g2.bx.psu.edu/view/peterjc/blastxml_to_top_descr 2014-01-29 missing blast_datatypes dependency (why?) 2014-01-28 showing failed tests for sambamba_filter (wrong tool) I'll have to take a look at the above issue. I'm completely swamped though, so it will be a while. I believe I have the above issue resolved in 12331:fd4936ae8219 ( at least I took a step in the right direction ). We'll have to wait for tonight's test run to make sure everything is handled correctly. Hi again Greg, That one is still weird - was that fix deployed to the main toolshed? http://toolshed.g2.bx.psu.edu/view/peterjc/blastxml_to_top_descr Moreover there is a similar problem with the matching repository on the Test Tool Shed, http://testtoolshed.g2.bx.psu.edu/view/peterjc/blastxml_to_top_descr 2014-01-04 showing failed tests for proteomics_search_tandem_1 2013-12-20 showing failed tests for proteomics_search_tandem_1 2013-12-18 tests for blastxml_to_top_descr passed 2013-12-17 tests for blastxml_to_top_descr passed -- Returning to the MIRA4 problem, I got a meaningful test failure which I fixed (I hadn't set an environment variable in the install script), but I now get only partial test output: http://testtoolshed.g2.bx.psu.edu/view/peterjc/mira4_assembler Not datestamped (a glitch if there is only one test entry?): Automated tool test results Automated test environment Time tested: 2014-02-03 16:25:08 System: Architecture: Python version: Galaxy revision: Galaxy database version: Tool shed revision: 12332:d9f6f3f24671 Tool shed database version: 22 Tool shed mercurial version: 2.2.3 Tools missing tests or test data Tool id: mira_4_0_mapping Tool version: 0.0.2 Tool guid: testtoolshed.g2.bx.psu.edu/repos/peterjc/mira4_assembler/mira_4_0_mapping/0.0.2 Missing components: Functional test definitions missing for mira_4_0_mapping. Tool id: mira_4_0_de_novo Tool version: 0.0.2 Tool guid: testtoolshed.g2.bx.psu.edu/repos/peterjc/mira4_assembler/mira_4_0_de_novo/0.0.2 Missing components: Functional test definitions missing for mira_4_0_de_novo. That's all fine, but where are the test results for mira_bait which I would expect to pass? Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Nightly testing status on the (Test) Tool Shed
Hi Peter, On Jan 29, 2014, at 1:18 PM, Greg Von Kuster g...@bx.psu.edu wrote: Hi Peter, http://toolshed.g2.bx.psu.edu/view/peterjc/blastxml_to_top_descr 2014-01-29 missing blast_datatypes dependency (why?) 2014-01-28 showing failed tests for sambamba_filter (wrong tool) I'll have to take a look at the above issue. I'm completely swamped though, so it will be a while. I believe I have the above issue resolved in 12331:fd4936ae8219 ( at least I took a step in the right direction ). We'll have to wait for tonight's test run to make sure everything is handled correctly. Thanks for letting me know about this. Greg Von Kuster Regards, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Nightly testing status on the (Test) Tool Shed
Hi Peter, On Jan 29, 2014, at 12:55 PM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi all, Are both the main and test Tool Sheds currently meant to be running nightly tests again? Yes I've not really got back into Galaxy tool testing since Christmas. Most of my tools' test results look sensible, but there seem to be some issues still, e.g. http://testtoolshed.g2.bx.psu.edu/view/peterjc/mira4_assembler Not tested since 2014-01-04 I've reset the do_not_test column on the record above ( it was set to not allow testing ), so it should now be tested. http://toolshed.g2.bx.psu.edu/view/peterjc/blastxml_to_top_descr 2014-01-29 missing blast_datatypes dependency (why?) 2014-01-28 showing failed tests for sambamba_filter (wrong tool) I'll have to take a look at the above issue. I'm completely swamped though, so it will be a while. Regards, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] The main Galaxy Tool Shed is now running the next-stable branch
The main Galaxy Tool Shed is now running the next-stable branch in preparation for the next Galaxy release tentatively scheduled for 2 weeks from today. The current changeset revision on the main Tool Shed is: changeset: 12285:3fdf673bdfc9 branch: next-stable Complete documentation for all of the new Tool Shed features will be available for the release in 2 weeks, but feel free to check them out now if you want. Here are some of the more prominent new features: 1) The main Tool Shed is now scheduled to run the daily Tool Shed install and test framework, so installation and test results should hopefully be available for most repositories tomorrow. 2) Exporting and importing repository capsules should now be working between tool sheds (e.g., your local Tool Shed and the test Tool Shed, the test Tool Shed and the main Tool Shed, etc.). 3) Repository owners can now authorize others (either users or groups) to administer their repository using the new Manage repository administrators option in the Repository Actions pop-up menu. Please report any issues you uncover and we'll get them handled in preparation for the upcoming release. Thanks! Greg Von Kuster ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Toolshed throwing error when reading tool's XML
Hello Damion, Tool Shed repositories that contain tools that include dynamically generated select list parameters that refer to an entry in the tool_data_table_conf.xml file must contain a tool_data_table_conf.xml.sample file that contains the required entry for each dynamic parameter. Similarly, any index files (i.e., ~/tool-data/xxx.loc files) to which the tool_data_table_conf.xml file entries refer must be defined in xxx.loc.sample files included in the tool shed repository along with the tools. If any of these tool_data_table_conf.xml entries or any of the required xxx.loc.sample files are missing from the tool shed repository, the tools will not properly load and metadata will not be generated for the repository. This means that the tools cannot be automatically installed into a Galaxy instance. For those tools that include dynamically generated select list parameters that require a missing entry in the tool_data_table_conf.xml file, this file will be modified in real time (when it is installed into a Galaxy instance) by adding the entry from a tool_data_table_conf.xml.sample file contained in the tool shed repository. Based on your problem description, it seems that your repository may not include a required xxx.loc.sample file. Let me know if adding one does not solve the problem. Is your repository in one of the Galaxy team;'s public tool sheds? Greg Von Kuster On Jan 20, 2014, at 2:24 PM, Dooley, Damion damion.doo...@bccdc.ca wrote: I'm stumped: Can tools that are destined for the toolshed use options from_data_table=... ... ? In a blast_reporting.xml tool script I have param name=filter_column type=select label=Col options from_data_table=blast_reporting_fields filter type=static_value value=numeric column=type / filter type=sort_by column=name/ /options /param But when this gets submitted as a tar file into a toolshed repository, toolshed comes back with cryptic: Metadata may have been defined for some items in revision '5bbb25b32de2'. Correct the following problems if necessary and reset metadata. blast_reporting.xml - invalid literal for int() with base 10: 'type' I eliminate the filter type=static_value value=numeric column=type / line, but similar problem is still reported blast_reporting.xml - invalid literal for int() with base 10: 'name' This hints that maybe the toolshed, when it tests the code, it doesn't know what/where blast_reporting_fields is? Is there some special location that the blast_reporting_fields.loc file should be located within the uploaded tar file? (I tried putting it in various relative paths in the tar file, i.e. root, and under /tool-data/) p.s. the tool runs file with menu functioning in galaxy itself. Thanks for any pro tips you folks can come up with! Damion ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Toolshed throwing error when reading tool's XML
It's discussed in the following page of the Tool Shed wiki. https://wiki.galaxyproject.org/InstallingRepositoriesToGalaxy#Installing_Galaxy_tool_shed_repository_tools_into_a_local_Galaxy_instance On Jan 20, 2014, at 7:27 PM, Dooley, Damion damion.doo...@bccdc.ca wrote: That advice did the trick, thanks Greg. I only had .loc files in the upload set; didn't know about the .sample version or tool_data_table_conf.xml.sample. One thing, did I miss reading the documentation on this somewhere or is this workshop type knowledge? From: Greg Von Kuster [g...@bx.psu.edu] Sent: Monday, January 20, 2014 1:21 PM To: Dooley, Damion Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Toolshed throwing error when reading tool's XML Hello Damion, Tool Shed repositories that contain tools that include dynamically generated select list parameters that refer to an entry in the tool_data_table_conf.xml file must contain a tool_data_table_conf.xml.sample file that contains the required entry for each dynamic parameter. Similarly, any index files (i.e., ~/tool-data/xxx.loc files) to which the tool_data_table_conf.xml file entries refer must be defined in xxx.loc.sample files included in the tool shed repository along with the tools. If any of these tool_data_table_conf.xml entries or any of the required xxx.loc.sample files are missing from the tool shed repository, the tools will not properly load and metadata will not be generated for the repository. This means that the tools cannot be automatically installed into a Galaxy instance. For those tools that include dynamically generated select list parameters that require a missing entry in the tool_data_table_conf.xml file, this file will be modified in real time (when it is installed into a Galaxy instance) by adding the entry from a tool_data_table_conf.xml.sample file contained in the tool shed repository. Based on your problem description, it seems that your repository may not include a required xxx.loc.sample file. Let me know if adding one does not solve the problem. Is your repository in one of the Galaxy team;'s public tool sheds? Greg Von Kuster ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] making tools with Galaxy
Hello Lionel, Please send correspondence like this to the galaxy-dev@lists.bx.psu.edu mailing list rather than individual email accounts as doing so will ensure more timely and optimal repoonses. I'm not quite sure what code you are reading, so could you please clarify? In general, though, when you write a Galaxy wrapper to integrate a tool into Galaxy, you will be able to load the tool into the Galaxy tool panel, and the tool UI will display in the main Galaxy panel. When you execute the tool, a Galaxy job will get created which will produce an output in the form of a dataset. This dataset will be displayed in you Galaxy history as the result of excuting the tool. So, in your case, the history item will include the png file and, when clicked , it should display in your main Galaxy panel. There are a lot of details that are calrified in the Galaxy wiki - a good place to start is probably: https://wiki.galaxyproject.org/Admin/Tools/ToolConfigSyntax Greg Von Kuster On Jan 16, 2014, at 5:56 AM, Lionel Chiron lionel.chi...@nmrtec.com wrote: Hi Greg, I'm trying to make a tool in Galaxy for implementing our algorithm for mass spectrometry treatment but it is a little tricky for me.. To make run the algorithm in Python is ok, I produce a png with matplotlib .. but after that I 'm a little lost since I don't know where the picture is produced.. reading your code I can't figure out how you indicate where to produce the png file.. How can I do?? Thanks Lionel ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Change Tool Shed owner
Hello Philip,The owner cannot be changed because it is part of the name-spaced repository identifier ( i.e., the clone url includes the owner, tools have generated guids tha tinclude the owner, etc.). However, you can provide write authorization on the repository to others so that they can make changes to the repository. This is done in the following section of the Manage repository page:There is also a Trello card that relates to this. It specifies adding the ability for a erepository owner to authorizew other users to administer the repository. You can vote on it here:https://trello.com/c/guOeL1sF/28-toolshed-add-the-ability-for-a-repository-owner-to-grant-administrative-privileges-on-their-repository-in-the-tool-shed-to-otherThanks,Greg Von KusterOn Jan 14, 2014, at 4:44 PM, Philip Mabon philipma...@gmail.com wrote:Is it possible to permanentlychangethe owner of a Tool in the Tool Shed? Really want to transfer responsibility of my tools to others.Thanks! ___Please keep all replies on the list by using "reply all"in your mail client. To manage your subscriptions to thisand other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] sanitize_all_html option
Hello Pieter, Please make sure to address items like this to the galaxy-dev@lists.bx.psu.edu mailing list rather than individual email accounts as that will ensure more timely responses that include more optimal feedback. Sanitizing values from input text fields on tools and other Galaxy forms is an essential part of ensuring that the values will not wreak havoc within the Galaxy environment. Opening this up to being optional may be a concern to some Galaxy administrators. In any case, the Tool Shed probably should not have the ability to define the use of this feature since it has no affect within any of the Tool Shed environment ( only Galaxy or other applications in which things are installed from the Tool Shed will be affected ). So if it is decided by the Galaxy community that this feature ( i.e., sanitizing form text field values ) should be enhanced or altered, changes should be made within the Galaxy environment rather than the Tool Shed. As input regarding this request comes in from the community, perhaps we can create an appropriate Trello card to capture the direction we should go. Thanks very much for your request on this! Greg Von Kuster On Jan 13, 2014, at 6:16 AM, Lukasse, Pieter pieter.luka...@wur.nl wrote: Hi Greg, I have some tools which produce HTML and the default setting of the option sanitize_all_html will give problems and/or make the output look ugly. Would it be an option to let the administrator decide, for each tool he installs, whether this option should apply or not? Now is a global setting which applies to all tools, and in practice this results in it being set to “false”which means that in practice this is a “pseudo security item” as it will not be used that often. The alternative I have been thinking about is to add a checkbox to the “manage repository” screen to allow the admin to turn this feature on/off for a specific repository. See also the screenshot below. Maybe you are already working in this direction, but I thought I’d just share this idea with you. image001.png Best regards, Pieter Lukasse Wageningen UR, Plant Research International Departments of Bioscience and Bioinformatics Wageningen Campus, Building 107, Droevendaalsesteeg 1, 6708 PB, Wageningen, the Netherlands +31-317481122; skype: pieter.lukasse.wur http://www.pri.wur.nl ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Cloud Launch - Where is /mnt/galaxyData
Perfect thanks! On Mon, Jan 6, 2014 at 1:16 PM, Dannon Baker dannon.ba...@gmail.com wrote: Hey Greg, Sorry about the delay on this over the holidays. I see what you're working with now. If you install your customizations in a portable manner in /mnt/galaxyData that should be safe and we'll make sure to provide a smooth migration path going forward should that change. Only 'galaxy' type clusters will have /mnt/galaxy and data only clusters should continue to work just fine with /mnt/galaxyData. -Dannon On Mon, Dec 30, 2013 at 1:33 PM, greg margeem...@gmail.com wrote: Just following up on this. So it's ok to do all of my customizations in /mnt/galaxyData? Or am I doing something wrong, and I need to figure out how to boot up into a version that has /mnt/galaxy? Thanks again, Greg On Fri, Dec 20, 2013 at 2:49 PM, greg margeem...@gmail.com wrote: Yes, I just launched it a few hours ago. I just went here and filled out the form: https://usegalaxy.org/cloudlaunch I chose data cluster when prompted and enter 5GB. On Fri, Dec 20, 2013 at 2:48 PM, Dannon Baker dannon.ba...@gmail.com wrote: And is this a new instance you've just launched? If so, how'd you launch it? On Fri, Dec 20, 2013 at 2:37 PM, greg margeem...@gmail.com wrote: Here's what I'm seeing: ubuntu@ip-10-182-195-79:/$ ls / bin boot dev etc export home initrd.img lib lib64 lost+found media mnt opt proc root run sbin selinux srv sys tmp usr var vmlinuz ubuntu@ip-10-182-195-79:/$ ls mnt cm galaxyData lost+found transient_nfs ubuntu@ip-10-182-195-79:/$ ls /mnt/galaxyData/ export files tmp upload_store On Fri, Dec 20, 2013 at 2:20 PM, Dannon Baker dannon.ba...@gmail.com wrote: Does /mnt/galaxy exist, and does it have all of the expected galaxy components? /mnt/galaxyData might exist, but it should be either empty or a symlink to /mnt/galaxy if I remember correctly. If you're launching your cluster from usegalaxy.org/cloudlaunch, you should always be using the latest stuff. On Fri, Dec 20, 2013 at 2:13 PM, greg margeem...@gmail.com wrote: Thanks. I was wrong before. I actually do see a /mnt/galaxyData. Should I not being seeing that? Am I not on cloudman 2.0? -Greg On Fri, Dec 20, 2013 at 1:57 PM, Dannon Baker dannon.ba...@gmail.com wrote: With the Cloudman 2.0 release, the galaxyData and galaxyTools volumes have been merged to a single 'galaxy' volume. /mnt/galaxy is now your single persistent (by default, at least) volume, so, if you install your tool to here and share everything should work as expected. -Dannon On Fri, Dec 20, 2013 at 1:53 PM, greg margeem...@gmail.com wrote: Hi guys, I just launched an instance from https://usegalaxy.org/cloudlaunch and chose data cluster when prompted. Everything seems to have gone ok, but when I ssh in, I don't see /mnt/galaxyData Background: Basically we have a bioinformatics tool that needs SGE to run. So I want to install it on a galaxy cloud instance and then provide the share string to other researchers. I did this successfully a year or two ago by installing it to /mnt/galaxyData. Thanks, Greg ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Small icon bug in local galaxy install
Hello Damion, I believe this issue was corrected in https://bitbucket.org/galaxy/galaxy-central/commits/3d8841746ce9b65e19a44028dea2eac73a4eddf8 It looks like you are tracking the galaxy-central brach, so if you pull and update your repository you should get the fix. Greg Von Kuster On Jan 3, 2014, at 5:11 PM, Dooley, Damion damion.doo...@bccdc.ca wrote: A few of our local galaxy installs have a relative folder path (i.e. in universe_wsgi.ini, prefix = /galaxylab) . In the Admin tab, Manage installed tool shed repositories, the icons in the legend below aren't showing because they have an absolute url: img src=/static/june_2007_style/blue/ok_small.png / I can see in the galaxy-central/lib/tool_shed/galaxy_install/grids/admin_toolshed_grids.py code: return 'img src=/static/june_2007_style/blue/ok_small.png %s/' % latest_revision_tip_str Our /galaxylab is on galaxy changeset: 11247:a9a0ac9c1afa So I presume this code should be enhanced to get the prefix in there, or made a relative url? Regards, Damion ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] repository metadata reset not always working
Hello Pieter, I've added the following Trello card for this issue - we'll take a looke as soon as possible. https://trello.com/c/LZ3Lj9ye/125-problem-with-adding-deleting-adding-readme-files Thanks for reporing this, Greg Von Kuster On Dec 30, 2013, at 8:09 AM, Lukasse, Pieter pieter.luka...@wur.nl wrote: Hi Greg, I found an issue in Toolshed, not sure if it has been reported before. Steps: 1-Make a repository and add a tool .xml and a README file 2-Reset metadata and check if all information appears in the toolshed UI 3-Now delete the README file from the hg repository 4-Commit 5-(not sure whether I did this step) reset metadata 6-Now add the README file back but with different contents. Commit, reset metadata 7-In the toolshed UI you will see that the OLD REAME contents are displayed. This is just one example...I have the feeling it is not restricted to the README files of course...so it can be quite a serious issue. Regards, Pieter Lukasse Wageningen UR, Plant Research International Departments of Bioscience and Bioinformatics Wageningen Campus, Building 107, Droevendaalsesteeg 1, 6708 PB, Wageningen, the Netherlands +31-317481122; skype: pieter.lukasse.wur http://www.pri.wur.nl ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Cloud Launch - Where is /mnt/galaxyData
Just following up on this. So it's ok to do all of my customizations in /mnt/galaxyData? Or am I doing something wrong, and I need to figure out how to boot up into a version that has /mnt/galaxy? Thanks again, Greg On Fri, Dec 20, 2013 at 2:49 PM, greg margeem...@gmail.com wrote: Yes, I just launched it a few hours ago. I just went here and filled out the form: https://usegalaxy.org/cloudlaunch I chose data cluster when prompted and enter 5GB. On Fri, Dec 20, 2013 at 2:48 PM, Dannon Baker dannon.ba...@gmail.com wrote: And is this a new instance you've just launched? If so, how'd you launch it? On Fri, Dec 20, 2013 at 2:37 PM, greg margeem...@gmail.com wrote: Here's what I'm seeing: ubuntu@ip-10-182-195-79:/$ ls / bin boot dev etc export home initrd.img lib lib64 lost+found media mnt opt proc root run sbin selinux srv sys tmp usr var vmlinuz ubuntu@ip-10-182-195-79:/$ ls mnt cm galaxyData lost+found transient_nfs ubuntu@ip-10-182-195-79:/$ ls /mnt/galaxyData/ export files tmp upload_store On Fri, Dec 20, 2013 at 2:20 PM, Dannon Baker dannon.ba...@gmail.com wrote: Does /mnt/galaxy exist, and does it have all of the expected galaxy components? /mnt/galaxyData might exist, but it should be either empty or a symlink to /mnt/galaxy if I remember correctly. If you're launching your cluster from usegalaxy.org/cloudlaunch, you should always be using the latest stuff. On Fri, Dec 20, 2013 at 2:13 PM, greg margeem...@gmail.com wrote: Thanks. I was wrong before. I actually do see a /mnt/galaxyData. Should I not being seeing that? Am I not on cloudman 2.0? -Greg On Fri, Dec 20, 2013 at 1:57 PM, Dannon Baker dannon.ba...@gmail.com wrote: With the Cloudman 2.0 release, the galaxyData and galaxyTools volumes have been merged to a single 'galaxy' volume. /mnt/galaxy is now your single persistent (by default, at least) volume, so, if you install your tool to here and share everything should work as expected. -Dannon On Fri, Dec 20, 2013 at 1:53 PM, greg margeem...@gmail.com wrote: Hi guys, I just launched an instance from https://usegalaxy.org/cloudlaunch and chose data cluster when prompted. Everything seems to have gone ok, but when I ssh in, I don't see /mnt/galaxyData Background: Basically we have a bioinformatics tool that needs SGE to run. So I want to install it on a galaxy cloud instance and then provide the share string to other researchers. I did this successfully a year or two ago by installing it to /mnt/galaxyData. Thanks, Greg ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Cloud Launch - Where is /mnt/galaxyData
Hi guys, I just launched an instance from https://usegalaxy.org/cloudlaunch and chose data cluster when prompted. Everything seems to have gone ok, but when I ssh in, I don't see /mnt/galaxyData Background: Basically we have a bioinformatics tool that needs SGE to run. So I want to install it on a galaxy cloud instance and then provide the share string to other researchers. I did this successfully a year or two ago by installing it to /mnt/galaxyData. Thanks, Greg ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/