Re: [galaxy-dev] TestToolShed out of date? (not running unit tests)
On Wed, Apr 24, 2013 at 10:12 AM, Peter Cock p.j.a.c...@googlemail.com wrote: ..., I've still finding some other cases where despite having tests defined, nothing shows up on the Tool Shed test results page. Here's an example where I see that two tools (promoter2 and wolf_psort) have no tests, but the results of all the other tools in this suite are not mentioned (some expected to fail/be skipped due to missing dependencies): http://toolshed.g2.bx.psu.edu/view/peterjc/tmhmm_and_signalp Revision 13:dc958c2a963a Equivalently: http://testtoolshed.g2.bx.psu.edu/view/peterjc/tmhmm_and_signalp Revision: 14:0d8b1d20ce9c Here's another example where based on what happens locally, I am expecting a failure: http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_rename Revision: 1:6ce8634e2035 This is failing for me locally, but again I suspect this is due to an issue in the framework - much like the blastxml_to_top_descr test is failing due to the datatypes not being updated, here there seems to be something similar going wrong regarding the second uploaded input file (which is used for to provide the potential column parameters). See this thread: http://lists.bx.psu.edu/pipermail/galaxy-dev/2010-November/003867.html http://lists.bx.psu.edu/pipermail/galaxy-dev/2013-April/014261.html Regards, Peter Hi Dave, Yet another case, http://toolshed.g2.bx.psu.edu/view/devteam/ncbi_blast_plus Revision: 8:1f546099212f and, http://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus Revision: 5:d42346e675c4 These list five or six invalid tools which do not define any unit tests, yet there is no mention of the tools present which do have tests (and if they passed or not). Even if ncbi_blastdbcmd_wrapper doesn't (yet) have a test, I still want to know if the ncbi_blastp_wrapper tests pass ;) Are this further symptoms of whatever was solved here? https://trello.com/card/toolshed-automated-functional-test-bug/506338ce32ae458f6d15e4b3/793 Or does would a new Trello issue be best? Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] TestToolShed out of date? (not running unit tests)
Hi Dave, On Thu, Apr 25, 2013 at 2:27 PM, Dave Bouvier d...@bx.psu.edu wrote: Peter, That is the intended behavior, but I've added a trello card (https://trello.com/c/O9YmzUT4) for revisiting that decision at some point. Is that the right Trello card? Is seems to be all about multiple tool shed repository revisions rather than what I'm asking about. We are definitely willing to be flexible about the testing conditions, but the primary goal of the automated testing framework was to verify functional correctness of an entire repository, to simplify the approval process for the category of tools contained within the repository. --Dave B. Well as you can tell, I disagree with that design choice: If there are tools with tests within a repository I think those tests should be run. It seems far more pragmatic than the current all-or-nothing approach (for which I can't see any real justification). This is especially frustrating when there are still obstacles from the test framework itself which block writing tests for all my tools. e.g. http://lists.bx.psu.edu/pipermail/galaxy-dev/2013-April/014280.html Given the primary goal of the automated testing framework was to verify functional correctness of an entire repository, not running any tests if just one tool is missing tests seems very bizarre. Right now I can run those tests locally, but I have no way to know if they would work on the Tool Shed tests - until I reach full test coverage for the repository. If all the tests provided were actually run, I can fix any failures now, rather than waiting until issues blocking adding any missing tests are resolved. I hope this aspect of the design can be revisited sooner rather than later. Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] TestToolShed out of date? (not running unit tests)
On Thu, Apr 25, 2013 at 4:25 PM, Dave Bouvier d...@bx.psu.edu wrote: Peter, Your point is well taken about the utility of the testing framework in the tool development process. The framework has been modified as of 9520:41d8cdde4729 to only flag a changeset revision not to be tested if no valid tests have been found in that revision. If one or more tools have functional tests, and there is test data for one or more of those tests, the repository will now be installed and tested. I will be re-running the testing framework on the test tool shed shortly. --Dave B. Excellent :) I have to say since having had a nightly buildbot running unit tests, and then TravisCI doing continuous integration (testing when a fresh set of commits is pushed to the repository) for Biopython, I've come to value unit tests even more. Having something like this on the Galaxy Tool Shed is a really good move :) Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] TestToolShed out of date? (not running unit tests)
Peter, I've committed a fix for this in 9480:a80175a97194, and re-run the automated testing framework. However, as you will see, the test still reports an error. This is not due to an issue in your repository, but an problem on the Galaxy side, as described in the following trello card: https://trello.com/c/KdGX3hkh --Dave B. On 4/17/13 06:20:39.000, Peter Cock wrote: On Wed, Apr 17, 2013 at 9:44 AM, Peter Cock p.j.a.c...@googlemail.com mailto:p.j.a.c...@googlemail.com wrote: On Tue, Apr 16, 2013 at 9:42 PM, Greg Von Kuster g...@bx.psu.edu mailto:g...@bx.psu.edu wrote: Hello Peter, On Apr 15, 2013, at 11:04 AM, Peter Cock wrote: The View tool functional test results option clearly lists failed tests, but it is not obvious if there were any successful tests. Could that be indicated somehow (e.g. five tests passed for tool xxx). Here's a problem case, http://toolshed.g2.bx.psu.edu/view/peterjc/blastxml_to_top_descr This has one unit test defined, tests test param name=blastxml_file value=blastp_four_human_vs_rhodopsin.xml ftype=blastxml / param name=topN value=3 / output name=tabular_file file=blastp_four_human_vs_rhodopsin_top3.tabular ftype=tabular / /test /tests However, it is not shown via View tool functional test results: Tool functional test results Time tested: ~ 6 hours ago *Tool Shed environment* Tool shed version: 9252:da2d6ee29f30 Tool shed database version: 17 Mercurial version: 2.2.3 *Galaxy environment* Galaxy version: unknown Galaxy database version: unknown Architecture: x86_64 Operating system: Linux 3.0.0-1-amd64 Python version: 2.7.2+ i.e. No mention of the fact there was (presumably) one successful test. Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] TestToolShed out of date? (not running unit tests)
On Fri, Apr 19, 2013 at 4:46 PM, Dave Bouvier d...@bx.psu.edu wrote: Peter, I've committed a fix for this in 9480:a80175a97194, and re-run the automated testing framework. However, as you will see, the test still reports an error. This is not due to an issue in your repository, but an problem on the Galaxy side, as described in the following trello card: https://trello.com/c/KdGX3hkh --Dave B. Thanks - once this and the best way forward for defining Python modules as dependencies are sorted out I should be able to get more of my tools in the Tool Shed passing their unit tests :) Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] TestToolShed out of date? (not running unit tests)
On Tue, Apr 16, 2013 at 9:42 PM, Greg Von Kuster g...@bx.psu.edu wrote: Hello Peter, On Apr 15, 2013, at 11:04 AM, Peter Cock wrote: The View tool functional test results option clearly lists failed tests, but it is not obvious if there were any successful tests. Could that be indicated somehow (e.g. five tests passed for tool xxx). If I understand your request, I believe we already provide it. Functional test results are displayed for tests that passed or failed. For example, see the functional test results for revision 3:aa76c8dd97e6http://toolshed.g2.bx.psu.edu/repository/view_changelog?id=9a07f99536a91e29 of jjohnson's pileup_to_vcf repository, which displays the following. Is this what you are asking for? Thanks! *Tests that failed* Tool id: *pileup_to_vcf* version: *2.0* *Test id* test_tool_01 ( functional.test_toolbox.TestForTool_toolshed.g2.bx.psu.edu/repos/jjohnson/pileup_to_vcf/pileup_to_vcf/2.0 ) *Status* Test failed *Stderr* *Traceback* Traceback (most recent call last): File /var/opt/buildbot/slaves/buildbot-install-test-1/buildbot-install-test-main-tool-shed-py27/build/test/functional/test_toolbox.py, line 171, in test_tool self.do_it( td, shed_tool_id=shed_tool_id ) File /var/opt/buildbot/slaves/buildbot-install-test-1/buildbot-install-test-main-tool-shed-py27/build/test/functional/test_toolbox.py, line 102, in do_it self.verify_dataset_correctness( outfile, hid=elem_hid, maxseconds=testdef.maxseconds, attributes=attributes, shed_tool_id=shed_tool_id ) File /var/opt/buildbot/slaves/buildbot-install-test-1/buildbot-install-test-main-tool-shed-py27/build/test/base/twilltestcase.py, line 798, in verify_dataset_correctness raise AssertionError( errmsg ) AssertionError: History item 2 different than expected No text matching expression 'chr1\t158571283\t.\tA\tC\t.\tPASS\t.*' was found in output file. *Tests that passed successfully* Tool id: *pileup_to_vcf* version: *2.0* *Test id* test_tool_00 ( functional.test_toolbox.TestForTool_toolshed.g2.bx.psu.edu/repos/jjohnson/pileup_to_vcf/pileup_to_vcf/2.0 ) *Status* Test passed That does look good - maybe I've been unlucky in the examples I've looked at due to missing dependencies and/or true results being hidden by this issue: https://trello.com/card/toolshed -automated-functional-test-bug/506338ce32ae458f6d15e4b3/793 Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] TestToolShed out of date? (not running unit tests)
On Wed, Apr 17, 2013 at 9:44 AM, Peter Cock p.j.a.c...@googlemail.comwrote: On Tue, Apr 16, 2013 at 9:42 PM, Greg Von Kuster g...@bx.psu.edu wrote: Hello Peter, On Apr 15, 2013, at 11:04 AM, Peter Cock wrote: The View tool functional test results option clearly lists failed tests, but it is not obvious if there were any successful tests. Could that be indicated somehow (e.g. five tests passed for tool xxx). Here's a problem case, http://toolshed.g2.bx.psu.edu/view/peterjc/blastxml_to_top_descr This has one unit test defined, tests test param name=blastxml_file value=blastp_four_human_vs_rhodopsin.xml ftype=blastxml / param name=topN value=3 / output name=tabular_file file=blastp_four_human_vs_rhodopsin_top3.tabular ftype=tabular / /test /tests However, it is not shown via View tool functional test results: Tool functional test results Time tested: ~ 6 hours ago *Tool Shed environment* Tool shed version: 9252:da2d6ee29f30 Tool shed database version: 17 Mercurial version: 2.2.3 *Galaxy environment* Galaxy version: unknown Galaxy database version: unknown Architecture: x86_64 Operating system: Linux 3.0.0-1-amd64 Python version: 2.7.2+ i.e. No mention of the fact there was (presumably) one successful test. Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] TestToolShed out of date? (not running unit tests)
Hello Peter, On Apr 15, 2013, at 11:04 AM, Peter Cock wrote: The View tool functional test results option clearly lists failed tests, but it is not obvious if there were any successful tests. Could that be indicated somehow (e.g. five tests passed for tool xxx). If I understand your request, I believe we already provide it. Functional test results are displayed for tests that passed or failed. For example, see the functional test results for revision 3:aa76c8dd97e6 of jjohnson's pileup_to_vcf repository, which displays the following. Is this what you are asking for? Thanks! Tests that failed Tool id: pileup_to_vcf version: 2.0 Test id test_tool_01 (functional.test_toolbox.TestForTool_toolshed.g2.bx.psu.edu/repos/jjohnson/pileup_to_vcf/pileup_to_vcf/2.0) Status Test failed Stderr Traceback Traceback (most recent call last): File /var/opt/buildbot/slaves/buildbot-install-test-1/buildbot-install-test-main-tool-shed-py27/build/test/functional/test_toolbox.py, line 171, in test_tool self.do_it( td, shed_tool_id=shed_tool_id ) File /var/opt/buildbot/slaves/buildbot-install-test-1/buildbot-install-test-main-tool-shed-py27/build/test/functional/test_toolbox.py, line 102, in do_it self.verify_dataset_correctness( outfile, hid=elem_hid, maxseconds=testdef.maxseconds, attributes=attributes, shed_tool_id=shed_tool_id ) File /var/opt/buildbot/slaves/buildbot-install-test-1/buildbot-install-test-main-tool-shed-py27/build/test/base/twilltestcase.py, line 798, in verify_dataset_correctness raise AssertionError( errmsg ) AssertionError: History item 2 different than expected No text matching expression 'chr1\t158571283\t.\tA\tC\t.\tPASS\t.*' was found in output file. Tests that passed successfully Tool id: pileup_to_vcf version: 2.0 Test id test_tool_00 (functional.test_toolbox.TestForTool_toolshed.g2.bx.psu.edu/repos/jjohnson/pileup_to_vcf/pileup_to_vcf/2.0) Status Test passed ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] TestToolShed out of date? (not running unit tests)
Peter, The automated testing framework is also running against the test tool shed. Could you send me a link to the repository that is behaving oddly? --Dave B. On 4/12/13 06:42:04.000, Peter Cock wrote: Hi all, I'd like to be able to experiment with the new automated testing recently added to the main Tool Shed http://toolshed.g2.bx.psu.edu/ by first trying my updates on the Test Tool Shed (to make sure my tests will work before making the tool update public). It appears that the testing features are not available on the Test Tool Shed, http://testtoolshed.g2.bx.psu.edu/ Is this correct? Is this intentional? Thanks, Peter P.S. This would be less urgent if I could solve this issue which is blocking my local testing: http://lists.bx.psu.edu/pipermail/galaxy-dev/2013-April/014123.html ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] TestToolShed out of date? (not running unit tests)
On Mon, Apr 15, 2013 at 2:20 PM, Dave Bouvier d...@bx.psu.edu wrote: Peter, The automated testing framework is also running against the test tool shed. Could you send me a link to the repository that is behaving oddly? e.g. http://toolshed.g2.bx.psu.edu/view/peterjc/tmhmm_and_signalp On the main tool shed, when I am logged in, on this page under the top right menu Repository actions the second entry is View tool functional test results. However, on the test tool shed, that menu item is missing: http://testtoolshed.g2.bx.psu.edu/view/peterjc/tmhmm_and_signalp Although the history is different, I believe the same tarball for v0.2.1 of the suite was uploaded to both the main and test tool sheds. Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] TestToolShed out of date? (not running unit tests)
Peter, I'm unable to duplicate that behavior, the View tool functional test results option shows up on the test tool shed both when I'm logged in and logged out. My suggestion would be to clear your browser's cookies and cache, and see if that makes a difference. --Dave B. On 4/15/13 09:35:40.000, Peter Cock wrote: On Mon, Apr 15, 2013 at 2:20 PM, Dave Bouvier d...@bx.psu.edu wrote: Peter, The automated testing framework is also running against the test tool shed. Could you send me a link to the repository that is behaving oddly? e.g. http://toolshed.g2.bx.psu.edu/view/peterjc/tmhmm_and_signalp On the main tool shed, when I am logged in, on this page under the top right menu Repository actions the second entry is View tool functional test results. However, on the test tool shed, that menu item is missing: http://testtoolshed.g2.bx.psu.edu/view/peterjc/tmhmm_and_signalp Although the history is different, I believe the same tarball for v0.2.1 of the suite was uploaded to both the main and test tool sheds. Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] TestToolShed out of date? (not running unit tests)
On Mon, Apr 15, 2013 at 3:56 PM, Dave Bouvier d...@bx.psu.edu wrote: Peter, I'm unable to duplicate that behavior, the View tool functional test results option shows up on the test tool shed both when I'm logged in and logged out. My suggestion would be to clear your browser's cookies and cache, and see if that makes a difference. --Dave B. That did it - thanks :) I also have a feature request now though ;) The View tool functional test results option clearly lists failed tests, but it is not obvious if there were any successful tests. Could that be indicated somehow (e.g. five tests passed for tool xxx). Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] TestToolShed out of date? (not running unit tests)
Hi all, I'd like to be able to experiment with the new automated testing recently added to the main Tool Shed http://toolshed.g2.bx.psu.edu/ by first trying my updates on the Test Tool Shed (to make sure my tests will work before making the tool update public). It appears that the testing features are not available on the Test Tool Shed, http://testtoolshed.g2.bx.psu.edu/ Is this correct? Is this intentional? Thanks, Peter P.S. This would be less urgent if I could solve this issue which is blocking my local testing: http://lists.bx.psu.edu/pipermail/galaxy-dev/2013-April/014123.html ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
