Re: [galaxy-dev] Tool Testing Enhancements
On Tue, Mar 18, 2014 at 7:10 PM, John Chilton wrote: > Hi Peter, > > Thanks for the bug report. It looks like if requests is available to > the framework these unicode errors go away (it doesn't have to fall > back on my poor attempt to provide a similar interface). > > https://github.com/jmchilton/pico_galaxy/commit/e7d37f31951d8e729cfdbda0ca9085feb7f2da73 > > I'll create a Trello card and try to add an egg for this - other > developers on the team have likewise said they would like a requests > dependency available in the past so I doubt there will be objections. > > -John Great, I'll use the manual 'requests' install for TravisCI in the short term... > P.S. > > It looks like the ftype changeset has already caught some errors (your > sample_seqs tool only produces fasta outputs but some outputs are > labelled as sff): > > https://travis-ci.org/jmchilton/pico_galaxy/jobs/21036295 Thanks - it was missing a tag: https://github.com/peterjc/pico_galaxy/commit/a2bc5fe8866fc9587e04a9fd84df50e010cdd52b Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Testing Enhancements
Hi Peter, Thanks for the bug report. It looks like if requests is available to the framework these unicode errors go away (it doesn't have to fall back on my poor attempt to provide a similar interface). https://github.com/jmchilton/pico_galaxy/commit/e7d37f31951d8e729cfdbda0ca9085feb7f2da73 I'll create a Trello card and try to add an egg for this - other developers on the team have likewise said they would like a requests dependency available in the past so I doubt there will be objections. -John P.S. It looks like the ftype changeset has already caught some errors (your sample_seqs tool only produces fasta outputs but some outputs are labelled as sff): https://travis-ci.org/jmchilton/pico_galaxy/jobs/21036295 On Tue, Mar 18, 2014 at 6:46 AM, Peter Cock wrote: > On Fri, Mar 14, 2014 at 8:36 PM, John Chilton wrote: >> Hello Tool Developers, >> >> Haven't known when to send this out, but I figure since you haven't >> received any e-mail from me today it might be a good time. >> >> tl;dr - Tool functional tests experienced a significant overhaul over >> the last release and will continue to change over the next couple >> releases, but in a backward compatible so hopefully you will not need >> to care unless you want to. >> >> ... > > Thank you John for this detailed report - I knew bits and pieces > from prior discussions, but still found this recap very useful. > > I'm now taking advantage of the new environment variable to test > with both GALAXY_TEST_DEFAULT_INTERACTOR=api (the new > framework) and GALAXY_TEST_DEFAULT_INTERACTOR=twill > (the old framework) under TravisCI, see: > > https://github.com/peterjc/galaxy_blast/commit/b9db5c9edc57314c5ab4122bce0b00fa2f9cfb94 > https://github.com/peterjc/pico_galaxy/commit/ceed9e0698989b7a617d75d6c483fa28ae61b333 > http://blastedbio.blogspot.co.uk/2013/09/using-travis-ci-for-testing-galaxy-tools.html > http://lists.bx.psu.edu/pipermail/galaxy-dev/2014-March/018677.html > > The BLAST+ tests are fine both ways, but there appears to be a > unicode issue with the API based testing of pico_bio (twill is fine): > https://travis-ci.org/peterjc/pico_galaxy/builds/21008616 > > UnicodeDecodeError: 'ascii' codec can't decode byte 0xbe in position > 847: ordinal not in range(128) > > Thanks, > > Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Testing Enhancements
On Fri, Mar 14, 2014 at 8:36 PM, John Chilton wrote: > Hello Tool Developers, > > Haven't known when to send this out, but I figure since you haven't > received any e-mail from me today it might be a good time. > > tl;dr - Tool functional tests experienced a significant overhaul over > the last release and will continue to change over the next couple > releases, but in a backward compatible so hopefully you will not need > to care unless you want to. > > ... Thank you John for this detailed report - I knew bits and pieces from prior discussions, but still found this recap very useful. I'm now taking advantage of the new environment variable to test with both GALAXY_TEST_DEFAULT_INTERACTOR=api (the new framework) and GALAXY_TEST_DEFAULT_INTERACTOR=twill (the old framework) under TravisCI, see: https://github.com/peterjc/galaxy_blast/commit/b9db5c9edc57314c5ab4122bce0b00fa2f9cfb94 https://github.com/peterjc/pico_galaxy/commit/ceed9e0698989b7a617d75d6c483fa28ae61b333 http://blastedbio.blogspot.co.uk/2013/09/using-travis-ci-for-testing-galaxy-tools.html http://lists.bx.psu.edu/pipermail/galaxy-dev/2014-March/018677.html The BLAST+ tests are fine both ways, but there appears to be a unicode issue with the API based testing of pico_bio (twill is fine): https://travis-ci.org/peterjc/pico_galaxy/builds/21008616 UnicodeDecodeError: 'ascii' codec can't decode byte 0xbe in position 847: ordinal not in range(128) Thanks, Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Tool Testing Enhancements
Hello Tool Developers, Haven't known when to send this out, but I figure since you haven't received any e-mail from me today it might be a good time. tl;dr - Tool functional tests experienced a significant overhaul over the last release and will continue to change over the next couple releases, but in a backward compatible so hopefully you will not need to care unless you want to. The last release included a considerable overhaul to the framework used to functionally test tools (i.e. actually running those tags in tool XML). This includes changes to how the tests are executed and an expansion in the expressiveness of these tests. As some background for why the changes are being made, currently tools are tested by starting up a test Galaxy instance and using a library called Twill to exercise the GUI and upload data and run tools through by exercising actual HTML pages. The last release added the optional ability to exercise these tool tests via Galaxy's tool API instead of via web forms, with the slightly longer term goal of eliminating the Twill testing option altogether. This will allow the Galaxy tool form API to evolve more quickly, eliminate much of the testing complexity and many of the limitations imposed by Twill while ensuring that Galaxy's tool running API is sufficiently expressive. I have put a significant amount of effort into backward compatibility - so all current tests SHOULD pass using the new method. I know some people have included significant hacks with their test definitions to sculpt them deal with Twill-isms. These may fail - I would like to catalog what these are and see if we can hack up the test framework to deal with them or transition these tests to use cleaner mechanisms allowed for by the more direct representation of tool parameters consumed by the API. If you are running tests locally with the run_functional_tests.sh script distributed with Galaxy, simply setting the environment variable GALAXY_TEST_DEFAULT_INTERACTOR to 'api' before running the script and all tests will be run through the API. Individual tests can be set to target the API (for instance if they use newer, API-only features outlined below) by adding the XML attribute interactor="api" to the test tag in the tool XML. This change works right now if you want to update test tool shed tools to target the new testing method and provide feedback. As part of this transition, many bugs were fixed that affect both styles of test and new features were added. The biggest improvement is the ability to specify parameters in a nested structure matching the inputs themselves. This allows disambiguation of repeat and conditional parameters for both styles of functional tests and in the case of API driven tests allows specification of any number of repeat block instances in tests (one still can only specify at most one instance for any repeat tag while using Twill driven tests). The following annotated tool XML examples concretely demonstrate this new expressiveness. https://bitbucket.org/galaxy/galaxy-central/src/tip/test/functional/tools/disambiguate_repeats.xml https://bitbucket.org/galaxy/galaxy-central/src/tip/test/functional/tools/disambiguate_cond.xml Additional Enhancements: Enhancement (Twill and API): Allow unspecified conditional test parameters. Previously specifying a parameter in a conditional but not the value of the conditional test parameter itself (the select or boolean parameter the conditional changes based on) resulted in cryptic test errors. Now the default is assumed if unspecified. https://bitbucket.org/galaxy/galaxy-central/commits/8b99c7bc5e11cd7c452c2ad86f951109b495271c https://bitbucket.org/galaxy/galaxy-central/commits/76bd49e658876f5cc90ec5b80033f13b058593b8 Enhancement (Twill and API): Allow specifying tests for composite extra files without checking primary file. https://bitbucket.org/galaxy/galaxy-central/commits/45c1d11cfff552ce095c0561b5fba2cb5d3610e3 Enhancement (API-only): Allow testing metadata properties on test output datasets. https://bitbucket.org/galaxy/galaxy-central/commits/01f05ab8df7092b6e03f35faf7f80b47acafa0eb Enhancement (API-only): Remove restriction of output order specification. https://bitbucket.org/galaxy/galaxy-central/commits/d05be33ad6b772c2a688d875c4cf895d6afac4e3 Enhancement: Allow testing repeat blocks with min="1" (twill) or any non-zero value (API) https://bitbucket.org/galaxy/galaxy-central/commits/4615f7424dcb4d2b61be5f388c761f45a1302a03 Enhancement (API-only): Allow testing of data parameters with multiple="True". https://bitbucket.org/galaxy/galaxy-central/commits/211f30207d48101a1250983d4f0f55c937f00a2e Fuller break down of this is development is available on Trello https://trello.com/c/MsJctFem. Once Galaxy and the tool shed have been modified to target the API when testing tools by default the tool XML syntax wiki page will be updated with examples of this new functionality. -John _
