Hello Tool Developers, Haven't known when to send this out, but I figure since you haven't received any e-mail from me today it might be a good time.
tl;dr - Tool functional tests experienced a significant overhaul over the last release and will continue to change over the next couple releases, but in a backward compatible so hopefully you will not need to care unless you want to. The last release included a considerable overhaul to the framework used to functionally test tools (i.e. actually running those <test> tags in tool XML). This includes changes to how the tests are executed and an expansion in the expressiveness of these tests. As some background for why the changes are being made, currently tools are tested by starting up a test Galaxy instance and using a library called Twill to exercise the GUI and upload data and run tools through by exercising actual HTML pages. The last release added the optional ability to exercise these tool tests via Galaxy's tool API instead of via web forms, with the slightly longer term goal of eliminating the Twill testing option altogether. This will allow the Galaxy tool form API to evolve more quickly, eliminate much of the testing complexity and many of the limitations imposed by Twill while ensuring that Galaxy's tool running API is sufficiently expressive. I have put a significant amount of effort into backward compatibility - so all current tests SHOULD pass using the new method. I know some people have included significant hacks with their test definitions to sculpt them deal with Twill-isms. These may fail - I would like to catalog what these are and see if we can hack up the test framework to deal with them or transition these tests to use cleaner mechanisms allowed for by the more direct representation of tool parameters consumed by the API. If you are running tests locally with the run_functional_tests.sh script distributed with Galaxy, simply setting the environment variable GALAXY_TEST_DEFAULT_INTERACTOR to 'api' before running the script and all tests will be run through the API. Individual tests can be set to target the API (for instance if they use newer, API-only features outlined below) by adding the XML attribute interactor="api" to the test tag in the tool XML. This change works right now if you want to update test tool shed tools to target the new testing method and provide feedback. As part of this transition, many bugs were fixed that affect both styles of test and new features were added. The biggest improvement is the ability to specify parameters in a nested structure matching the inputs themselves. This allows disambiguation of repeat and conditional parameters for both styles of functional tests and in the case of API driven tests allows specification of any number of repeat block instances in tests (one still can only specify at most one instance for any repeat tag while using Twill driven tests). The following annotated tool XML examples concretely demonstrate this new expressiveness. https://bitbucket.org/galaxy/galaxy-central/src/tip/test/functional/tools/disambiguate_repeats.xml https://bitbucket.org/galaxy/galaxy-central/src/tip/test/functional/tools/disambiguate_cond.xml Additional Enhancements: Enhancement (Twill and API): Allow unspecified conditional test parameters. Previously specifying a parameter in a conditional but not the value of the conditional test parameter itself (the select or boolean parameter the conditional changes based on) resulted in cryptic test errors. Now the default is assumed if unspecified. https://bitbucket.org/galaxy/galaxy-central/commits/8b99c7bc5e11cd7c452c2ad86f951109b495271c https://bitbucket.org/galaxy/galaxy-central/commits/76bd49e658876f5cc90ec5b80033f13b058593b8 Enhancement (Twill and API): Allow specifying tests for composite extra files without checking primary file. https://bitbucket.org/galaxy/galaxy-central/commits/45c1d11cfff552ce095c0561b5fba2cb5d3610e3 Enhancement (API-only): Allow testing metadata properties on test output datasets. https://bitbucket.org/galaxy/galaxy-central/commits/01f05ab8df7092b6e03f35faf7f80b47acafa0eb Enhancement (API-only): Remove restriction of output order specification. https://bitbucket.org/galaxy/galaxy-central/commits/d05be33ad6b772c2a688d875c4cf895d6afac4e3 Enhancement: Allow testing repeat blocks with min="1" (twill) or any non-zero value (API) https://bitbucket.org/galaxy/galaxy-central/commits/4615f7424dcb4d2b61be5f388c761f45a1302a03 Enhancement (API-only): Allow testing of data parameters with multiple="True". https://bitbucket.org/galaxy/galaxy-central/commits/211f30207d48101a1250983d4f0f55c937f00a2e Fuller break down of this is development is available on Trello https://trello.com/c/MsJctFem. Once Galaxy and the tool shed have been modified to target the API when testing tools by default the tool XML syntax wiki page will be updated with examples of this new functionality. -John ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/