On Thu, 23 Sep 2010 10:29:58 -0700, Brion Vibber wrote: > On Thu, Sep 23, 2010 at 9:46 AM, Dan Nessett <[email protected]> wrote: > >> I am very much in favor of keeping it simple. I think the issue is >> whether we will support more than one regression test (or individual >> test associated with a regression test) running concurrently on the >> same test wiki. If not, then I agree, no switching logic is necessary. >> >> > *nod* > > It might be a good idea to divide the test set up into, say 'channels' > or 'bundles' which are independent of each other, but whose individual > steps must run in sequence. If the tests are designed well, you should > be able to run tests from multiple 'channels' on the same wiki > simultaneously -- just as in the real world, multiple users are doing > multiple things on your wiki at the same time. > > So one test set might be: > * create page A-{{unique-id}} as user One-{{unique-id}} * open editing > page as user One-{{unique-id}} * open and save the page as user > Two-{{unique-id}} * save the page as user One-{{unique-id}} * confirm > edit conflict / merging behavior was as expected > > And another might be: > * register a new user account User--{{unique-id}} * change skin option > in preferences > * confirm that the skin changed as expected > > These tests don't interfere with each other -- indeed if they did, that > would be information you'd need to know about a serious bug! > > Most test sets should be fairly separate like this; only some that > change global state (say, a site administrator using a global > configuration panel to change the default skin) would need to be run > separately. > > -- brion
After thinking about this some more I think you are right. We should at least start with something simple and only make it more complex (e.g., wiki resource switching) if the simple approach has significant problems. There is already a way to 'bundle' individual tests together. It is the selenium test suite. We could use that. We could then break up a regression test into separate test suites that could run concurrently. Summarizing, here is my understanding of your proposal: + A regression test run comprises a set of test suites, each of which may run concurrently. + If you want to run multiple regression tests concurrently, use different test wikis (which can run over the same code base, but which are identified by different URLs - i.e., rely on httpd to multiplex multiple concurrent regression tests). + If you want to run parts of a regression test concurrently, the unit of concurrency is the test suite. + A regression test begins by establishing a fresh wiki. Each test suite starts by establishing the wiki state it requires (e.g., for PagedTiffHanlder, loading Multipage.tiff). + It is an open question whether test suite or total regression test cleanup is necessary. It may be possible to elide this step and simply rely on regression test initialization to cleanup any wiki state left around by a previous test run. There are still some open questions: + How do you establish a fresh wiki for a URL used previously for a test run? + URLs identify test wikis. Only one regression test can run at time on any one of these. How do you synchronize regression test initiation so there is some sort of lock on a test wiki currently running a regression test? -- -- Dan Nessett _______________________________________________ Wikitech-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikitech-l
