Benji York wrote:
I'm working on making the zope.testing test runner run tests in
parallelized subprocesses.


I have some recent experience parallelising (and distributing across machines) test runs. This was in Java, with TestNG and Selenium, but we learned some interesting things. We basically cut a 45 minute test run to 10 minutes by distributing the tests across three machines, each running a full stack (Oracle, JBoss, Firefox) and Selenium Grid. I realise you're not trying to do anything quite as complex as that, but a parallel test runner ought to be extensible to support distribution across nodes in a grid. The main challenge there is to distribute deployment of the code to run, and to sync test setup so that all environments are identical. I suspect you'll find this out of scope to begin with, but I'd keep it in the back of your mind.

You will likely need some way of declaring tests that have to run in series. Sometimes that's just for sanity's sake, other times it's a requirement due to shared resources.

A nice way to do this is to make it possible to annotate tests to group them, and then to be able to declaratively configure some groups as serial. Any functional test that uses a shared external resource will require this. TestNG supports (as far as I recall):

 - Run all tests (methods) randomly and parallelise

- Run groups of tests (classes or declaratively specified named groups) in parallel, but run tests within the groups sequentially

 - Run all tests in series (i.e. single-threaded)

We should probably use test layers as the main grouping mechanism here. If you could declare a layer as "can be run in parallel with other layers" or "tests in this layer run in series", that'd be pretty powerful. I'm not 100% sure how this works with layers that derive from one another, and where you'd have two layers with a shared base class, though.

Parallisation can offer huge (!) speed increases, but it can also be hard to debug tests. I'd be tempted to let single threaded by the default, safe choice, and let people opt into parallisation only when they know what they are doing. Most test runs are quite quick anyway.

Test result reporting can be difficult. You'll probably need to collect all failures with tracebacks and report at the end. For long running test suites, this may not be ideal, since it's helpful to get early warning, so if you can find a way to get test output to be atomically output, then that'd be nice.

Debugging stuff that happens in parallel with pdb is also tricky. It must be easy to turn off parallel running and to run individual tests in a single process for each debugging.

To make this work with Selenium grid, we ended up building some infrastructure to manage environments (i.e. an allocation of database, web server and so on), and locks on those environments. We'd spawn one thread for each environment and feed tests to those threads as fast as they could run them. Each test run then grabbed an environment on setup, executed, and then released the lock for another test.

Oh, and please don't get rid of any tear-down. You'll definitely need it one day. Letting environments go dirty is generally troublesome, and gets only more difficult when you may have multiple threads trying to use those environments at once. I don't know how you've structured this, but I'd consider whether one layer could be shared across multiple threads/subprocesses, or if it's always a one-to-one thing.

I realise this is somewhat rambling, but I hope it's useful in any case. :)

Cheers,
Martin

--
Author of `Professional Plone Development`, a book for developers who
want to work with Plone. See http://martinaspeli.net/plone-book

_______________________________________________
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )

Reply via email to