Benji York wrote:
I'm working on making the zope.testing test runner run tests in
parallelized subprocesses.
I have some recent experience parallelising (and distributing across
machines) test runs. This was in Java, with TestNG and Selenium, but we
learned some interesting things. We basically cut a 45 minute test run
to 10 minutes by distributing the tests across three machines, each
running a full stack (Oracle, JBoss, Firefox) and Selenium Grid. I
realise you're not trying to do anything quite as complex as that, but a
parallel test runner ought to be extensible to support distribution
across nodes in a grid. The main challenge there is to distribute
deployment of the code to run, and to sync test setup so that all
environments are identical. I suspect you'll find this out of scope to
begin with, but I'd keep it in the back of your mind.
You will likely need some way of declaring tests that have to run in
series. Sometimes that's just for sanity's sake, other times it's a
requirement due to shared resources.
A nice way to do this is to make it possible to annotate tests to group
them, and then to be able to declaratively configure some groups as
serial. Any functional test that uses a shared external resource will
require this. TestNG supports (as far as I recall):
- Run all tests (methods) randomly and parallelise
- Run groups of tests (classes or declaratively specified named
groups) in parallel, but run tests within the groups sequentially
- Run all tests in series (i.e. single-threaded)
We should probably use test layers as the main grouping mechanism here.
If you could declare a layer as "can be run in parallel with other
layers" or "tests in this layer run in series", that'd be pretty
powerful. I'm not 100% sure how this works with layers that derive from
one another, and where you'd have two layers with a shared base class,
though.
Parallisation can offer huge (!) speed increases, but it can also be
hard to debug tests. I'd be tempted to let single threaded by the
default, safe choice, and let people opt into parallisation only when
they know what they are doing. Most test runs are quite quick anyway.
Test result reporting can be difficult. You'll probably need to collect
all failures with tracebacks and report at the end. For long running
test suites, this may not be ideal, since it's helpful to get early
warning, so if you can find a way to get test output to be atomically
output, then that'd be nice.
Debugging stuff that happens in parallel with pdb is also tricky. It
must be easy to turn off parallel running and to run individual tests in
a single process for each debugging.
To make this work with Selenium grid, we ended up building some
infrastructure to manage environments (i.e. an allocation of database,
web server and so on), and locks on those environments. We'd spawn one
thread for each environment and feed tests to those threads as fast as
they could run them. Each test run then grabbed an environment on setup,
executed, and then released the lock for another test.
Oh, and please don't get rid of any tear-down. You'll definitely need it
one day. Letting environments go dirty is generally troublesome, and
gets only more difficult when you may have multiple threads trying to
use those environments at once. I don't know how you've structured this,
but I'd consider whether one layer could be shared across multiple
threads/subprocesses, or if it's always a one-to-one thing.
I realise this is somewhat rambling, but I hope it's useful in any case. :)
Cheers,
Martin
--
Author of `Professional Plone Development`, a book for developers who
want to work with Plone. See http://martinaspeli.net/plone-book
_______________________________________________
Zope-Dev maillist - Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
