On Thu, Jul 03, 2008 at 05:22:11PM -0400, Benji York wrote: > I'm working on making the zope.testing test runner run tests in > parallelized subprocesses. The option will likely be spelled -j N, > where N is the maximum number of processes.
That's wonderful news! > I have it basically working, but have noticed a couple odd corners of > the test runner that I'd like to clean up. They may be controversial, > so I'll ask about them here first. > > I'd like to 1) remove the layer tear-down mechanism entirely, and 2) > make (almost) all layers run in a subprocess. -1 in general. +1 if you do that only for -j N where N > 1. Running all the tests in a single process has the following benefits: * test coverage analysis produces results that are correct (well, correct often enough -- but it has no chance at all when the test runner forks a subprocess) * import pdb; pdb.set_trace() works > I want to do #1 because it would simplify the test runner code and no > one seems to be using the functionality anyway. That's news to me. A while ago I went through Zope 3 trunk (it was pre-eggsplosion IIRC) and made sure all test layers defined in it supported teardown. Granted, FunctionalTestLayer() has allow_teardown=False as the default, for two reasons: * backwards compatibility: in olden days functional test layers didn't support teardown * paranoia: it is in general impossible to determine whether calling CleanUp().cleanUp() will correctly clear all the global state (someone could easily write a custom ZCML directive that changed a global variable and forget to register a CleanUp hook), so disallowing teardowns were the conservative safe choice. It is entirely my fault that I haven't evangelized the allow_teardown=True option for creating new test layers. > It also appears from > reading the code that any tests run in a subprocess (and most are) will > never exercise the tear-down mechanism anyway. I guess that's fine for process state, but not so fine for external state (temporary files etc.). Hey, this might explain why SchoolTool's tests tend to fill up my buildbot's /tmp without cleaning up after themselves! I'll have to investigate some day. > #2 will add some process start-up overhead, but it'll only be one more > process than is already started (and any reasonably-sized test corpus > already starts several processes for each test run). The one exception > is for running with -D or with a pdb.set_trace() embedded in the code > under test. For that case we need a switch to say "don't start any > subprocesses at all", I suspect that will be spelled -j0. If that case needs to be supported anyway, what's the advantage of spawning exactly one subprocess when you run it with -j 1? I would also question whether pdb-unfriendly non-performance-enhancing option should be the default. > For motivation, some speed comparisons: running a particular test suite > with 3876 tests (mostly doctests, and mostly functional) without the > patch takes 6 minutes, 42 seconds; my branch runs the same tests in 3 > minutes and 22 seconds (give or take) on a dual-core box with 3 > simultaneous subprocesses. I know; for large test suites (by "large" I mean 40 minutes) I've been using an ugly hack (--odd/--even test filtering) that lets me use both CPUs if I manually run two instances of the test runner in two xterms in parallel. Regards, Marius Gedminas -- "Wipe Info uses hexadecimal values to wipe files. This provides more security than wiping with decimal values." -- Norton SystemWorks 2002 Manual
Description: Digital signature
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )