On Thu, Jul 03, 2008 at 05:22:11PM -0400, Benji York wrote:
> I'm working on making the zope.testing test runner run tests in
> parallelized subprocesses.  The option will likely be spelled -j N,
> where N is the maximum number of processes.

That's wonderful news!

> I have it basically working, but have noticed a couple odd corners of
> the test runner that I'd like to clean up.  They may be controversial,
> so I'll ask about them here first.
> I'd like to 1) remove the layer tear-down mechanism entirely, and 2)
> make (almost) all layers run in a subprocess.

-1 in general.  +1 if you do that only for -j N where N > 1.

Running all the tests in a single process has the following benefits:

  * test coverage analysis produces results that are correct (well,
    correct often enough -- but it has no chance at all when the test
    runner forks a subprocess)

  * import pdb; pdb.set_trace() works

> I want to do #1 because it would simplify the test runner code and no
> one seems to be using the functionality anyway.

That's news to me.  A while ago I went through Zope 3 trunk (it was
pre-eggsplosion IIRC) and made sure all test layers defined in it
supported teardown.

Granted, FunctionalTestLayer() has allow_teardown=False as the default,
for two reasons:

  * backwards compatibility: in olden days functional test layers didn't
    support teardown

  * paranoia: it is in general impossible to determine whether calling
    CleanUp().cleanUp() will correctly clear all the global state
    (someone could easily write a custom ZCML directive that changed a
    global variable and forget to register a CleanUp hook), so
    disallowing teardowns were the conservative safe choice.

It is entirely my fault that I haven't evangelized the
allow_teardown=True option for creating new test layers.

> It also appears from
> reading the code that any tests run in a subprocess (and most are) will
> never exercise the tear-down mechanism anyway.

I guess that's fine for process state, but not so fine for external
state (temporary files etc.).

Hey, this might explain why SchoolTool's tests tend to fill up my
buildbot's /tmp without cleaning up after themselves!  I'll have to
investigate some day.

> #2 will add some process start-up overhead, but it'll only be one more
> process than is already started (and any reasonably-sized test corpus
> already starts several processes for each test run).  The one exception
> is for running with -D or with a pdb.set_trace() embedded in the code
> under test.  For that case we need a switch to say "don't start any
> subprocesses at all", I suspect that will be spelled -j0.

If that case needs to be supported anyway, what's the advantage of
spawning exactly one subprocess when you run it with -j 1?  I would also
question whether pdb-unfriendly non-performance-enhancing option should
be the default.

> For motivation, some speed comparisons: running a particular test suite
> with 3876 tests (mostly doctests, and mostly functional) without the
> patch takes 6 minutes, 42 seconds; my branch runs the same tests in 3
> minutes and 22 seconds (give or take) on a dual-core box with 3
> simultaneous subprocesses.

I know; for large test suites (by "large" I mean 40 minutes) I've been
using an ugly hack (--odd/--even test filtering) that lets me use both
CPUs if I manually run two instances of the test runner in two xterms in

Marius Gedminas
"Wipe Info uses hexadecimal values to wipe files. This provides more security
than wiping with decimal values."
        -- Norton SystemWorks 2002 Manual

Attachment: signature.asc
Description: Digital signature

Zope-Dev maillist  -  Zope-Dev@zope.org
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope )

Reply via email to